An image feature extraction method, device and electronic equipment
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU HUYA TECH CO LTD
- Filing Date
- 2026-04-10
- Publication Date
- 2026-06-26
Smart Images
Figure CN122289707A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing, and more specifically, to an image feature extraction method, apparatus, and electronic device. Background Technology
[0002] With the continuous development of image recognition technology, it has been applied to various technical fields. In the image recognition process, feature extraction is a core technical step; simply put, the quality of image feature extraction determines the final recognition result.
[0003] However, existing image feature extraction techniques are easily affected by changes in the image when dealing with dynamic scenes presented in continuous images, leading to distortion of the extracted image features and ultimately affecting the image recognition results. Therefore, there is an urgent need for a method that can effectively extract image features. Summary of the Invention
[0004] This invention provides an image feature extraction method, apparatus, and electronic device for effectively extracting image features.
[0005] According to a first aspect of this application, an image feature extraction method is provided, the method comprising: Obtain the coordinate parameters of the detection box in the image to be processed; the detection box is generated based on the object to be processed in the image to be processed. Based on the coordinate parameters of the detection box, inverse affine sampling is performed on the image to be processed to obtain an inverse affine image; The inverse affine image is obtained by adding position information to the image to be processed; The first image is resampled to obtain the second image; Extract features from the first image and the second image, and perform cross-attention fusion on the features of the first image and the second image to obtain attention fusion features as image features of the image to be processed.
[0006] By performing inverse affine sampling on the image to be processed to obtain an inverse affine image, geometric correction of the detection box region is achieved, ensuring accurate alignment of the object to be processed in the image. Combined with positional information based on coordinate parameters, the spatial consistency and positioning accuracy of features are enhanced, enabling accurate location and extraction of the object to be processed in the image, reducing errors caused by image movement or changes in viewpoint. Furthermore, by extracting features from the corrected image and its resampled version separately, and using a cross-attention mechanism for multi-scale feature fusion, the ability to extract local details and global structural features in the image is significantly improved, avoiding feature distortion caused by differences between consecutive images, thus achieving better and more effective feature extraction from the image.
[0007] Optionally, the step of adding position information to the inverse affine image based on the image to be processed to obtain the first image includes: Obtain the coordinate values of the pixels in the inverse affine image in the image to be processed; Based on the corresponding coordinate values, construct the coordinate value matrix of the inverse affine image; Based on the coordinate value matrix, position information is added to the inverse affine image to obtain the first image.
[0008] By constructing a coordinate matrix aligned with the inverse affine image space, and using this coordinate matrix to add positional information to the inverse affine image, pixel-level fusion of image content and positional information is achieved, providing spatial awareness for subsequent image feature extraction.
[0009] Optionally, the step of adding position information to the inverse affine image based on the coordinate value matrix to obtain the first image includes: The coordinate value matrix is normalized to obtain the coordinate normalization matrix of the inverse affine image; Based on the coordinate normalization matrix, positional information is added to the inverse affine image to obtain the first image.
[0010] By normalizing the coordinate value matrix according to a unified standard, the coordinate information contained in the coordinate value matrix is mapped to a unified numerical range, eliminating the order-of-magnitude changes caused by differences in the scale of the detection box, and providing stable positional information for subsequent feature extraction.
[0011] Optionally, the coordinate value matrix includes an abscissa value matrix and a ordinate value matrix; the coordinate normalization matrix includes an abscissa normalization matrix and a ordinate normalization matrix. The normalization process of the coordinate value matrix to obtain the coordinate normalization matrix of the inverse affine image includes: The width and height of the inverse affine image are calculated based on the coordinate parameters of the detection box; Based on the width of the inverse affine image, the horizontal coordinate value matrix is normalized to obtain the horizontal coordinate normalized matrix. Based on the height of the inverse affine image, the ordinate value matrix is normalized to obtain the ordinate normalized matrix.
[0012] By constructing horizontal and vertical coordinate value matrices respectively, and normalizing them according to the width and height of the inverse affine image, a horizontal coordinate normalized matrix and a vertical coordinate normalized matrix aligned with the geometric direction of the detection box are obtained. The horizontal and vertical coordinate normalized matrices explicitly encode the relative positions of each pixel in the inverse affine image within the local region, thereby adding corresponding positional information to each pixel position and providing clear spatial structure information for feature extraction.
[0013] Optionally, the resampling is downsampling; The step of performing cross-attention fusion on the features of the first image and the features of the second image to obtain attention fusion features as the image features of the image to be processed includes: The features of the second image are used as the query vector, and the features of the first image are used as the key vector and value vector for cross-attention fusion to obtain attention fusion features as the image features of the image to be processed.
[0014] By downsampling the first image, a low-resolution second image is obtained. The features of the low-resolution second image are used as the query vector, and the features of the high-resolution first image are used as the key vector and value vector for cross-attention fusion. This makes the fusion process dominated by low-frequency, global semantics, thereby selectively extracting useful information consistent with the overall structure from high-frequency details, improving the robustness and consistency of the features.
[0015] Optionally, the step of performing inverse affine sampling processing on the image to be processed based on the coordinate parameters of the detection box to obtain an inverse affine image includes: The detection box is expanded to obtain an expanded detection box, and the coordinate parameters of the expanded detection box in the image to be processed are obtained. A sampling grid is constructed based on the expanded detection box, and the inverse affine transformation matrix from the sampling grid to the image to be processed is calculated based on the coordinate parameters of the expanded detection box. Based on the inverse affine transformation matrix and the sampling grid, the image to be processed is subjected to inverse affine sampling to obtain the inverse affine image.
[0016] By expanding the detection box, the range of inverse affine sampling is increased, improving the tolerance to minor changes in the position or scale of the detection box and ensuring that the inverse affine image completely covers the main area of the object to be processed.
[0017] Optionally, expanding the detection box to obtain an expanded detection box and obtaining the coordinate parameters of the expanded detection box in the image to be processed includes: The geometric center coordinates of the detection box, the rotation angle of the detection box relative to the image to be processed, and the size of the detection box are calculated based on the coordinate parameters of the detection box. The size of the detection frame is expanded to obtain the expanded detection frame size, and the expanded detection frame is obtained based on the expanded detection frame size; The coordinate parameters of the expanded detection frame are calculated based on the geometric center coordinates, the rotation angle, and the expanded detection frame size.
[0018] By using the geometric center coordinates and rotation angle of the detection frame as a reference, and only expanding the size of the detection frame, the sampling area corresponding to the expanded detection frame can still maintain an overall stable position when the detection frame is locally offset, thereby improving the consistency of the inverse affine sampling results.
[0019] Optionally, the step of performing inverse affine sampling processing on the image to be processed according to the inverse affine transformation matrix and the sampling grid to obtain the inverse affine image includes: Based on the grid structure of the sampling grid, construct the grid coordinate system of the sampling grid; For the grid coordinates of the sampling grid in the grid coordinate system, calculate its floating-point coordinates in the image to be processed according to the inverse affine transformation matrix; In the image to be processed, obtain several integer coordinates that are closest to the floating-point coordinates, and the pixel values at the locations of the several integer coordinates; Calculate the pixel value at the location of the floating-point coordinates based on the pixel values at the locations of the integer coordinates. The inverse affine image is generated by using the pixel value at the location of the floating-point coordinates as the pixel value at the grid coordinates.
[0020] By using pixel values of neighboring integer coordinates around floating-point coordinates for interpolation calculations, continuous and smooth resampling of the image to be processed is achieved, effectively avoiding image distortion caused by coordinate discretization and improving the geometric fidelity and content integrity of inverse affine images.
[0021] According to a second aspect of this application, an image feature extraction apparatus is provided, the apparatus comprising: A coordinate acquisition module is used to acquire the coordinate parameters of the detection box in the image to be processed; the detection box is generated based on the object to be processed in the image to be processed. The inverse affine processing module is used to perform inverse affine sampling processing on the image to be processed according to the coordinate parameters of the detection box to obtain an inverse affine image; A location information adding module is used to add location information to the inverse affine image based on the image to be processed, so as to obtain a first image; A resampling module is used to resample the first image to obtain a second image; The feature extraction module is used to extract features from the first image and features from the second image, and to perform cross-attention fusion on the features of the first image and features of the second image to obtain attention fusion features as image features of the image to be processed.
[0022] According to a third aspect of this application, an electronic device is provided, comprising: Memory, used to store one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements the image feature extraction method described in the first aspect above.
[0023] According to a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image feature extraction method described in the first aspect above.
[0024] Based on any of the above aspects, the image feature extraction method, apparatus, electronic device, and computer storage medium provided in this application obtain an inverse affine image by performing inverse affine sampling on the image to be processed, thereby achieving geometric correction of the detection box region and ensuring accurate alignment of the object to be processed in the image. Combined with positional information based on coordinate parameters, it enhances the spatial consistency and positioning accuracy of features, enabling accurate positioning and extraction of the object to be processed in the image, reducing errors caused by image movement or changes in viewing angle. Furthermore, by extracting features from the corrected image and its resampled version respectively, and using a cross-attention mechanism for multi-scale feature fusion, it significantly improves the ability to extract local details and global structural features in the image, avoiding feature distortion caused by differences between consecutive images, thus better achieving effective feature extraction from the image.
[0025] This application provides an image feature extraction method, apparatus, electronic device, and computer storage medium. By constructing a coordinate value matrix aligned with the inverse affine image space, and using the coordinate value matrix to add position information to the inverse affine image, pixel-level fusion of image content and position information is achieved, providing spatial awareness capability for subsequent image feature extraction.
[0026] The image feature extraction method, apparatus, electronic device, and computer storage medium provided in this application normalize the coordinate value matrix according to a unified standard, mapping the coordinate information contained in the coordinate value matrix to a unified numerical range, eliminating the order-of-magnitude changes caused by differences in the detection box scale, and providing stable position information for subsequent feature extraction.
[0027] This application provides an image feature extraction method, apparatus, electronic device, and computer storage medium. By constructing horizontal and vertical coordinate value matrices respectively and normalizing them according to the width and height of the inverse affine image, a horizontal coordinate normalized matrix and a vertical coordinate normalized matrix aligned with the geometric direction of the detection box are obtained. The horizontal and vertical coordinate normalized matrices explicitly encode the relative positions of each pixel in the inverse affine image within the local region, thereby adding corresponding positional information to each pixel position and providing clear spatial structure information for feature extraction.
[0028] This application provides an image feature extraction method, apparatus, electronic device, and computer storage medium. By downsampling the first image, a low-resolution second image is obtained. The features of the low-resolution second image are used as a query vector, and the features of the high-resolution first image are used as key vectors and value vectors for cross-attention fusion. This makes the fusion process dominated by low-frequency, global semantics, thereby selectively extracting useful information consistent with the overall structure from high-frequency details, improving the robustness and consistency of the features.
[0029] The image feature extraction method, apparatus, electronic device, and computer storage medium provided in this application expand the detection box, thereby increasing the range of inverse affine sampling and improving the tolerance to minor changes in the position or scale of the detection box, ensuring that the inverse affine image completely covers the main area of the object to be processed.
[0030] This application provides an image feature extraction method, apparatus, electronic device, and computer storage medium. By using the geometric center coordinates and rotation angle of the detection box as a reference, and only expanding the size of the detection box, the sampling area corresponding to the expanded detection box can still maintain the overall position stability when the detection box is locally offset, thereby improving the consistency of the inverse affine sampling results.
[0031] The image feature extraction method, apparatus, electronic device, and computer storage medium provided in this application embodiment utilize pixel values of neighboring integer coordinates around floating-point coordinates for interpolation calculation, thereby achieving continuous and smooth resampling of the image to be processed, effectively avoiding image distortion caused by coordinate discretization, and improving the geometric fidelity and content integrity of inverse affine images. Attached Figure Description
[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0033] Figure 1 This is a schematic application scenario diagram of the image feature extraction method provided in this embodiment.
[0034] Figure 2 This is a flowchart illustrating the steps of the image feature extraction method provided in this embodiment.
[0035] Figure 3 This is a schematic flowchart illustrating the steps for acquiring the first image in this embodiment.
[0036] Figure 4 This is a schematic diagram of the steps for obtaining an inverse affine image provided in this embodiment.
[0037] Figure 5 This is a flowchart illustrating the sub-steps of step S21 provided in this embodiment.
[0038] Figure 6 This is a flowchart illustrating the sub-steps of step S23 provided in this embodiment.
[0039] Figure 7 This is a schematic diagram of the functional modules of the image feature extraction device provided in this embodiment.
[0040] Figure 8 This embodiment provides a schematic diagram of the electronic device. Detailed Implementation
[0041] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this application. To better illustrate the following embodiments, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0042] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0044] With the continuous development of image recognition technology, it has been applied to various technical fields. In the image recognition process, feature extraction is a core technical step; simply put, the quality of image feature extraction determines the final recognition result.
[0045] Existing image feature extraction techniques primarily involve detecting target objects in an image, selecting them using bounding boxes, cropping the image based on these boxes to extract the target objects, and then extracting features from the cropped portions. However, this approach is highly sensitive to coordinate changes. During bounding box generation, the image resolution significantly affects the coordinates of the corner points, often resulting in random pixel shifts or scaling. This leads to cropping bias, which, amplified by resizing, causes significant displacement of the image content. Consequently, the extracted features appear farther apart in high-dimensional space, ultimately impacting image recognition.
[0046] Meanwhile, in continuous images, the position, scale, or angle of the object to be processed (such as text, face, or object) often undergoes slight but frequent changes due to the shaking of the acquisition device, the movement of the target, or the change of the viewing angle. As mentioned above, existing image feature extraction techniques are easily affected by the shaking of the image changes when facing the dynamic scene presented by continuous images, which leads to the distortion of the extracted image features and ultimately affects the image recognition effect.
[0047] Therefore, there is an urgent need for a method that can effectively extract image features.
[0048] This embodiment provides a technical solution that can solve the above problems. The specific implementation of this application will be described in detail below with reference to the accompanying drawings.
[0049] An exemplary diagram illustrating an application scenario of an image feature extraction method provided in this application embodiment is shown below. Figure 1As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100.
[0050] Understandably, the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smartphone terminal, personal computer, tablet computer, vehicle terminal, etc., but is not limited to these.
[0051] In one possible implementation, server 100 and terminal 200 may each execute an image feature extraction method provided in the embodiments of this application. Alternatively, the image feature extraction method provided in the embodiments of this application may be partially executed in server 100 and partially executed in terminal 200.
[0052] like Figure 2 As shown, this embodiment provides an image feature extraction method, which may include the following steps: S1: Obtain the coordinate parameters of the detection box in the image to be processed; In this embodiment, the detection box is a rectangular box used to locate a specific object in the image, and the detection box can be generated according to the object to be processed in the image to be processed; In one implementation, the objects to be processed in the image can be identified, and a detection box can be used to select the objects based on their coverage area. It is understood that although the detection box is a rectangle, it is not necessarily parallel to the object; the detection box can be rotated at a certain angle to the object, and this rotation angle is determined by the coverage area of the object.
[0053] In this embodiment, the coordinate parameters can be the coordinates of the four corners of the detection box in the image to be processed. The position of the detection box in the image to be processed can be determined using these coordinate parameters.
[0054] In one implementation, the image to be processed can be processed using an existing target detection model with detection capabilities to obtain the location of the object to be processed in the image and output the detection box of the object to be processed.
[0055] S2: Based on the coordinate parameters of the detection box, perform inverse affine sampling processing on the image to be processed to obtain an inverse affine image; In this embodiment, as Figure 4 As shown, step S2 may include the following sub-steps: S21: Expand the detection box to obtain an expanded detection box, and obtain the coordinate parameters of the expanded detection box in the image to be processed; Understandably, the detection box generated based on the object to be processed usually fits the object perfectly. However, during the generation of the detection box, factors such as the resolution of the image to be processed can cause random pixel shifts or size scaling of the corner coordinates of the actual generated detection box. Therefore, in order to obtain the object to be processed more completely and facilitate the subsequent extraction of complete features, this embodiment extends the detection box.
[0056] In one implementation, such as Figure 5 As shown, step S21 may include the following sub-steps: S211: Calculate the geometric center coordinates of the detection box, the rotation angle of the detection box relative to the image to be processed, and the size of the detection box based on the coordinate parameters of the detection box; S212: Expand the size of the detection frame to obtain the expanded detection frame size, and obtain the expanded detection frame based on the expanded detection frame size; S213: Calculate the coordinate parameters of the expanded detection frame based on the geometric center coordinates, the rotation angle, and the expanded detection frame size.
[0057] In this embodiment, the geometric center coordinates of the detection box can be calculated using the coordinate parameters of the detection box in the image to be processed, such as the coordinates of the four corner points. Simultaneously, the width and height of the detection box can also be calculated as its dimensions. The sides corresponding to the width and height of the detection box can be determined based on the shape, type, and position of the image to be processed. For example, for text-based objects to be processed, if the text in the object is read horizontally, the width of the detection box can be set to be parallel to the reading direction of the text, and the height of the detection box can be determined based on the number of lines of text.
[0058] The rotation angle of the detection frame can be understood as the angle between the width of the detection frame and the width of the image to be processed, which can be calculated using the coordinates of the two corner points of the width of the detection frame.
[0059] In one implementation, the size of the detection frame can be expanded based on a preset expansion coefficient. For example, the width and height of the detection frame are multiplied by the expansion coefficient to obtain the expanded width and height as the expanded detection frame size.
[0060] Understandably, in this embodiment, by expanding the size of the detection frame, the geometric center coordinates of the detection frame and the rotation angle of the detection frame are retained. This ensures that even when the sampling area corresponding to the expanded detection frame is shifted locally, such as at corner points, its geometric center remains stable. Consequently, the expanded detection frame can always completely cover the object to be processed and keep the object to be processed at the center of the detection frame.
[0061] The coordinate parameters of the expanded detection frame may include the coordinates of the four corner points of the expanded detection frame. The distances from the four corner points to the geometric center coordinates can be determined based on the coordinates of the four corner points and the geometric center coordinates. The expanded detection frame is then extended based on these distances to obtain the distances from the four corner points to the geometric center coordinates. Furthermore, the coordinates of the four corner points of the expanded detection frame can be calculated simultaneously based on the rotation angle and the size of the expanded detection frame, and used as the coordinate parameters of the expanded detection frame.
[0062] S22: Construct a sampling grid based on the expanded detection box, and calculate the inverse affine transformation matrix from the sampling grid to the image to be processed based on the coordinate parameters of the expanded detection box; In this embodiment, the coordinate parameters of the expanded detection box may include the coordinates of the four corner points of the expanded detection box in the image to be processed. Therefore, the inverse affine transformation matrix can be calculated by mapping the coordinates of the four corner points of the expanded detection box in the image to be processed with the coordinates of the four corner points corresponding to the sampling grid relative to the sampling grid itself.
[0063] For example, a grid coordinate system can be constructed in the sampling grid, and the grid coordinates of the four corner points of the sampling grid can be obtained in the grid coordinate system, represented as (0, 0), (W, 0), (0, H) and (W, H), where W is the width of the sampling grid and H is the height of the sampling grid. The coordinates of the four corner points of the expanded detection box in the image to be processed can be represented as (u1, v1), (u2, v2), (u3, v3) and (u4, v4). Based on the grid coordinates of the four corner points of the sampling grid and the coordinates of the four corner points of the expanded detection box, the inverse affine transformation matrix is calculated, such that for a grid coordinate (x, y) in the sampling grid and a coordinate (u, v) in the image to be processed, the following formula can be satisfied: In the formula, Let be the inverse affine transformation matrix.
[0064] S23: Perform inverse affine sampling processing on the image to be processed according to the inverse affine transformation matrix and the sampling grid, to obtain the inverse affine image.
[0065] In this embodiment, as Figure 6 shown, the step S23 may include the following sub-steps: S231: Construct a grid coordinate system of the sampling grid according to the grid structure of the sampling grid; In this embodiment, the sampling grid has a "field" - shaped grid structure, and thus a grid coordinate system can be constructed in the sampling grid through the "field" - shaped grid structure. Further, the density of the "field" - shaped grid structure of the sampling grid is set according to the resolution of the inverse affine image to be obtained, so that the grid coordinate system in the sampling grid can well match the distribution of the sampled pixel points.
[0066] It can be understood that the grid coordinate system can be constructed simultaneously when the sampling grid is constructed, which is convenient for calculating the inverse affine transformation matrix by using the grid coordinate system.
[0067] S232: For the grid coordinates of the sampling grid in the grid coordinate system, calculate its floating - point value coordinates in the image to be processed according to the inverse affine transformation matrix; In this embodiment, since it is necessary to match the distribution of the pixels in the inverse affine image, the abscissa value and the ordinate value of the grid coordinates are both integers, so that each grid coordinate in the grid coordinate system matches a pixel in the inverse affine image, and thus the inverse affine image can be generated based on the sampling grid.
[0068] As described above, the inverse affine transformation matrix can realize the mapping between the grid coordinates of the sampling grid and the coordinates of the image to be processed. Therefore, all the grid coordinates in the grid coordinate system can obtain their coordinates in the image to be processed through the inverse affine transformation matrix. Further, due to reasons such as the ratio mismatch between the grid coordinate system in the sampling grid and the coordinate system in the image to be processed, and the rotation angle between the detection frame and the image to be processed, the coordinate values of the grid coordinates in the sampling grid may not be integers after being converted by the inverse affine transformation matrix. So, in this embodiment, the grid coordinates in the grid coordinate system can be calculated to obtain a floating - point value coordinate according to the inverse affine transformation matrix, so as to facilitate subsequent calculations directly using the floating - point value coordinates.
[0069] S233: In the image to be processed, obtain several integer - valued coordinates closest to the floating - point value coordinates, and the pixel values at the positions of the several integer - valued coordinates; S234: Calculate the pixel value at the location of the floating-point value based on the pixel values at the locations of the coordinates of the plurality of integer values; In one implementation, the pixel values at the locations of the plurality of integer coordinates can be weighted to obtain the pixel value at the location of the floating-point coordinates. The weight of the integer coordinates can be set according to the distance between the integer coordinates and the floating-point coordinates.
[0070] Preferably, the plurality of integer coordinate values may include four integer coordinate values, and the pixel values at the locations of the four integer coordinate values can be used to perform bilinear interpolation calculations on the floating-point coordinate values.
[0071] S235: Use the pixel value at the location of the floating-point coordinates as the pixel value of the grid coordinates to generate the inverse affine image.
[0072] Understandably, by calculating the pixel value of the floating-point coordinates corresponding to the grid coordinates, the pixel value is mapped back to the grid coordinates of the sampling grid, so that the grid coordinates can be used as a pixel, and finally the inverse affine image is generated based on the sampling grid.
[0073] In this embodiment, by employing inverse affine sampling, geometric correction is performed on the region of the detection box on the image to be processed, normalizing the target with arbitrary orientation to a standard posture, effectively eliminating the effects of deformation such as rotation and scaling, reducing the impact of image jitter, and helping to improve the stability and consistency of subsequent feature extraction.
[0074] In one embodiment, the processing of the image to be processed can be performed in the computer's video memory, thereby enabling the bilinear interpolation process in step S234 and the inverse affine sampling process in step S235 to be executed in the computer's video memory, avoiding decoding in the computer's central processing unit, effectively improving the efficiency of inverse affine sampling, and preserving the accuracy of the image to be processed.
[0075] S3: Add position information to the inverse affine image based on the image to be processed to obtain the first image; In this embodiment, as Figure 3 As shown, step S3 may include the following sub-steps: S31: Obtain the coordinate values of the pixels in the inverse affine image in the image to be processed; In this embodiment, the pixels in the inverse affine image correspond to the grid coordinates in the sampling grid. Therefore, the grid coordinates corresponding to the pixels in the inverse affine image can be obtained, and then the coordinate values corresponding to them in the image to be processed can be calculated through the inverse affine transformation matrix.
[0076] S32: Construct the coordinate value matrix of the inverse affine image based on the corresponding coordinate values; Since the coordinate values include horizontal coordinates, in this embodiment, the coordinate matrix of the constructed inverse affine image can include a horizontal coordinate matrix and a vertical coordinate matrix. It can be understood that the elements in the horizontal coordinate matrix record the horizontal coordinate values of the corresponding coordinates, and the elements in the vertical coordinate matrix record the vertical coordinate values of the corresponding coordinates.
[0077] For example, if the pixel (x1, y1) in the inverse affine image has coordinates (x2, y2) in the image to be processed, then the value of the element (x1, y1) in the horizontal coordinate matrix is x2, and the value of the element (x1, y1) in the vertical coordinate matrix is y2.
[0078] By dividing the coordinate value matrix into the horizontal coordinate value matrix and the vertical coordinate value matrix, the spatial semantic information of the horizontal and vertical directions in the inverse affine image can be fully expressed. At the same time, the vertical and horizontal coordinates can be added to the inverse affine image through different and independent channels, so that the vertical and horizontal coordinates can be perceived simultaneously during the feature extraction process, thereby realizing the perception of position information.
[0079] S33: Based on the coordinate value matrix, add position information to the inverse affine image to obtain the first image.
[0080] In this embodiment, step S33 may include the following sub-steps: The coordinate value matrix is normalized to obtain the coordinate normalization matrix of the inverse affine image; Based on the coordinate normalization matrix, positional information is added to the inverse affine image to obtain the first image.
[0081] In this embodiment, by normalizing the coordinate value matrix according to a unified standard, the coordinate information contained in the coordinate value matrix is mapped to a unified numerical range, eliminating the order-of-magnitude changes caused by differences in the detection box scale, and providing stable positional information for subsequent feature extraction.
[0082] As described above, in this embodiment, the coordinate value matrix includes a horizontal coordinate value matrix and a vertical coordinate value matrix. Therefore, the coordinate normalization matrix may include a horizontal coordinate normalization matrix and a vertical coordinate normalization matrix.
[0083] In this embodiment, step S331 may include the following sub-steps: The width and height of the inverse affine image are calculated based on the coordinate parameters of the detection box; In this embodiment, the inverse affine image is generated based on the sampling grid of the expanded detection box; therefore, the calculation of the dimensions (width and height) of the inverse affine image may include: First, the coordinate parameters of the expanded detection box are calculated based on the coordinate parameters of the original detection box. Then, the width and height of the inverse affine image are calculated based on the coordinate parameters of the expanded detection box. It is understood that the width and height of the inverse affine image match the width and height of the detection box.
[0084] After calculating the width and height of the inverse affine image, the horizontal coordinate matrix can be normalized according to the width of the inverse affine image to obtain a normalized horizontal coordinate matrix; and the vertical coordinate matrix can be normalized according to the height of the inverse affine image to obtain a normalized vertical coordinate matrix.
[0085] In this embodiment, the horizontal and vertical coordinate matrices are normalized based on the size of the inverse affine image, so that all normalized horizontal coordinate values are distributed within [0,1], and all normalized vertical coordinate values are distributed within [0,1]. Furthermore, the maximum normalized horizontal coordinate value is 1, and the maximum normalized vertical coordinate value is 1. This aligns the positional information contained in the normalized horizontal and vertical coordinate matrices with the spatial scale of the inverse affine image, effectively providing positional encoding for better identification of positional information during subsequent feature extraction.
[0086] In one implementation, two new channels of the same size as the inverse affine image can be added in the channel dimension. One channel stores the x-coordinate values in the x-coordinate normalization matrix, and the other channel stores the y-coordinate values in the y-coordinate normalization matrix. Therefore, adding position information to the inverse affine image based on the coordinate normalization matrix to obtain the first image can be achieved by filling the x-coordinate values of each element in the x-coordinate normalization matrix into one channel, and filling the y-coordinate values of each element in the y-coordinate normalization matrix into the other channel, thus obtaining the first image.
[0087] In another implementation, the step of adding position information to the inverse affine image based on the coordinate normalization matrix to obtain the first image can be achieved by using the horizontal coordinate values in the horizontal coordinate normalization matrix as pixel values to construct a horizontal coordinate normalized image, and using the vertical coordinate values in the vertical coordinate normalization matrix as pixel values to construct a vertical coordinate normalized image. The horizontal coordinate normalized image and the vertical coordinate normalized image are then concatenated with the inverse affine image based on the channel dimension, respectively, to obtain the first image.
[0088] Understandably, by adding position information to the inverse affine image, the resulting first image contains not only the original three RGB color channels of the inverse affine image, but also two position information channels. Furthermore, through the above processing, the object to be processed contained in the inverse affine image can be located at the center of the detection box. This allows the subsequent feature extraction process to focus on processing the image at the center based on the position information contained in the first image, suppressing noise in edge regions, enhancing tolerance to detection box positioning errors, and effectively avoiding feature distortion caused by differences between consecutive images, thereby achieving better effective feature extraction from the image.
[0089] S4: Resample the first image to obtain the second image; In a preferred embodiment of this example, the resampling process of the first image can be performed by downsampling the first image. Therefore, the resolution of the second image obtained by downsampling the first image is lower than that of the first image. Downsampling can preserve all features of the first image while avoiding the introduction of additional noise, thus providing a basis for subsequent processing.
[0090] S5: Extract the features of the first image and the features of the second image, and perform cross-attention fusion on the features of the first image and the features of the second image to obtain attention fusion features as the image features of the image to be processed.
[0091] In this embodiment, a local feature extraction network can be used to extract features from the first image, and a global feature extraction network can be used to extract features from the second image. Furthermore, the local feature extraction network can be a feature extraction network with fewer layers, such as a shallow convolutional network. Shallow convolutional networks have fewer layers, fewer downsampling operations, and a smaller overall receptive field, thus better preserving the high-frequency features of the first image and better distinguishing similar features. The global feature extraction network can be a feature extraction network with more stacked Transformer encoders, such as a deep Transformer network. Because it contains more Transformer encoders, the deep Transformer network has powerful global context modeling capabilities and can effectively extract low-frequency features from the second image.
[0092] In this embodiment, step S5, which involves performing cross-attention fusion on the features of the first image and the features of the second image to obtain attention fusion features as the image features of the image to be processed, may include the following steps: The features of the second image are used as the query vector, and the features of the first image are used as the key vector and value vector for cross-attention fusion to obtain attention fusion features as the image features of the image to be processed.
[0093] Understandably, by using the features of the low-resolution second image as the query vector and the features of the high-resolution first image as the key and value vectors for cross-attention fusion, the fusion process is dominated by low-frequency, global semantics, thereby selectively extracting useful information consistent with the overall structure from high-frequency details, thus improving the robustness and consistency of the features.
[0094] like Figure 7 As shown in the illustration, this application also provides an image feature extraction device. Optionally, the image feature extraction device may include: The coordinate acquisition module 11 is used to acquire the coordinate parameters of the detection box in the image to be processed; the detection box is generated based on the object to be processed in the image to be processed. In this embodiment, the coordinate acquisition module 11 can be used to perform... Figure 2 For a detailed description of the coordinate acquisition module 11 shown in step S1, please refer to the description of step S1.
[0095] The inverse affine processing module 12 is used to perform inverse affine sampling processing on the image to be processed according to the coordinate parameters of the detection box to obtain an inverse affine image; In this embodiment, the inverse affine processing module 12 can be used to perform... Figure 2For a detailed description of the inverse affine processing module 12 shown in step S2, please refer to the description of step S2.
[0096] The location information adding module 13 is used to add location information to the inverse affine image based on the image to be processed, so as to obtain a first image; In this embodiment, the location information adding module 13 can be used to perform... Figure 2 For a detailed description of the location information adding module 13 shown in step S3, please refer to the description of step S3.
[0097] The resampling module 14 is used to resample the first image to obtain the second image; In this embodiment, the resampling module 14 can be used to perform... Figure 2 For a detailed description of the resampling module 14 shown in step S4, please refer to the description of step S4.
[0098] Feature extraction module 15 is used to extract features of the first image and features of the second image, and to perform cross-attention fusion on the features of the first image and features of the second image to obtain attention fusion features as image features of the image to be processed; In this embodiment, the feature extraction module 15 can be used to perform... Figure 2 For a detailed description of the feature extraction module 15 shown in step S5, please refer to the description of step S5.
[0099] It is understood that the above-described device embodiments and method embodiments can correspond to each other, and similar descriptions of the device embodiments can be referred to the method embodiments. To avoid repetition, further details are omitted here. The image feature extraction device provided in this application can execute an image feature extraction method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. The functional modules of the image feature extraction device can be implemented in hardware, in software instructions, or in a combination of hardware and software modules.
[0100] Specifically, the steps of the method embodiments of this application can be implemented by integrated logic circuits in the processor hardware and / or instructions in software form. The steps of the image feature extraction method in conjunction with the embodiments of this application can be directly implemented by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in random access memory, such as read-only memory, programmable read-only memory, flash memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.
[0101] This application provides an electronic device with the following structure: Figure 8 As shown. The electronic device can be as described in this embodiment. Figure 1 The server 100 or terminal 200 shown.
[0102] The electronic device includes a memory 21, a processor 22, a communication module 23, and an input / output interface 24, etc. Optionally, the memory 21, the processor 22, the communication module 23, and the input / output interface 24 can be connected and communicate with each other through a bus 25.
[0103] The memory 21 is used to store one or more computer programs and to transfer the code of the computer programs to the processor 22; when the one or more computer programs are executed by the processor 22, the image feature extraction method in this embodiment of the application is implemented.
[0104] Optionally, the electronic device can be connected to a network via communication module 23 to communicate with other devices, such as terminals or servers, to achieve data interaction. The electronic device can be various forms of digital computers, exemplarily such as desktop computers, servers, workbenches, mainframes, or other types of computers. The electronic device can also be various forms of mobile terminals, exemplarily such as smartphones, tablets, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.
[0105] Optionally, the electronic device can connect to required input / output devices, such as a keyboard or display device, via the input / output interface 24. The electronic device itself may have a display device, and other display devices can also be connected externally via the input / output interface 24. Optionally, a storage device, such as a hard disk, can also be connected via the input / output interface 24 to store data from the electronic device, read data from the storage device, or store data from the storage device in the memory 21. It is understood that the input / output interface 24 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 24 can be a component of the electronic device or an external device connected to the electronic device when needed.
[0106] Optionally, the memory 21 may be a volatile memory and / or a non-volatile memory. The volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.
[0107] Optionally, the computer program stored in the processor 22 can be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to perform the method provided in this embodiment. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.
[0108] Optionally, the processor 22 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 22 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and can also be any suitable controller, microcontroller, processor, etc. The processor 22 executes the various methods and processes of this embodiment, exemplarily, such as an image feature extraction method according to an embodiment of this application.
[0109] Optionally, the bus 25 may include a path for transmitting information. Depending on its function, the bus 25 may be divided into an address bus, a data bus, a control bus, etc.
[0110] In an optional implementation, this application embodiment also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods described in the above-described method embodiments. Part or all of the computer program can be loaded and / or installed on the memory 21 of an electronic device. When the computer program is executed by the processor 22, one or more steps of an image feature extraction method according to this application embodiment can be performed.
[0111] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.
[0112] Obviously, the above embodiments of this application are merely examples for clearly illustrating the technical solution of this application, and are not intended to limit the specific implementation of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of this application should be included within the protection scope of the claims of this application.
Claims
1. An image feature extraction method, characterized in that, The method includes: Obtain the coordinate parameters of the detection box in the image to be processed; the detection box is generated based on the object to be processed in the image to be processed. Based on the coordinate parameters of the detection box, inverse affine sampling is performed on the image to be processed to obtain an inverse affine image; The inverse affine image is obtained by adding position information to the image to be processed; The first image is resampled to obtain the second image; Extract features from the first image and the second image, and perform cross-attention fusion on the features of the first image and the second image to obtain attention fusion features as image features of the image to be processed.
2. The image feature extraction method according to claim 1, characterized in that, The step of adding position information to the inverse affine image based on the image to be processed to obtain a first image includes: Obtain the coordinate values of the pixels in the inverse affine image in the image to be processed; Based on the corresponding coordinate values, construct the coordinate value matrix of the inverse affine image; Based on the coordinate value matrix, position information is added to the inverse affine image to obtain the first image.
3. The image feature extraction method according to claim 2, characterized in that, The step of adding position information to the inverse affine image based on the coordinate value matrix to obtain the first image includes: The coordinate value matrix is normalized to obtain the coordinate normalization matrix of the inverse affine image; Based on the coordinate normalization matrix, positional information is added to the inverse affine image to obtain the first image.
4. The image feature extraction method according to claim 3, characterized in that, The coordinate value matrix includes an abscissa value matrix and a ordinate value matrix; the coordinate normalization matrix includes an abscissa normalization matrix and a ordinate normalization matrix. The normalization process of the coordinate value matrix to obtain the coordinate normalization matrix of the inverse affine image includes: The width and height of the inverse affine image are calculated based on the coordinate parameters of the detection box; Based on the width of the inverse affine image, the horizontal coordinate value matrix is normalized to obtain the horizontal coordinate normalized matrix. Based on the height of the inverse affine image, the ordinate value matrix is normalized to obtain the ordinate normalized matrix.
5. The image feature extraction method according to claim 1, characterized in that, The resampling is downsampling; The step of performing cross-attention fusion on the features of the first image and the features of the second image to obtain attention fusion features as the image features of the image to be processed includes: The features of the second image are used as the query vector, and the features of the first image are used as the key vector and value vector for cross-attention fusion to obtain attention fusion features as the image features of the image to be processed.
6. An image feature extraction method according to any one of claims 1-5, characterized in that, The step of performing inverse affine sampling processing on the image to be processed based on the coordinate parameters of the detection box to obtain an inverse affine image includes: The detection box is expanded to obtain an expanded detection box, and the coordinate parameters of the expanded detection box in the image to be processed are obtained. A sampling grid is constructed based on the expanded detection box, and the inverse affine transformation matrix from the sampling grid to the image to be processed is calculated based on the coordinate parameters of the expanded detection box. Based on the inverse affine transformation matrix and the sampling grid, the image to be processed is subjected to inverse affine sampling to obtain the inverse affine image.
7. The image feature extraction method according to claim 6, characterized in that, The step of expanding the detection box to obtain an expanded detection box and obtaining the coordinate parameters of the expanded detection box in the image to be processed includes: The geometric center coordinates of the detection box, the rotation angle of the detection box relative to the image to be processed, and the size of the detection box are calculated based on the coordinate parameters of the detection box. The size of the detection frame is expanded to obtain the expanded detection frame size, and the expanded detection frame is obtained based on the expanded detection frame size; The coordinate parameters of the expanded detection frame are calculated based on the geometric center coordinates, the rotation angle, and the expanded detection frame size.
8. The image feature extraction method according to claim 6, characterized in that, The step of performing inverse affine sampling processing on the image to be processed according to the inverse affine transformation matrix and the sampling grid to obtain the inverse affine image includes: Based on the grid structure of the sampling grid, construct the grid coordinate system of the sampling grid; For the grid coordinates of the sampling grid in the grid coordinate system, calculate its floating-point coordinates in the image to be processed according to the inverse affine transformation matrix; In the image to be processed, obtain several integer coordinates that are closest to the floating-point coordinates, and the pixel values at the locations of the several integer coordinates; Calculate the pixel value at the location of the floating-point coordinates based on the pixel values at the locations of the integer coordinates. The inverse affine image is generated by using the pixel value at the location of the floating-point coordinates as the pixel value at the grid coordinates.
9. An image feature extraction device, characterized in that, The device includes: A coordinate acquisition module is used to acquire the coordinate parameters of the detection box in the image to be processed; the detection box is generated based on the object to be processed in the image to be processed. The inverse affine processing module is used to perform inverse affine sampling processing on the image to be processed according to the coordinate parameters of the detection box to obtain an inverse affine image; A location information adding module is used to add location information to the inverse affine image based on the image to be processed, so as to obtain a first image; A resampling module is used to resample the first image to obtain a second image; The feature extraction module is used to extract features from the first image and features from the second image, and to perform cross-attention fusion on the features of the first image and features of the second image to obtain attention fusion features as image features of the image to be processed.
10. An electronic device, characterized in that, include: Memory, used to store one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements an image feature extraction method as described in any one of claims 1-8.