Image feature fusion method and device, electronic equipment and storage medium

Abstract

The invention relates to an image feature fusion method and device, electronic equipment and a storage medium. A high-resolution feature and a low-resolution feature of a to-be-recognized image are obtained, the high-resolution feature and the low-resolution feature are coded respectively, a first feature of the high-resolution feature and a second feature of the low-resolution feature are generated, the first feature comprises a first sub-feature and a second sub-feature, and the first sub-feature and the second sub-feature are subjected to image recognition based on a plurality of preset transformer modules. Performing feature extraction on the first sub-feature and the second feature, respectively generating a component of the first feature, sampling the component of the second feature by the component of the second feature, performing normalization processing on the sampled features, generating a normalized feature, fusing the component of the first feature and the normalized feature, and performing residual connection on a fusion result to obtain a fusion result; and the fusion feature is calculated based on the residual connection result, so that the high-resolution feature and the low-resolution feature of the image can be fused, and the image recognition precision is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image feature fusion method, device, electronic device and storage medium. Background Art

[0002] In semantic segmentation, depth estimation or other intensive prediction tasks, it is often necessary to fuse features of different resolutions. For features of different resolutions, the related technology upsamples the low-resolution features and then directly adds them to the high-resolution features, or directly splices them in the channel dimension. This method ignores the rich detail information contained in the two different high-resolution features and the rich semantic information contained in the low-resolution features; or uses convolution to design a fusion module. This method generally downsamples the high-resolution features and fuses them with the low-resolution features, and also upsamples the low-resolution features and fuses them with the high-resolution features.

[0003] However, simple concatenation or addition usually ignores the difference between low-resolution features and high-resolution features, and convolution-based fusion methods ignore the correlation between features, affecting the recognition accuracy. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides an image feature fusion method, device, electronic device and storage medium, which can fuse the high-resolution features and low-resolution features of the image to be identified to improve the accuracy of image recognition.

[0005] The first aspect of the present application provides an image feature fusion method, comprising: Obtain high-resolution features and low-resolution features of the image to be identified; Encoding the high-resolution feature and the low-resolution feature respectively to generate a first feature of the high-resolution feature and a second feature of the low-resolution feature; the first feature includes: a first sub-feature and a second sub-feature; Based on a plurality of preset transformer modules, feature extraction is performed on the first sub-feature, the second sub-feature, and the second feature to generate a component of the first feature and a component of the second feature respectively; Sampling components of the second feature, and normalizing the sampled features to generate normalized features; Fusing the components of the first feature with the normalized feature, and performing a residual connection on the fusion result; A fusion feature is calculated based on the result of the residual connection, and the fusion feature is used to identify the image to be identified.

[0006] Preferably, encoding the high-resolution feature and the low-resolution feature respectively to generate a first feature of the high-resolution feature and a second feature of the low-resolution feature comprises: Based on the first step of the convolution module, the high-resolution feature and the low-resolution feature are encoded respectively to generate the first sub-feature and the second feature respectively; Based on a convolution module with a second step length, the high-resolution feature is encoded to generate the second sub-feature. Preferably, the components of the first feature include the first component of the first feature and the second component of the first feature, and the components of the second feature include the first component of the second feature and the second component of the second feature; and the extracting of the first feature and the second feature based on the plurality of preset transformer modules to generate the components of the first feature and the components of the second feature respectively includes: Based on the first encoding module in the preset transformer module, feature extraction is performed on the first sub-feature, the second sub-feature, and the second feature to generate a first component of the first feature, a second component of the first feature, and a second component of the second feature respectively; Based on the second encoding module in the preset transformer, feature extraction is performed on the second feature to generate a first component of the second feature.

[0007] Preferably, the normalized feature includes: a first intermediate component of the second feature and a second intermediate component of the second feature; and the sampling of the components of the second feature and normalizing the sampled features to generate the normalized feature includes: Sampling the first component of the second feature to generate an intermediate component of the second feature; The second component of the second feature and the intermediate component of the second feature are respectively normalized to generate a second intermediate component of the second feature and a first intermediate component of the second feature.

[0008] Preferably, the fusing the components of the first feature with the normalized feature and performing residual connection on the fusion result comprises: Fusing the first component of the first feature with the first intermediate component of the second feature to generate a first fused feature of the first feature; Fusing the second component of the first feature with the second intermediate component of the second feature to generate a first fused feature of the second feature; A first fused feature of the first feature and the first sub-feature are combined to generate a second fused feature of the first feature; The first fused feature of the second feature is added to the second feature to generate a second fused feature of the second feature.

[0009] Preferably, the steps before calculating the fusion feature based on the result of the residual connection include: The second fused feature of the second feature is sampled to generate a third fused feature of the second feature.

[0010] Preferably, the calculation of fusion features based on the result of residual connection includes: The third fused feature of the second feature is added to the second fused feature of the first feature to generate the fused feature.

[0011] A second aspect of the present application provides an image feature fusion device, comprising: An acquisition module, used for acquiring high-resolution features and low-resolution features of an image to be identified; An encoding module, used for encoding the high-resolution feature and the low-resolution feature respectively, to generate a first feature of the high-resolution feature and a second feature of the low-resolution feature; the first feature includes: a first sub-feature and a second sub-feature; An extraction module, used for performing feature extraction on the first sub-feature, the second sub-feature, and the second feature based on a plurality of preset transformer modules, and generating components of the first feature and components of the second feature respectively; A sampling module, used for sampling the components of the second feature, normalizing the sampled features, and generating normalized features; A fusion module, used for fusing the first feature with the normalized feature, and performing a residual connection on the fusion result; The calculation module is used to calculate fusion features based on the result of residual connection, and the fusion features are used to identify the image to be identified.

[0012] A third aspect of the present application provides an electronic device, including: Processor; and The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0013] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0014] The technical solution provided by the present application may include the following beneficial effects: by acquiring high-resolution features and low-resolution features of an image to be identified, encoding the high-resolution features and the low-resolution features respectively, generating a first feature of the high-resolution feature and a second feature of the low-resolution feature, the first feature including: a first sub-feature and a second sub-feature, based on multiple preset transformer modules, extracting features of the first sub-feature, the second sub-feature, and the second feature, generating components of the first feature and components of the second feature respectively, sampling the components of the second feature, normalizing the sampled features to generate normalized features, fusing the components of the first feature with the normalized features, performing residual connection on the fusion result, calculating the fusion feature based on the result of the residual connection, the fusion feature is used to identify the image to be identified, and can fuse the high-resolution features and low-resolution features of the image to be identified, thereby improving the accuracy of image recognition.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0017] Figure 1 is a flow chart of an image feature fusion method shown in an embodiment of the present application; Figure 2 is another flow chart of the image feature fusion method shown in the embodiment of the present application; Figure 3 is a flow chart of the image feature fusion method shown in an embodiment of the present application; Figure 4 is a structural schematic diagram of an image feature fusion device shown in an embodiment of the present application; Figure 5 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0019] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0021] In semantic segmentation, depth estimation or other intensive prediction tasks, it is often necessary to fuse features of different resolutions. For features of different resolutions, the relevant technology upsamples the low-resolution features and then directly adds them to the high-resolution features, or directly splices them in the channel dimension. Simple splicing or addition usually ignores the difference between low-resolution features and high-resolution features. The convolution-based fusion method ignores the correlation between the features, affecting the recognition accuracy.

[0022] It is understandable that the high resolution and low resolution in this application are relative concepts, and different resolution features include high resolution features with relatively high resolution and low resolution features with relatively low resolution. The value of the resolution itself is not limited.

[0023] In response to the above problems, an embodiment of the present application provides an image feature fusion method, which can fuse high-resolution features with low-resolution features to improve the accuracy of image recognition.

[0024] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0025] Figure 1 It is a flowchart of the image feature fusion method shown in the embodiment of the present application.

[0026] See also Figure 1 , the method comprising: Step 101, obtaining high-resolution features and low-resolution features of an image to be identified; For the image to be identified, feature extraction is first performed to obtain high-resolution features and low-resolution features of the image to be identified. The image to be identified can be a street view image collected during the autonomous driving of an intelligent vehicle that requires semantic segmentation, or a depth estimation image that requires obstacle detection and distance estimation during the autonomous driving of an intelligent vehicle.

[0027] Step 102, respectively encode the high-resolution feature and the low-resolution feature to generate a first feature of the high-resolution feature and a second feature of the low-resolution feature; the first feature includes: a first sub-feature and a second sub-feature; The high-resolution feature may be encoded to obtain a first feature, wherein the first feature may include a first sub-feature (left1) and a second sub-feature (left2); the second-resolution feature may be encoded to obtain a second feature (right).

[0028] Step 103, based on a plurality of preset transformer modules, feature extraction is performed on the first sub-feature, the second sub-feature, and the second feature to generate components of the first feature and components of the second feature respectively; For the first sub-feature (left1), the second sub-feature (left2), and the second feature (right) of the first feature, multiple preset transformer modules can be used to perform feature extraction respectively to obtain the components of the first feature and the components of the second feature. It can be understood that the number of components of the first feature and the number of components of the second feature can be multiple.

[0029] Step 104, sampling the components of the second feature, and normalizing the sampled features to generate normalized features; For the components of the second feature, sampling operations are performed separately, and then the sampled features are normalized to generate normalized features. Normalizing the features can increase the nonlinearity of the features on the one hand, and normalize the elements in the features to the interval [-1, 1] on the other hand to avoid overflow of subsequent operations on the features.

[0030] Step 105, fusing the components of the first feature with the normalized feature, and performing residual connection on the fusion result; The components of the first feature are multiplied by the normalized features respectively, so as to fuse the components of the first feature and the normalized features. Then, the residual connection is realized by adding the fusion results to the components of the first feature and the normalized features respectively, so as to avoid the phenomenon that the normalization function may cause the gradient to disappear and the network parameters are not updated.

[0031] Step 106, calculating fusion features based on the result of residual connection, and the fusion features are used to identify the image to be identified.

[0032] The results of the residual connection are added and fused to generate fused features of high-resolution features and low-resolution features. The fused features can be used to identify the image to be identified. Using the fused features to identify the image can improve the accuracy of recognition.

[0033] The embodiment of the present application discloses an image feature fusion method, which obtains high-resolution features and low-resolution features of an image to be identified, encodes the high-resolution features and the low-resolution features respectively, generates a first feature of the high-resolution feature and a second feature of the low-resolution feature, the first feature includes: a first sub-feature, a second sub-feature, based on multiple preset transformer modules, extracts features of the first sub-feature, the second sub-feature, and the second feature, generates components of the first feature and components of the second feature respectively, samples the components of the second feature, normalizes the sampled features to generate normalized features, fuses the components of the first feature with the normalized features, performs residual connection on the fusion result, calculates the fusion feature based on the result of the residual connection, and uses the fusion feature to identify the image to be identified, and can fuse the high-resolution features and the low-resolution features of the image to be identified, thereby improving the accuracy of image recognition.

[0034] Figure 2 It is another flowchart of the image feature fusion method shown in an embodiment of the present application.

[0035] See also Figure 2 , the method comprising: Step 201, obtaining high-resolution features and low-resolution features of an image to be identified; For the image to be identified, feature extraction is first performed to obtain high-resolution features and low-resolution features of the image to be identified. The image to be identified can be a street view image collected during the autonomous driving of an intelligent vehicle that requires semantic segmentation, or a depth estimation image that requires obstacle detection and distance estimation during the autonomous driving of an intelligent vehicle.

[0036] Step 202, based on the first step of the convolution module, respectively encode the high-resolution feature and the low-resolution feature to generate a first sub-feature and a second feature; First, a convolution module with the first step length is used to encode the high-resolution features and low-resolution features respectively, generating the first sub-feature of the high-resolution feature (left1) and the second feature of the low-resolution feature (right). The first step length of the convolution module can be set to 1, the convolution kernel size can be 3×3, and the inner margin padding can be 1.

[0037] Step 203: Encode the high-resolution features based on the convolution module with the second step length to generate second sub-features.

[0038] Then, a convolution module with a second step length is used to encode the high-resolution feature to generate the second sub-feature (left2) of the high-resolution feature. The second step length of the convolution module can be set to 2, the convolution kernel size can be 3×3, and the inner margin padding can be 1.

[0039] Step 204, based on a plurality of preset transformer modules, feature extraction is performed on the first sub-feature, the second sub-feature, and the second feature to generate components of the first feature and components of the second feature respectively; For the first sub-feature (left1), the second sub-feature (left2) and the second feature (right) of the first feature, three preset transformer modules can be used to extract features respectively to obtain the components of the first feature and the components of the second feature. It can be understood that the number of components of the first feature and the number of components of the second feature can be multiple.

[0040] In an optional embodiment of the present application, the components of the first feature include: a first component of the first feature and a second component of the first feature, and the components of the second feature include: a first component of the second feature and a second component of the second feature; step 204 includes: Sub-step S11, based on the first encoding module in the preset transformer module, feature extraction is performed on the first sub-feature, the second sub-feature, and the second feature, respectively, to generate a first component of the first feature, a second component of the first feature, and a second component of the second feature; In the embodiment of the present application, the components of the first feature may include the first component of the first feature (left1_), the second component of the first feature (left2_), and the components of the second feature may include the first component of the second feature (right1_), the second component of the second feature (right2_). The first sub-feature (left1), the second sub-feature (left2), and the second feature may be respectively input into the first encoding module of the three preset transformer modules, and the first sub-feature (left1), the second sub-feature (left2), and the second feature may be respectively feature extracted to generate the first component of the first feature (left1_), the second component of the first feature (left2_), and the second component of the second feature (right2_).

[0041] Sub-step S12: extracting the second feature based on the second encoding module in the preset transformer to generate a first component of the second feature.

[0042] The second feature is then input into a second encoding module in a preset transformer, feature extraction is performed on the second feature, and the first component (right1_) of the second feature is generated.

[0043] Step 205, sampling the components of the second feature, normalizing the sampled features, and generating normalized features; For the components of the second feature, sampling operations are performed separately, and then the sampled features are normalized to generate normalized features. Normalizing the features can increase the nonlinearity of the features on the one hand, and normalize the elements in the features to the interval [-1, 1] on the other hand to avoid overflow of subsequent operations on the features.

[0044] In an optional embodiment of the present application, the normalized feature includes: a first intermediate component of the second feature and a second intermediate component of the second feature; step 205 includes: Sub-step S21, sampling the first component of the second feature to generate an intermediate component of the second feature; Bilinear interpolation, also known as bilinear interpolation. Mathematically, bilinear interpolation is an extension of linear interpolation with an interpolation function of two variables. The core idea is to perform linear interpolation in two directions. Deconvolution is an algorithm-based process that reverses the effect of convolution on recorded data and reconstructs unknown inputs by measuring outputs and known inputs.

[0045] In the embodiment of the present application, a bilinear interpolation or deconvolution method may be used to upsample the components of the second feature to generate an intermediate component (right1_1) of the second feature.

[0046] Sub-step S22, normalizing the second component of the second feature and the intermediate component of the second feature respectively to generate the second intermediate component of the second feature and the first intermediate component of the second feature.

[0047] The normalized features include the first intermediate component of the second feature (right1_2) and the second intermediate component of the second feature. After obtaining the intermediate component of the second feature, the sigmoid function (activation function) is used to normalize the second component of the second feature and the intermediate component of the second feature (right1_1), respectively, to obtain the second intermediate component of the second feature (right2_1) and the first intermediate component of the second feature (right1_2), respectively.

[0048] Step 206, fusing the components of the first feature with the normalized feature, and performing residual connection on the fusion result; The components of the first feature are fused with the normalized features respectively, and then the residual connection is realized by adding the fusion results to the components of the first feature and the normalized features respectively, so as to avoid the phenomenon that the normalization function may cause the gradient to disappear and the network parameters are not updated.

[0049] In an optional embodiment of the present application, step 206 includes: Sub-step S31, fusing the first component of the first feature with the first intermediate component of the second feature to generate a first fused feature of the first feature; The first component (left1_) of the first feature and the first intermediate component (right1_2) of the second feature may be multiplied or added to perform fusion to generate a first fused feature (left_o1) of the first feature.

[0050] Sub-step S32, fusing the second component of the first feature with the second intermediate component of the second feature to generate a first fused feature of the second feature; The second component (left2_) of the first feature and the second intermediate component (right2_1) of the second feature may be fused to generate a first fused feature (right_o1) of the second feature.

[0051] Sub-step S33, adding the first fused feature of the first feature to the first sub-feature to generate a second fused feature of the first feature; The first fused feature (left_o1) of the first feature is added to the first sub-feature (left1) to add a residual connection to generate the second fused feature (left_o2) of the first feature.

[0052] Sub-step S34, adding the first fused feature of the second feature to the second feature to generate a second fused feature of the second feature.

[0053] The first fused feature (right_o1) of the second feature is added to the second feature (right) to add a residual connection to generate the second fused feature (right_o2) of the second feature.

[0054] Step 207: Calculate fusion features based on the result of residual connection, and use the fusion features to identify the image to be identified.

[0055] The results of the residual connection are added and fused to generate high-resolution and low-resolution fusion features. The fusion features can be used to identify the image to be identified. Using the fusion features to identify the image can improve the accuracy of recognition.

[0056] In an optional embodiment of the present application, the steps before step 207 include: Step S1, sampling the second fused feature of the second feature to generate a third fused feature of the second feature.

[0057] Before fusion, the second fused feature (right_o2) of the second feature can be upsampled by bilinear interpolation or deconvolution to obtain the third fused feature (right_o3) of the second feature.

[0058] In an optional embodiment of the present application, step 207 includes: Sub-step S41, adding the third fused feature of the second feature to the second fused feature of the first feature to generate a fused feature.

[0059] The third fusion feature (right_o3) of the second feature is added to the second fusion feature (left_o2) of the first feature to generate a fusion feature. The fusion feature is the result of the fusion of the high-resolution feature and the low-resolution feature of the image to be identified. The fusion feature can be used to identify the image to be identified. Using the fusion feature to identify the image can improve the accuracy of recognition.

[0060] Figure 3 It is a flow chart of the image feature fusion method shown in an embodiment of the present application.

[0061] See also Figure 3 , the method comprising: Step 301, obtaining high-resolution features and low-resolution features of the image to be identified; Step 302, encoding the high-resolution feature and the low-resolution feature respectively to generate a first sub-feature (left1) and a second feature (right); First, a convolution module with the first step length is used to encode the high-resolution features and low-resolution features respectively, generating the first sub-feature (left1) of the high-resolution feature and the low-resolution feature of the low-resolution feature. The first step length of the convolution module can be set to 1, the convolution kernel size can be 3×3, and the inner margin padding can be 1.

[0062] Step 303, encode the high-resolution feature to generate the second sub-feature (left2); Then, a convolution module with a second step length is used to encode the high-resolution feature to generate the second sub-feature (left2) of the high-resolution feature. The second step length of the convolution module can be set to 2, the convolution kernel size can be 3×3, and the inner margin padding can be 1.

[0063] Step 304, input the first sub-feature (left1), the second sub-feature (left2), and the second feature (right) into three preset transformer modules; Step 305 , extract features from the first sub-feature (left1), the second sub-feature (left2), and the second feature (right), and generate a first component of the first feature (left1_), a second component of the first feature (left2_), and a second component of the second feature (right2_), respectively.

[0064] Step 306, input the second feature (right) into another preset transformer module; Step 307, extracting the second feature (right) to generate the first component of the second feature (right1_); Step 308, sampling the first component (right1_) of the second feature to generate an intermediate component (right1_1) of the second feature; The first component (right1_) of the second feature can be sampled by bilinear interpolation or deconvolution to generate the intermediate component (right1_1) of the second feature.

[0065] Step 309, normalizing the second component (right2_) of the second feature and the middle component (right1_1) of the second feature respectively to generate the second middle component (right2_1) of the second feature and the first middle component (right1_2) of the second feature; The sigmoid function can be used to normalize the second component (right2_) of the second feature and the middle component (right1_1) of the second feature, respectively, to obtain the second middle component (right2_1) of the second feature and the first middle component (right1_2) of the second feature, respectively.

[0066] Step 3010, fusing the first component (left1_) of the first feature with the first intermediate component (right1_2) of the second feature to generate a first fused feature (left_o1) of the first feature; Step 3011, fuse the second component (left2_) of the first feature with the second intermediate component (right2_1) of the second feature to generate a first fused feature (right_o1) of the second feature; Step 3012, adding the first fused feature (left_o1) of the first feature to the first sub-feature (left1) to generate a second fused feature (left_o2) of the first feature; Step 3013, adding the first fused feature (right_o1) of the second feature to the second feature (right) to generate a second fused feature (right_o2) of the second feature; Step 3014, sampling the second fused feature (right_o2) of the second feature to generate a third fused feature (right_o3) of the second feature; The second fused feature (right_o2) of the second feature may be sampled by using a bilinear interpolation or deconvolution method to obtain a third fused feature (right_o3) of the second feature.

[0067] Step 3015, add the third fused feature (right_o3) of the second feature to the second fused feature (left_o2) of the first feature to generate a fused feature.

[0068] The third fusion feature (right_o3) of the second feature is added to the second fusion feature (left_o2) of the first feature to generate a fusion feature. The fusion feature is the result of the fusion of the high-resolution feature and the low-resolution feature of the image to be identified. The fusion feature can be used to identify the image to be identified. Using fusion features to identify images can improve the accuracy of recognition.

[0069] The embodiment of the present application discloses an image feature fusion method, which obtains high-resolution features and low-resolution features of an image to be identified, encodes the high-resolution features and low-resolution features respectively based on a convolution module with a first step length, generates a first sub-feature and a low-resolution feature respectively, encodes the high-resolution features based on a convolution module with a second step length, generates a second sub-feature, extracts features from the first sub-feature, the second sub-feature, and the second feature based on multiple preset transformer modules, generates components of the first feature and components of the second feature respectively, samples the components of the second feature, normalizes the sampled features to generate normalized features, fuses the components of the first feature with the normalized features, performs residual connection on the fusion result, calculates the fusion feature based on the result of the residual connection, and uses the fusion feature to identify the image to be identified, and can fuse the high-resolution features and the low-resolution features of the image to be identified, thereby improving the accuracy of image recognition.

[0070] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides an image feature fusion device, an electronic device and corresponding embodiments.

[0071] Figure 4 It is a schematic diagram of the structure of the image feature fusion device shown in an embodiment of the present application.

[0072] See also Figure 4 , the device comprises: An acquisition module 401 is used to acquire high-resolution features and low-resolution features of an image to be recognized; The encoding module 402 is used to encode the high-resolution feature and the low-resolution feature respectively to generate a first feature of the high-resolution feature and a second feature of the low-resolution feature; the first feature includes: a first sub-feature and a second sub-feature; An extraction module 403, used to perform feature extraction on the first sub-feature, the second sub-feature, and the second feature based on a plurality of preset transformer modules, and generate a component of the first feature and a component of the second feature respectively; A sampling module 404 is used to sample the components of the second feature, and normalize the sampled features to generate normalized features; A fusion module 405 is used to fuse the first feature with the normalized feature and perform residual connection on the fusion result; The calculation module 406 is used to calculate fusion features based on the result of residual connection, and the fusion features are used to identify the image to be identified.

[0073] In an optional embodiment of the present application, the encoding module 402 includes: A first encoding submodule, used for encoding the high-resolution feature and the low-resolution feature based on the first step length convolution module, to generate a first sub-feature and a second feature respectively; The second encoding submodule is used to encode the high-resolution features based on the convolution module of the second step size to generate second sub-features. In an optional embodiment of the present application, the component of the first feature includes a first component of the first feature and a second component of the first feature, and the component of the second feature includes: a first component of the second feature and a second component of the second feature; the extraction module 403 includes: A first extraction submodule, used for extracting the first sub-feature, the second sub-feature, and the second feature based on the first encoding module in the preset transformer module, and generating a first component of the first feature, a second component of the first feature, and a second component of the second feature respectively; The second extraction submodule is used to extract the second feature based on the second encoding module in the preset transformer to generate the first component of the second feature.

[0074] In an optional embodiment of the present application, the normalized feature includes: a first intermediate component of the second feature and a second intermediate component of the second feature; the sampling module 404 includes: A first sampling submodule, used for sampling the first component of the second feature to generate an intermediate component of the second feature; The normalization submodule is used to normalize the second component of the second feature and the intermediate component of the second feature respectively to generate the second intermediate component of the second feature and the first intermediate component of the second feature.

[0075] In an optional embodiment of the present application, the fusion module 405 includes: A first fusion submodule, used for fusing the first component of the first feature with the first intermediate component of the second feature to generate a first fused feature of the first feature; A second fusion submodule, used for fusing the second component of the first feature with the second intermediate component of the second feature to generate a first fused feature of the second feature; A first connection submodule, used for adding a first fused feature of the first feature to the first sub-feature to generate a second fused feature of the first feature; The second connection submodule is used to add the first fused feature of the second feature to the second feature to generate a second fused feature of the second feature.

[0076] In an optional embodiment of the present application, the device further includes: The second sampling submodule is used to sample the second fused feature of the second feature to generate a third fused feature of the second feature).

[0077] In an optional embodiment of the present application, the calculation module 406 includes: The third connection submodule is used to add the third fused feature of the second feature to the second fused feature of the first feature to generate a fused feature.

[0078] The embodiment of the present application discloses an image feature fusion device, which obtains high-resolution features and low-resolution features of an image to be identified, encodes the high-resolution features and the low-resolution features respectively, generates a first feature of the high-resolution feature and a second feature of the low-resolution feature, the first feature includes: a first sub-feature, a second sub-feature, based on multiple preset transformer modules, extracts features of the first sub-feature, the second sub-feature, and the second feature, generates components of the first feature and components of the second feature respectively, samples the components of the second feature, normalizes the sampled features to generate normalized features, fuses the components of the first feature with the normalized features, performs residual connection on the fusion result, calculates the fusion feature based on the result of the residual connection, and uses the fusion feature to identify the image to be identified, and can fuse the high-resolution features and the low-resolution features of the image to be identified, thereby improving the accuracy of image recognition.

[0079] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0080] Figure 5 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0081] See also Figure 5 , the electronic device 500 includes a memory 510 and a processor 520 .

[0082] The processor 520 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0083] The memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 520 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at runtime. In addition, the memory 510 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 510 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0084] The memory 510 stores executable codes, and when the executable codes are processed by the processor 520 , the processor 520 can execute part or all of the methods described above.

[0085] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0086] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0087] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image feature fusion method, characterized in that: Obtain high-resolution features and low-resolution features of the image to be identified; Encoding the high-resolution feature and the low-resolution feature respectively to generate a first feature of the high-resolution feature and a second feature of the low-resolution feature; the first feature includes: a first sub-feature and a second sub-feature; Based on a plurality of preset transformer modules, feature extraction is performed on the first sub-feature, the second sub-feature, and the second feature to generate a component of the first feature and a component of the second feature respectively; Sampling components of the second feature, and normalizing the sampled features to generate normalized features; Fusing the components of the first feature with the normalized feature, and performing a residual connection on the fusion result; A fusion feature is calculated based on the result of the residual connection, and the fusion feature is used to identify the image to be identified.

2. The method according to claim 1, characterized in that The encoding of the high-resolution feature and the low-resolution feature respectively to generate a first feature of the high-resolution feature and a second feature of the low-resolution feature includes: Based on the first step of the convolution module, the high-resolution feature and the low-resolution feature are encoded respectively to generate the first sub-feature and the second feature respectively; Based on a convolution module with a second step length, the high-resolution feature is encoded to generate the second sub-feature.

3. The method according to claim 1, characterized in that The components of the first feature include the first component of the first feature and the second component of the first feature, and the components of the second feature include the first component of the second feature and the second component of the second feature; the feature extraction of the first feature and the second feature based on multiple preset transformer modules to generate the components of the first feature and the components of the second feature respectively includes: Based on the first encoding module in the preset transformer module, feature extraction is performed on the first sub-feature, the second sub-feature, and the second feature to generate a first component of the first feature, a second component of the first feature, and a second component of the second feature respectively; Based on the second encoding module in the preset transformer, feature extraction is performed on the second feature to generate a first component of the second feature.

4. The method according to claim 3, characterized in that The normalized feature includes: a first intermediate component of the second feature and a second intermediate component of the second feature; the sampling of the components of the second feature and normalizing the sampled features to generate the normalized feature includes: Sampling the first component of the second feature to generate an intermediate component of the second feature; The second component of the second feature and the intermediate component of the second feature are respectively normalized to generate a second intermediate component of the second feature and a first intermediate component of the second feature.

5. The method according to claim 4, characterized in that The fusing the component of the first feature with the normalized feature and performing residual connection on the fusion result includes: Fusing the first component of the first feature with the first intermediate component of the second feature to generate a first fused feature of the first feature; Fusing the second component of the first feature with the second intermediate component of the second feature to generate a first fused feature of the second feature; Adding a first fused feature of the first feature to the first sub-feature to generate a second fused feature of the first feature; The first fused feature of the second feature is added to the second feature to generate a second fused feature of the second feature.

6. The method according to claim 5, characterized in that The steps before calculating the fusion feature based on the result of the residual connection include: The second fused feature of the second feature is sampled to generate a third fused feature of the second feature.

7. The method according to claim 6, characterized in that The calculation of fusion features based on the result of residual connection includes: The third fused feature of the second feature is added to the second fused feature of the first feature to generate the fused feature.

8. An image feature fusion device, characterized in that: An acquisition module, used for acquiring high-resolution features and low-resolution features of an image to be identified; An encoding module, used for encoding the high-resolution feature and the low-resolution feature respectively, to generate a first feature of the high-resolution feature and a second feature of the low-resolution feature; the first feature includes: a first sub-feature and a second sub-feature; An extraction module, used for performing feature extraction on the first sub-feature, the second sub-feature, and the second feature based on a plurality of preset transformer modules, and generating components of the first feature and components of the second feature respectively; A sampling module, used for sampling the components of the second feature, normalizing the sampled features, and generating normalized features; A fusion module, used for fusing the first feature with the normalized feature, and performing a residual connection on the fusion result; The calculation module is used to calculate fusion features based on the result of residual connection, and the fusion features are used to identify the image to be identified.

9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 7.

Images