Image feature fusion and model training method and device, equipment and storage medium

Abstract

This disclosure provides an image feature fusion and model training method, apparatus, device, and storage medium, relating to the field of image processing technology, particularly computer vision and deep learning. The specific implementation scheme is as follows: Image features with different receptive fields are acquired; channel information enhancement is performed on high-level image features from different receptive fields using low-level image features, resulting in enhanced high-level features with enhanced channel information; wherein the receptive field of the high-level image features is larger than that of the low-level image features; spatial information enhancement is performed on low-level image features from different receptive fields using high-level image features, resulting in enhanced low-level features with enhanced spatial information; the enhanced high-level features and enhanced low-level features are fused to obtain the fused features. This disclosure enables better image feature fusion and improves the accuracy of image features.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to the fields of computer vision and deep learning. Background Technology

[0002] In many fields, fusing features from different receptive fields is an important means of improving performance. For example, in image processing, fusing images with different receptive fields is an important method to improve image processing results. Summary of the Invention

[0003] This disclosure provides an image feature fusion and model training method, apparatus, device, and storage medium.

[0004] According to a first aspect of this disclosure, an image feature fusion method is provided, comprising:

[0005] Acquire image features from different receptive fields;

[0006] By utilizing the low-level image features in the image features with different receptive fields, channel information enhancement is performed on the high-level image features in the image features with different receptive fields to obtain enhanced high-level features after channel information enhancement; wherein, the receptive field of the high-level image features is larger than that of the low-level image features.

[0007] By utilizing the high-level image features in the image features of different receptive fields, spatial information enhancement is performed on the low-level image features in the image features of different receptive fields to obtain enhanced low-level features with spatial information enhancement.

[0008] The enhanced high-level features and enhanced low-level features are fused to obtain the fused features.

[0009] According to a second aspect of this disclosure, a model training method for image feature fusion is provided, comprising:

[0010] Obtain multiple sample images and the corresponding labels for each sample image;

[0011] For each sample image, the sample image is input into a model for image feature fusion to obtain the fused features;

[0012] Image processing is performed based on the fused features to obtain the image processing result;

[0013] Calculate the difference between the image processing result and the label corresponding to the sample image;

[0014] The model parameters of the model used for image feature fusion are adjusted based on the differences.

[0015] Based on the adjusted model parameters and the multiple sample images, the model parameter adjustment process continues until the preset iteration termination condition is met.

[0016] The model parameters obtained when the preset iteration termination condition is met are used as the trained model parameters, and the model for image feature fusion including the trained model parameters is used as the trained model for image feature fusion.

[0017] The model for image feature fusion includes a channel attention network, a spatial attention network, and a fusion network.

[0018] The channel attention network is used to enhance the channel information of the high-level image features in the image features of different receptive fields by utilizing the low-level image features in the image features of different receptive fields, thereby obtaining enhanced high-level features after channel information enhancement; wherein, the receptive field of the high-level image features is larger than the receptive field of the low-level image features.

[0019] The spatial attention network is used to enhance the spatial information of the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields, thereby obtaining enhanced low-level features with enhanced spatial information.

[0020] The fusion network is used to fuse the enhanced high-level features and the enhanced low-level features to obtain the fused features.

[0021] According to a third aspect of this disclosure, an image feature fusion apparatus is provided, comprising:

[0022] The acquisition module is used to acquire image features from different receptive fields;

[0023] The feature enhancement module is used to enhance the channel information of the high-level image features in the image features of different receptive fields by utilizing the low-level image features in the image features of different receptive fields, thereby obtaining enhanced high-level features after channel information enhancement; wherein the receptive field of the high-level image features is larger than that of the low-level image features; and to enhance the spatial information of the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields, thereby obtaining enhanced low-level features after spatial information enhancement.

[0024] The fusion module is used to fuse the enhanced high-level features and the enhanced low-level features to obtain the fused features.

[0025] According to a fourth aspect of this disclosure, a model training apparatus for image feature fusion is provided, comprising:

[0026] The acquisition module is used to acquire multiple sample images and the corresponding labels for each sample image.

[0027] The feature fusion acquisition module is used to input the sample images into a model for image feature fusion to obtain fused features for each sample image.

[0028] The image processing result acquisition module is used to perform image processing based on the fused features to obtain the image processing result.

[0029] The calculation module is used to calculate the difference between the image processing result and the label corresponding to the sample image;

[0030] The training module is used to adjust the model parameters of the model for image feature fusion based on the differences; based on the adjusted model parameters and the multiple sample images, the model parameter adjustment process continues until a preset iteration termination condition is met; the model parameters obtained when the preset iteration termination condition is met are used as the trained model parameters, and the model for image feature fusion including the trained model parameters is used as the trained model for image feature fusion.

[0031] The model for image feature fusion includes a channel attention network, a spatial attention network, and a fusion network.

[0032] The channel attention network is used to enhance the channel information of the high-level image features in the image features of different receptive fields by utilizing the low-level image features in the image features of different receptive fields, thereby obtaining enhanced high-level features after channel information enhancement; wherein, the receptive field of the high-level image features is larger than the receptive field of the low-level image features.

[0033] The spatial attention network is used to enhance the spatial information of the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields, thereby obtaining enhanced low-level features with enhanced spatial information.

[0034] The fusion network is used to fuse the enhanced high-level features and the enhanced low-level features to obtain the fused features.

[0035] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0036] At least one processor; and

[0037] A memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect or the second aspect.

[0039] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first or second aspect.

[0040] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to the first or second aspect.

[0041] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0042] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0043] Figure 1 This is a flowchart of the image feature fusion method provided in the embodiments of this disclosure;

[0044] Figure 2 This is a schematic diagram illustrating the image feature fusion method provided in the embodiments of this disclosure;

[0045] Figure 3 This is a flowchart of a training method for a model for image feature fusion provided in this embodiment of the disclosure;

[0046] Figure 4 This is a schematic diagram of a model for image feature fusion in an embodiment of this disclosure;

[0047] Figure 5 This is another schematic diagram of a model for image feature fusion in an embodiment of this disclosure;

[0048] Figure 6 This is yet another schematic diagram of a model for image feature fusion in an embodiment of this disclosure;

[0049] Figure 7 This is a schematic diagram of the image feature fusion apparatus provided in the embodiments of this disclosure;

[0050] Figure 8 This is a schematic diagram of the structure of a model training device for image feature fusion provided in an embodiment of this disclosure;

[0051] Figure 9This is a block diagram of an electronic device used to implement the image feature fusion method or the training method of a model for image feature fusion according to the embodiments of this disclosure. Detailed Implementation

[0052] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0053] This disclosure provides an image feature fusion method, including:

[0054] Acquire image features from different receptive fields;

[0055] By utilizing the low-level image features in image features with different receptive fields, channel information enhancement is performed on the high-level image features in image features with different receptive fields to obtain enhanced high-level features with enhanced channel information; wherein, the receptive field of the high-level image features is larger than that of the low-level image features.

[0056] By utilizing high-level image features from different receptive fields, spatial information enhancement is performed on low-level image features from different receptive fields to obtain enhanced low-level features with spatial information enhancement.

[0057] The enhanced high-level features and enhanced low-level features are fused together to obtain the fused features.

[0058] In this embodiment, channel information enhancement is performed on the high-level image features in the image features of different receptive fields by utilizing the low-level image features in the image features of different receptive fields, and spatial information enhancement is performed on the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields, thereby obtaining enhanced low-level features with spatial information enhancement. This enables better utilization of the feature information of different receptive fields through the interaction of high-level and low-level image features, and better image feature fusion, thereby making the extracted image features more accurate, that is, improving the accuracy of image features.

[0059] Figure 1 This is a flowchart of the image feature fusion method provided in the embodiments of this disclosure, with reference to... Figure 1 The image feature fusion method provided in this disclosure embodiment may include:

[0060] S101, acquire image features of different receptive fields.

[0061] The receptive field represents the size of the region mapped to the original image. For example, image features are represented by image feature maps, and image features with different receptive fields can be image feature maps with different receptive fields.

[0062] In one implementation, image features from different receptive fields of an image can be extracted using a convolutional neural network (CNN). The receptive field represents the area on the original image that a pixel in the feature map output by each layer of the CNN maps back to. In simpler terms, the receptive field refers to the size of a point on the feature map relative to the original image, and is also the area of ​​the original image that the CNN's features can see. Specifically, image features from different receptive fields of the original image (which can also be understood as the image to be processed) can be extracted using a CNN.

[0063] S102, using the low-level image features in the image features of different receptive fields, channel information enhancement is performed on the high-level image features in the image features of different receptive fields to obtain enhanced high-level features after channel information enhancement.

[0064] S103, using high-level image features in image features of different receptive fields, spatial information enhancement is performed on low-level image features in image features of different receptive fields to obtain enhanced low-level features after spatial information enhancement.

[0065] Among them, the receptive field of high-level image features is larger than that of low-level image features.

[0066] S104 fuses the enhanced high-level features and enhanced low-level features to obtain the fused features.

[0067] Channel importance coefficients can be calculated based on low-level image features; high-level image features can then be adjusted based on these coefficients to obtain enhanced high-level features with enhanced channel information. The channel importance coefficient is directly proportional to the importance of the high-level image feature in the channel dimension; simply put, the larger the channel importance coefficient, the more important the high-level image feature is in terms of its channel dimension.

[0068] Spatial importance coefficients can be calculated based on high-level image features; low-level image features can be adjusted based on spatial importance coefficients to obtain enhanced low-level features with spatial information enhancement.

[0069] The spatial importance coefficient is directly proportional to the importance of the underlying image features in the spatial dimension. Simply put, the larger the spatial importance coefficient, the more important the underlying image feature is in terms of spatial dimension.

[0070] You can directly add the enhanced high-level features and enhanced low-level features to obtain the fused features. Alternatively, you can first concatenate the features along the channel dimension, and then aggregate the channel dimension information through convolution.

[0071] In one possible implementation, image features with different receptive fields include image features at different scales. In this embodiment, image features with different receptive fields are fused, that is, image features at different scales are fused to obtain image features at different scales. Using the low-level image features in the image features at different scales, channel information enhancement is performed on the high-level image features in the image features at different scales to obtain enhanced high-level features with enhanced channel information. The receptive field of the high-level image features is larger than that of the low-level image features. Using the high-level image features in the image features at different scales, spatial information enhancement is performed on the low-level image features in the image features at different scales to obtain enhanced low-level features with enhanced spatial information. The enhanced high-level features and the enhanced low-level features are fused to obtain the fused features.

[0072] In another possible implementation, image features of different receptive fields include image features of different receptive fields at the same scale, such as different regions in a feature map at the same scale.

[0073] Compared to related technologies that scale features from different scales to the same scale and then add or stitch them together without considering the effectiveness of features at different scales, or that use channel attention mechanisms for feature fusion without considering the differences between different spatial features, or that use self-attention mechanisms to construct correlations between features at different spatial locations over a large scale without considering the importance of different channels, this disclosure utilizes low-level image features from image features in different receptive fields to enhance the channel information of high-level image features in image features in different receptive fields, and utilizes high-level image features from image features in different receptive fields to enhance the spatial information of low-level image features in image features in different receptive fields. This results in enhanced low-level features with enhanced spatial information, enabling better utilization of feature information from different receptive fields through the interaction of high-level and low-level image features, thus improving image feature fusion and making the extracted image features more accurate.

[0074] In one alternative embodiment, such as Figure 2 As shown, the following may also be included before S101:

[0075] S201, Obtain the image to be processed.

[0076] Obtain image features from different receptive fields, i.e., extract image features from different receptive fields of the image to be processed.

[0077] S104 may also include:

[0078] S202, perform image classification, image detection and / or image segmentation based on the fused features to obtain classification results, detection results and / or segmentation results for the image to be processed.

[0079] Based on the fused features, image classification, image detection, or image segmentation can be performed, or both image classification and image detection can be performed, or both image segmentation and image detection can be performed, or both image classification, image detection, and image segmentation can be performed.

[0080] Specifically, when classifying an image, the process involves acquiring the image to be processed, extracting image features from different receptive fields, then using the low-level image features from each receptive field to enhance the high-level image features with channel information, resulting in enhanced high-level features. Similarly, using the high-level image features from each receptive field, the process involves enhancing the low-level image features with spatial information, resulting in enhanced low-level features. Finally, the enhanced high-level and low-level features are fused to obtain the fused features. Image classification is then performed based on these fused features of the image to be processed.

[0081] When performing image segmentation, the image to be processed can be acquired, and image features from different receptive fields can be extracted. Then, using the low-level image features from different receptive fields, channel information enhancement can be applied to the high-level image features from different receptive fields to obtain enhanced high-level features. Similarly, using the high-level image features from different receptive fields, spatial information enhancement can be applied to the low-level image features from different receptive fields to obtain enhanced low-level features. The enhanced high-level and enhanced low-level features are then fused to obtain the fused features. Image segmentation is then performed based on these fused features of the image to be processed.

[0082] When performing image detection on an image, the image to be processed can be acquired, and image features from different receptive fields can be extracted. Then, using the low-level image features from different receptive fields, channel information enhancement can be applied to the high-level image features from different receptive fields to obtain enhanced high-level features. Similarly, using the high-level image features from different receptive fields, spatial information enhancement can be applied to the low-level image features from different receptive fields to obtain enhanced low-level features. The enhanced high-level and enhanced low-level features are then fused to obtain the fused features. Image detection is then performed based on these fused features of the image to be processed.

[0083] In this embodiment, low-level image features from different receptive fields are used to enhance high-level image features with channel information, resulting in enhanced high-level features. High-level image features from different receptive fields are then used to enhance low-level image features with spatial information, resulting in enhanced low-level features with spatial information. The enhanced high-level and enhanced low-level features are then fused to obtain fused features. This allows for better utilization of different receptive field feature information through the interaction of high-level and low-level image features, leading to better image feature fusion and more accurate extracted image features. The fused features are then used for image detection, image classification, and image segmentation, resulting in more accurate image detection, image classification, and image segmentation results.

[0084] In one possible implementation, the fused features can be obtained through a model for image feature fusion. The specific model training process will be detailed in the following embodiments and will not be repeated here.

[0085] Specifically, models used for image feature fusion can include feature extraction networks, channel attention networks, spatial attention networks, and fusion networks.

[0086] Obtain image features from different receptive fields, including:

[0087] The image to be processed is obtained and input into the model for image feature fusion. The image features of different receptive fields of the image to be processed are extracted through the feature extraction network in the model for image feature fusion.

[0088] By utilizing low-level image features from different receptive fields, channel information enhancement is applied to high-level image features from different receptive fields to obtain enhanced high-level features, including:

[0089] By using the channel attention network in the model for image feature fusion, the channel information of the high-level image features in the image features of different receptive fields is enhanced by utilizing the low-level image features in the image features of different receptive fields, resulting in enhanced high-level features after channel information enhancement.

[0090] By utilizing high-level image features from different receptive fields, spatial information enhancement is performed on low-level image features from different receptive fields to obtain enhanced low-level features, including:

[0091] By using a spatial attention network in the model for image feature fusion, high-level image features in different receptive fields are used to spatially enhance low-level image features in different receptive fields, resulting in enhanced low-level features with spatial information enhancement.

[0092] The enhanced high-level features and enhanced low-level features are fused to obtain the fused features, including:

[0093] The fusion network in the model used for image feature fusion fuses enhanced high-level features and enhanced low-level features to obtain fused features.

[0094] Simply put, the image to be processed is input into a model for image feature fusion. This model can enhance the channel information of high-level image features in different receptive fields by utilizing low-level image features from different receptive fields, and enhance the spatial information of low-level image features in different receptive fields by utilizing high-level image features from different receptive fields. The fused features are obtained through this model. On the one hand, the interaction between high-level and low-level image features allows for better utilization of feature information from different receptive fields, resulting in better image feature fusion and more accurate extracted image features. On the other hand, implementing this process through the model makes image fusion more convenient.

[0095] After obtaining the fused features through the model used for image feature fusion, image classification, image detection, and / or image segmentation can be performed based on the fused features to obtain classification results, detection results, and / or segmentation results for the image to be processed, as detailed above. Figure 2 Corresponding implementation examples.

[0096] In one optional embodiment, the channel attention network may include a global pooling layer, a first perception layer, and a first normalization layer;

[0097] By using a channel attention network in the model for image feature fusion, channel information enhancement is performed on high-level image features in different receptive fields by utilizing low-level image features from different receptive fields, including:

[0098] Global pooling is used to perform global pooling on the features of the underlying image.

[0099] Using the result of the global pooling operation in the first perception layer, a column vector with the same channel dimension as the high-level image features is obtained through a fully connected operation;

[0100] The first normalization layer is used to normalize the column vectors that have the same channel dimension as the high-level image features, resulting in the first normalized column vector. The values ​​in the first normalized column vector represent the channel importance coefficients.

[0101] Channel importance coefficients are used to adjust high-level image features to obtain enhanced high-level features with enhanced channel information. The channel importance coefficient is directly proportional to the importance of the high-level image feature in the channel dimension; simply put, the larger the channel importance coefficient, the more important the high-level image feature is in terms of its channel dimension. It can also be understood that lower-level image features provide channel importance information to higher-level image features.

[0102] Spatial attention networks may include a second perceptual layer and a second normalization layer;

[0103] By using a spatial attention network in the model for image feature fusion, spatial information enhancement is performed on low-level image features in different receptive fields by utilizing high-level image features from different receptive fields, including:

[0104] The second perception layer is used to perform a fully connected operation on the high-level image features to obtain a column vector with the same spatial dimension as the low-level image features.

[0105] The second normalization layer is used to normalize the column vectors with the same spatial dimension as the features of the underlying image, resulting in the second normalized column vector. The values ​​in the second normalized column vector represent the spatial importance coefficients.

[0106] Spatial importance coefficients are used to adjust low-level image features to obtain enhanced low-level features with spatial information augmentation. The spatial importance coefficient is directly proportional to the spatial importance of the low-level image features; simply put, the larger the spatial importance coefficient, the more important the low-level image feature is in terms of its spatial dimension. It can also be understood that high-level image features provide spatial importance information to low-level image features.

[0107] The channel attention network includes a global pooling layer, a first perception layer, and a first normalization layer. The spatial attention network includes a second perception layer and a second normalization layer. Specifically, the global pooling layer, the first perception layer, and the first normalization layer can enhance the channel information of high-level image features from different receptive fields by utilizing low-level image features from different receptive fields. The second perception layer and the second normalization layer can enhance the spatial information of low-level image features from different receptive fields by utilizing high-level image features from different receptive fields. This results in low model complexity, reducing computational complexity when using the model for feature fusion and subsequent image classification, segmentation, and detection. It achieves simple and effective image processing through feature fusion.

[0108] This disclosure provides a training method for a model used in image feature fusion, such as... Figure 3 As shown, it may include:

[0109] S301, Obtain multiple sample images and the labels corresponding to each sample image;

[0110] S302, For each sample image, input the sample image into the model for image feature fusion to obtain the fused features.

[0111] S303, Image processing is performed based on the fused features to obtain the image processing result;

[0112] S304, calculate the difference between the image processing result and the label corresponding to the sample image;

[0113] S305, Adjusting the model parameters of the model used for image feature fusion based on differences;

[0114] S306, Based on the adjusted model parameters and multiple sample images, continue the process of adjusting the model parameters until the preset iteration termination condition is met;

[0115] S307, the model parameters obtained when the preset iteration termination condition is met are used as the trained model parameters, and the model for image feature fusion including the trained model parameters is used as the trained model for image feature fusion.

[0116] Among them, the models used for image feature fusion include feature extraction networks, channel attention networks, spatial attention networks, and fusion networks;

[0117] Feature extraction network is used to extract image features from different receptive fields of the image to be processed;

[0118] Channel attention network is used to enhance the channel information of high-level image features in different receptive fields by utilizing low-level image features in different receptive fields, resulting in enhanced high-level features with enhanced channel information; wherein, the receptive field of high-level image features is larger than that of low-level image features.

[0119] Spatial attention network is used to enhance the spatial information of low-level image features in different receptive fields by utilizing high-level image features in different receptive fields, resulting in enhanced low-level features with spatial information enhancement.

[0120] Fusion networks are used to fuse enhanced high-level features and enhanced low-level features to obtain fused features.

[0121] First, a model for image feature fusion can be constructed, which includes a feature extraction network, a channel attention network, a spatial attention network, and a fusion network. Then, the model is trained using multiple sample images and the labels corresponding to each sample image until the preset iteration termination condition is met, resulting in the trained model.

[0122] Specifically, the sample image is input into a model for image feature fusion. The feature extraction network in this model extracts image features from different receptive fields of the image to be processed. The channel attention network in the model then uses low-level image features from different receptive fields to enhance the channel information of high-level image features, resulting in enhanced high-level features. Similarly, the spatial attention network in the model uses high-level image features from different receptive fields to enhance the spatial information of low-level image features, resulting in enhanced low-level features. Finally, the fusion network in the model fuses the enhanced high-level and low-level features to obtain the fused features. Figure 4 As shown, the high-level image features are high-level semantic feature maps (Global features), and the low-level image features are low-level semantic feature maps (Local features). Spatial Interaction is used to enhance the spatial information of the low-level semantic feature maps using the high-level semantic feature maps. That is, using high-level image features from different receptive fields, spatial information enhancement is applied to the low-level image features from different receptive fields, resulting in enhanced low-level features with enhanced spatial information. Channel Interaction is used to enhance the channel dimension information of the high-level semantic feature maps using the low-level semantic feature maps. That is, using low-level image features from different receptive fields, channel information enhancement is applied to the high-level image features from different receptive fields, resulting in enhanced high-level features with enhanced channel information. Fuse is the fusion of the enhanced high-level and low-level features, that is, fusing the enhanced high-level and enhanced low-level features to obtain the fused features.

[0123] Then, image processing is performed based on the fused features to obtain the image processing result; the difference between the image processing result and the label corresponding to the sample image is calculated; the model parameters of the model used for image feature fusion are adjusted based on the difference; based on the adjusted model parameters and multiple sample images, the model parameter adjustment process continues until the preset iteration termination condition is met; the model parameters obtained when the preset iteration termination condition is met are used as the trained model parameters, and the model including the trained model parameters is used as the trained model for image feature fusion.

[0124] Model parameters can be adjusted by backpropagating the error gradient, which can also be understood as updating the weights by backpropagating the error gradient.

[0125] The preset iteration termination condition may include the convergence of the difference between the image processing result of a sample image and the label corresponding to the image processing result, or the preset iteration termination condition may include the number of iterations reaching a preset number, which can be determined according to actual needs or experience, such as 500 times, 600 times, 1000 times, etc.

[0126] Furthermore, after training is complete, i.e., after obtaining the trained model for image feature fusion, the model can be tested. Specifically, multiple test images (also known as data to be inferred) can be obtained, and the trained model for image feature fusion can be adjusted based on the test images. The specific process is the same as the process described above for training the model for image feature fusion based on sample images. If the model for image feature fusion adjusted based on the test data is superior to the trained model for image feature fusion, then the model for image feature fusion adjusted based on the test data is adopted as the final model.

[0127] In this embodiment, a model for image feature fusion can be pre-trained. This model can utilize low-level image features from different receptive fields to enhance the channel information of high-level image features from different receptive fields, resulting in enhanced high-level features. Similarly, it can utilize high-level image features from different receptive fields to enhance the spatial information of low-level image features from different receptive fields, resulting in enhanced low-level features. The enhanced high-level and enhanced low-level features are then fused to obtain fused features. This allows for better utilization of different receptive field feature information through the interaction of high-level and low-level image features, leading to more accurate image feature extraction. Furthermore, the fused features can be used for image detection, image classification, and image segmentation, resulting in more accurate image detection, classification, and segmentation results. Additionally, the pre-trained model for image feature fusion allows for convenient feature fusion during the actual implementation.

[0128] This disclosure does not limit the specific type of model. For example, it can be a convolutional neural network such as a residual network. The processing modules in the residual network are replaced by network layers consisting of a feature extraction network, a channel attention network, a spatial attention network, and a fusion network to construct a model for image feature fusion. Figure 5As shown, a model is built and its parameters are initialized using a deep residual network (ResNet) structure. The residual network includes an input layer, a feature extraction layer, a processing module, and an output layer. C In this model, there can be multiple blocks, such as xN. A residual network fuses high-level semantic feature maps (Global Feature Maps, GF) and low-level image features (Local Feature Maps, LF) using two 3×3 convolutions (conv3×3). In this embodiment, a network layer consisting of a feature extraction network, a channel attention network, a spatial attention network, and a fusion network replaces the blocks (i.e., the modules within the dashed boxes), thus constructing an initial model. Then, this model is trained to obtain a trained model for image feature fusion.

[0129] In one optional embodiment, for each sample image, the label corresponding to the sample image includes the category label, detection label, and / or segmentation label of the sample image;

[0130] S304 may include:

[0131] Based on the fusion results, image classification, image detection, and / or image segmentation are performed to obtain classification results, detection results, and / or segmentation results for the sample image;

[0132] S305 may include:

[0133] Calculate the difference between the classification result and the category label, the difference between the detection result and the detection label, and / or the difference between the segmentation result and the segmentation label.

[0134] In this embodiment of the disclosure, a model for image feature fusion can be trained using multiple sample images and their category labels; or, a model for image feature fusion can be trained using multiple sample images and their detection labels; or, a model for image feature fusion can be trained using multiple sample images and their segmentation labels; or, a model for image feature fusion can be trained using multiple sample images and their category and detection labels; or, a model for image feature fusion can be trained using multiple sample images and their category, detection, and segmentation labels; or, a model for image feature fusion can be trained using multiple sample images and their category, detection, and segmentation labels.

[0135] In one possible approach, a model for image feature fusion can be trained using multiple sample images and their category labels, and then this model can be used for image classification. Specifically, for each sample image, the sample image is input into the model for image feature fusion to obtain fused features. Image classification is performed based on the fused image to obtain a classification result for the sample image; the difference between the classification result and the category label corresponding to the sample image is calculated; the model parameters for image feature fusion are adjusted based on the difference; based on the adjusted model parameters and multiple sample images, the model parameter adjustment process continues until a preset iteration termination condition is met; the model parameters obtained when the preset iteration termination condition is met are used as the trained model parameters, and the model for image feature fusion including the trained model parameters is used as the trained model for image feature fusion. The process involves inputting sample images into an image feature fusion model to obtain fused features. This includes: inputting sample images into the model, extracting image features from different receptive fields of the image to be processed using the feature extraction network within the model; using the channel attention network within the model, enhancing the channel information of the high-level image features from the image features in different receptive fields using the low-level image features, resulting in enhanced high-level features; using the spatial attention network within the model, enhancing the spatial information of the low-level image features from the image features in different receptive fields using the high-level image features, resulting in enhanced low-level features; and finally, fusing the enhanced high-level and enhanced low-level features using the fusion network within the model to obtain the fused features.

[0136] This embodiment of the disclosure can train a model for image feature fusion using multiple sample images and their corresponding category labels. This model can then perform image feature fusion on the image to be processed, and subsequently perform image classification based on the fused features. The model utilizes low-level image features from different receptive fields to enhance the channel information of high-level image features from different receptive fields, resulting in enhanced high-level features. It also utilizes high-level image features from different receptive fields to enhance the spatial information of low-level image features from different receptive fields, resulting in enhanced low-level features. Finally, the enhanced high-level and enhanced low-level features are fused to obtain the fused features. This allows for better utilization of different receptive field feature information through the interaction of high-level and low-level image features, leading to better image feature fusion and more accurate extracted image features. Furthermore, because the model is trained based on sample images and their category labels, image classification using this model results in more accurate classification results.

[0137] In another possible implementation, a model for image feature fusion can be trained using multiple sample images and their detection labels, and then this model can be used for image detection. The specific training process is similar to training a model using multiple sample images and their category labels, but the difference lies in performing image detection based on the fused image, obtaining detection results for the sample images, and calculating the difference between the detection results and the detection labels. In this embodiment, after obtaining the model, image features can be fused using this model on the image to be processed, and then image detection can be performed based on the fused features. The model utilizes low-level image features from different receptive fields to enhance the channel information of high-level image features from different receptive fields, obtaining enhanced high-level features with enhanced channel information; it also utilizes high-level image features from different receptive fields to enhance the spatial information of low-level image features from different receptive fields, obtaining enhanced low-level features with enhanced spatial information; and then fuses the enhanced high-level features and enhanced low-level features to obtain the fused features. This allows for better utilization of different receptive field feature information through the interaction of high-level and low-level image features, resulting in better image feature fusion and more accurate extracted image features. In addition, because the model is trained based on sample images and their detection labels, using this model for image detection can make the image detection results more accurate.

[0138] In another possible approach, a model for image feature fusion can be trained using multiple sample images and their segmentation labels, and then this model can be used for image segmentation. The specific training process is similar to training a model using multiple sample images and their category labels, but the difference lies in performing image segmentation based on the fused image, obtaining the segmentation result for the sample image, and calculating the difference between the segmentation result and the segmentation label. In this embodiment, after obtaining the model, image features can be fused using this model on the image to be processed, and then image segmentation can be performed based on the fused features. The model utilizes low-level image features from different receptive fields to enhance the channel information of high-level image features from different receptive fields, obtaining enhanced high-level features with enhanced channel information; it also utilizes high-level image features from different receptive fields to enhance the spatial information of low-level image features from different receptive fields, obtaining enhanced low-level features with enhanced spatial information; and finally, it fuses the enhanced high-level and enhanced low-level features to obtain the fused features. This allows for better utilization of different receptive field feature information through the interaction of high-level and low-level image features, resulting in better image feature fusion and more accurate extracted image features. In addition, because the model is trained based on sample images and image segmentation labels, using this model for image segmentation can make the image segmentation results more accurate.

[0139] In another possible implementation, a model for image feature fusion is trained using multiple sample images and their category and detection labels; or, a model for image feature fusion is trained using multiple sample images and their category and segmentation labels; or, a model for image feature fusion is trained using multiple sample images and their detection and segmentation labels; or, a model for image feature fusion is trained using multiple sample images and their category, detection, and segmentation labels. The multiple sample images may include two parts: training the model using one part of the sample images and their category labels, and training the model using another part of the sample images and their detection labels; or, training the model using one part of the sample images and their category labels, and training the model using another part of the sample images and their segmentation labels; or, training the model using one part of the sample images and their detection labels, and training the model using another part of the sample images and their segmentation labels. Alternatively, multiple sample images can include three parts: one part of the sample images and its category label, another part of the sample images and its detection label, and a third part of the sample images and its segmentation label, used for model training. In this embodiment, training the model using multiple scenarios makes it easier to obtain multiple sample data, reducing the difficulty of model training. Furthermore, combining data from multiple scenarios improves the accuracy of model training, resulting in more accurate fused features obtained through image feature fusion, and consequently, more accurate results for image classification, detection, and segmentation based on these fused features.

[0140] In one optional embodiment, the channel attention network includes a global pooling layer, a first perception layer, and a first normalization layer;

[0141] The global pooling layer is used to perform global pooling operations on the features of the underlying image.

[0142] The first perceptual layer is used to obtain a column vector with the same channel dimension as the high-level image features by using a fully connected operation based on the result of the global pooling operation.

[0143] The first normalization layer is used to normalize the column vectors that have the same channel dimensions as the high-level image features, resulting in a first normalized column vector. The values ​​in the first normalized column vector represent the channel importance coefficients.

[0144] The channel attention network can further normalize the column vectors obtained from the first normalization layer by using an activation function, mapping the values ​​of the column vectors to values ​​in (0,1).

[0145] Channel attention network is used to calculate channel importance coefficients based on low-level image features. Based on the channel importance coefficients, high-level image features are adjusted to obtain enhanced high-level features with enhanced channel information. This achieves the goal of using low-level image features in image features with different receptive fields to enhance the channel information of high-level image features in image features with different receptive fields, thus obtaining enhanced high-level features with enhanced channel information.

[0146] Spatial attention networks consist of a second perceptual layer and a second normalization layer;

[0147] The second perception layer is used to perform fully connected operations on the high-level image features to obtain a column vector with the same spatial dimension as the low-level image features.

[0148] The second normalization layer is used to normalize the column vectors that have the same spatial dimension as the features of the underlying image, resulting in a second normalized column vector. The values ​​in the second normalized column vector represent the spatial importance coefficients.

[0149] A spatial attention network is used to calculate the spatial importance coefficient based on high-level image features. The low-level image features are then adjusted based on the spatial importance coefficient. The adjusted features are the spatially enhanced features, resulting in enhanced low-level features with spatial information enhancement. This allows for the use of high-level image features from different receptive fields to enhance the spatial information of low-level image features from different receptive fields, thus obtaining enhanced low-level features with spatial information enhancement.

[0150] like Figure 6 As shown, the high-level image features are high-level semantic feature maps (Global features), and the low-level image features are low-level semantic feature maps (Local features). Spatial Interaction is used to enhance the spatial information of the low-level semantic feature maps using the high-level semantic feature maps. That is, using high-level image features from different receptive fields, spatial information enhancement is applied to the low-level image features from different receptive fields, resulting in enhanced low-level features with enhanced spatial information. Channel Interaction is used to enhance the channel dimension information of the high-level semantic feature maps using the low-level semantic feature maps. That is, using low-level image features from different receptive fields, channel information enhancement is applied to the high-level image features from different receptive fields, resulting in enhanced high-level features with enhanced channel information. Fuse is the fusion of the enhanced high-level and low-level features, that is, fusing the enhanced high-level and enhanced low-level features to obtain the fused features.

[0151] Figure 6In (C1,H,W), C1 represents the number of channels, the height, and the width of the underlying semantic feature map; in (C2,H,W), C2 represents the number of channels, the height, and the width of the underlying semantic feature map.

[0152] Channel Interaction includes GAP, LP, BN, and Activation; Spatial Interaction also includes LP, BN, and Activation. BN is batch normalization, Activation represents an activation function, LP is the receptive layer, and GAP is global pooling. Concat is a feature concatenation operation; conv1x1 adjusts the feature channel dimensions, resulting in fused features through concatenation and feature channel dimension adjustment.

[0153] In the Channel Interaction process, GAP (Global Pooling Layer) performs global pooling on the underlying feature information, resulting in a C1-dimensional column vector. LP (Layered Perceptual Layer), specifically a linear perceptual layer, performs fully connected operations, outputting a C2-dimensional column vector with the same channel dimension as the high-level semantic feature map. This result is then input into BN (Browsing Normalization Layer) for normalization, resulting in the normalized output with the same dimension. Activation yields a number between (0, 1), which can be seen as adjusting the importance of each channel feature in the high-level semantic feature map. A larger value indicates a more important feature.

[0154] Similarly, Spatial Interaction can also calculate spatial importance coefficients based on high-level image features through LP, BN, and Activation; the low-level image features are then adjusted based on the spatial importance coefficients, and the adjusted features are the spatially enhanced features, resulting in the enhanced low-level features with spatial information. In Spatial Interaction, LP is the second perceptual layer mentioned above, and BN is the second normalization layer mentioned above.

[0155] In this embodiment, the spatial attention network uses convolutional operations to aggregate or reduce the dimensionality of information along the channel dimension. Compared to conventional operations that choose to perform max or mean pooling for dimensionality reduction along the channel, the convolutional operation in this embodiment is more flexible as it targets learnable parameters. The linear perception layer in the channel attention network is only one layer, which reduces the complexity of the structure.

[0156] In this embodiment, the channel attention network includes a global pooling layer, a first perception layer, and a first normalization layer, while the spatial attention network includes a second perception layer and a second normalization layer. Specifically, the global pooling layer, the first perception layer, and the first normalization layer can enhance the channel information of high-level image features in different receptive fields by utilizing low-level image features from different receptive fields. Similarly, the second perception layer and the second normalization layer can enhance the spatial information of low-level image features in different receptive fields by utilizing high-level image features from different receptive fields. This reduces model complexity and, consequently, reduces computational complexity when using the model to perform feature fusion and then using the fused features for image classification, image segmentation, image detection, and other processes. This achieves simple and effective image processing through feature fusion.

[0157] Corresponding to the image feature fusion method provided in the above embodiments, this disclosure also provides an image feature fusion apparatus, such as... Figure 7 As shown, it may include:

[0158] The acquisition module 701 is used to acquire image features of different receptive fields;

[0159] The feature enhancement module 702 is used to enhance the channel information of the high-level image features in the image features of different receptive fields by utilizing the low-level image features in the image features of different receptive fields, thereby obtaining enhanced high-level features after channel information enhancement; wherein, the receptive field of the high-level image features is larger than that of the low-level image features; and is used to enhance the spatial information of the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields, thereby obtaining enhanced low-level features after spatial information enhancement.

[0160] The fusion module 703 is used to fuse enhanced high-level features and enhanced low-level features to obtain fused features.

[0161] Optionally, the acquisition module 701 is specifically used to acquire the image to be processed, input the image to be processed into the model for image feature fusion, and extract image features of different receptive fields of the image to be processed through the feature extraction network in the model for image feature fusion.

[0162] The feature enhancement module 702 is specifically used to enhance the channel information of the high-level image features in different receptive fields by using the channel attention network in the image feature fusion model and utilizing the low-level image features in different receptive fields, thereby obtaining enhanced high-level features with enhanced channel information; and to enhance the spatial information of the low-level image features in different receptive fields by using the high-level image features in different receptive fields, thereby obtaining enhanced low-level features with enhanced spatial information.

[0163] The fusion module 703 is used to fuse enhanced high-level features and enhanced low-level features through a fusion network in a model for image feature fusion to obtain fused features.

[0164] Optionally, the channel attention network includes a global pooling layer, a first perception layer, and a first normalization layer; the spatial attention network includes a second perception layer and a second normalization layer.

[0165] The feature enhancement module 702 is specifically used to perform global pooling operations on the low-level image features using a global pooling layer; using the result of the global pooling operation in the first perception layer, a column vector with the same channel dimension as the high-level image features is obtained through a fully connected operation; a first normalization layer is used to normalize the column vector with the same channel dimension as the high-level image features to obtain a first normalized column vector, where the values ​​in the first normalized column vector represent channel importance coefficients; a second perception layer is used to perform a fully connected operation on the high-level image features to obtain a column vector with the same spatial dimension as the low-level image features; a second normalization layer is used to normalize the column vector with the same spatial dimension as the low-level image features to obtain a second normalized column vector, where the values ​​in the second normalized column vector represent spatial importance coefficients.

[0166] Corresponding to the model training method for image feature fusion provided in the above embodiments, this disclosure also provides a model training apparatus for image feature fusion, such as... Figure 8 As shown, it may include:

[0167] The acquisition module 801 is used to acquire multiple sample images and the labels corresponding to each sample image;

[0168] The feature fusion acquisition module 802 is used to input the sample images into the image feature fusion model for each sample image to obtain the fused features.

[0169] The image processing result acquisition module 803 is used to perform image processing based on the fused features to obtain the image processing result;

[0170] The calculation module 804 is used to calculate the difference between the image processing result and the label corresponding to the sample image;

[0171] The training module 805 is used to adjust the model parameters of the model for image feature fusion based on differences; based on the adjusted model parameters and multiple sample images, the model parameter adjustment process continues until the preset iteration termination condition is met; the model parameters obtained when the preset iteration termination condition is met are used as the trained model parameters, and the model for image feature fusion including the trained model parameters is used as the trained model for image feature fusion.

[0172] Among them, the models used for image feature fusion include channel attention networks, spatial attention networks, and fusion networks;

[0173] Channel attention network is used to enhance the channel information of high-level image features in different receptive fields by utilizing low-level image features in different receptive fields, resulting in enhanced high-level features with enhanced channel information; wherein, the receptive field of high-level image features is larger than that of low-level image features.

[0174] Spatial attention network is used to enhance the spatial information of low-level image features in different receptive fields by utilizing high-level image features in different receptive fields, resulting in enhanced low-level features with spatial information enhancement.

[0175] Fusion networks are used to fuse enhanced high-level features and enhanced low-level features to obtain fused features.

[0176] Optionally, for each sample image, the label corresponding to the sample image includes the category label, detection label, and / or segmentation label of the sample image;

[0177] The image processing result acquisition module 803 is specifically used to perform image classification, image detection and / or image segmentation based on the fusion result, and to obtain the classification result, detection result and / or segmentation result for the sample image;

[0178] The calculation module 804 is specifically used to calculate the difference between the classification result and the category label, the difference between the detection result and the detection label, and / or the difference between the segmentation result and the segmentation label.

[0179] Optionally, the channel attention network includes a global pooling layer, a first perception layer, and a first normalization layer;

[0180] The global pooling layer is used to perform global pooling operations on the features of the underlying image.

[0181] The first perceptual layer is used to obtain a column vector with the same channel dimension as the high-level image features by using a fully connected operation based on the result of the global pooling operation.

[0182] The first normalization layer is used to normalize the column vectors that have the same channel dimension as the high-level image features, resulting in the first normalized column vector. The values ​​in the first normalized column vector represent the channel importance coefficients.

[0183] Spatial attention networks consist of a second perceptual layer and a second normalization layer;

[0184] The second perception layer is used to perform fully connected operations on the high-level image features to obtain a column vector with the same spatial dimension as the low-level image features.

[0185] The second normalization layer is used to normalize the column vectors that have the same spatial dimension as the features of the underlying image, resulting in a second normalized column vector. The values ​​in the second normalized column vector represent the spatial importance coefficients.

[0186] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0187] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0188] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0189] like Figure 9As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0190] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0191] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as image feature fusion methods or model training methods for image feature fusion. For example, in some embodiments, the image feature fusion method or the model training method for image feature fusion can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the image feature fusion method or the model training method for image feature fusion described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured by any other suitable means (e.g., by means of firmware) to perform an image feature fusion method or a model training method for image feature fusion.

[0192] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0194] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0196] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0197] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0198] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0199] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image feature fusion method, comprising: Acquire image features from different receptive fields; By utilizing the low-level image features in the image features with different receptive fields, channel information enhancement is performed on the high-level image features in the image features with different receptive fields to obtain enhanced high-level features after channel information enhancement; wherein, the receptive field of the high-level image features is larger than that of the low-level image features. By utilizing the high-level image features in the image features of different receptive fields, spatial information enhancement is performed on the low-level image features in the image features of different receptive fields to obtain enhanced low-level features with spatial information enhancement. The enhanced high-level features and enhanced low-level features are fused to obtain the fused features; The step of enhancing the high-level image features in the image features of different receptive fields by utilizing the low-level image features in the image features of different receptive fields includes: The underlying image features are globally pooled using the global pooling layer of the channel attention network in the model for image feature fusion. Using the result of global pooling operation from the first perceptual layer of the channel attention network, a column vector with the same channel dimension as the high-level image features is obtained through a fully connected operation. The first normalization layer of the channel attention network is used to normalize the column vector with the same channel dimension as the high-level image features to obtain the first normalized column vector. The value in the first normalized column vector represents the channel importance coefficient. The step of enhancing the spatial information of the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields includes: The second perceptual layer of the spatial attention network in the model for image feature fusion is used to perform a fully connected operation on the high-level image features to obtain a column vector with the same spatial dimension as the low-level image features. The second normalization layer of the spatial attention network is used to normalize the column vector that has the same spatial dimension as the underlying image features, resulting in a second normalized column vector. The values ​​in the second normalized column vector represent spatial importance coefficients.

2. The method of claim 1, wherein, The acquisition of image features from different receptive fields includes: The image to be processed is obtained and input into a model for image feature fusion. Image features of different receptive fields of the image to be processed are extracted through the feature extraction network in the model for image feature fusion. The step of using low-level image features from image features in different receptive fields to enhance high-level image features with channel information, resulting in enhanced high-level features with enhanced channel information, includes: By using the channel attention network in the model for image feature fusion, the channel information of the high-level image features in the image features of different receptive fields is enhanced using the low-level image features in the image features of different receptive fields, resulting in enhanced high-level features with enhanced channel information. The step of using high-level image features from image features in different receptive fields to spatially enhance low-level image features in image features in different receptive fields, resulting in enhanced low-level features with spatially enhanced information, includes: By using the spatial attention network in the model for image feature fusion, the high-level image features in the image features of different receptive fields are used to enhance the spatial information of the low-level image features in the image features of different receptive fields, thereby obtaining the enhanced low-level features after spatial information enhancement. The process of fusing the enhanced high-level features and the enhanced low-level features to obtain the fused features includes: The enhanced high-level features and enhanced low-level features are fused together through the fusion network in the model for image feature fusion to obtain the fused features.

3. A model training method for image feature fusion, comprising: Obtain multiple sample images and the corresponding labels for each sample image; For each sample image, the sample image is input into a model for image feature fusion to obtain the fused features; Image processing is performed based on the fused features to obtain the image processing result; Calculate the difference between the image processing result and the label corresponding to the sample image; The model parameters of the model used for image feature fusion are adjusted based on the differences. Based on the adjusted model parameters and the multiple sample images, the model parameter adjustment process continues until the preset iteration termination condition is met. The model parameters obtained when the preset iteration termination condition is met are used as the trained model parameters, and the model for image feature fusion including the trained model parameters is used as the trained model for image feature fusion. The model for image feature fusion includes a channel attention network, a spatial attention network, and a fusion network. The channel attention network is used to enhance the channel information of the high-level image features in the image features with different receptive fields by utilizing the low-level image features in the image features with different receptive fields, thereby obtaining enhanced high-level features with enhanced channel information; wherein the receptive field of the high-level image features is larger than that of the low-level image features. The spatial attention network is used to enhance the spatial information of the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields, thereby obtaining enhanced low-level features with enhanced spatial information. The fusion network is used to fuse the enhanced high-level features and the enhanced low-level features to obtain fused features; The channel attention network includes a global pooling layer, a first perception layer, and a first normalization layer; The global pooling layer is used to perform global pooling operations on the underlying image features; The first perception layer is used to obtain a column vector with the same channel dimension as the high-level image features by using a fully connected operation based on the result of the global pooling operation. The first normalization layer is used to normalize the column vector that has the same channel dimension as the high-level image features to obtain the first normalized column vector. The value in the first normalized column vector represents the channel importance coefficient. The spatial attention network includes a second perception layer and a second normalization layer; The second perception layer is used to perform a fully connected operation on the high-level image features to obtain a column vector with the same spatial dimension as the low-level image features; The second normalization layer is used to normalize the column vectors that have the same spatial dimension as the underlying image features to obtain a second normalized column vector. The values ​​in the second normalized column vector represent spatial importance coefficients.

4. The method of claim 3, wherein, For each sample image, the label corresponding to the sample image includes the category label, detection label, and / or segmentation label of the sample image; The image processing based on the fused features to obtain the image processing result includes: Based on the fusion result, image classification, image detection, and / or image segmentation are performed to obtain classification results, detection results, and / or segmentation results for the sample image; The calculation of the difference between the image processing result and the label corresponding to the sample image includes: Calculate the difference between the classification result and the category label, the difference between the detection result and the detection label, and / or the difference between the segmentation result and the segmentation label.

5. An image feature fusion apparatus, comprising: The acquisition module is used to acquire image features from different receptive fields; The feature enhancement module is used to enhance the channel information of the high-level image features in the image features of different receptive fields by utilizing the low-level image features in the image features of different receptive fields, thereby obtaining enhanced high-level features after channel information enhancement; wherein the receptive field of the high-level image features is larger than that of the low-level image features; and to enhance the spatial information of the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields, thereby obtaining enhanced low-level features after spatial information enhancement. The fusion module is used to fuse the enhanced high-level features and the enhanced low-level features to obtain the fused features; Specifically, the feature enhancement module is used to perform a global pooling operation on the underlying image features using the global pooling layer of the channel attention network in the image feature fusion model; based on the result of the global pooling operation, the first perceptual layer of the channel attention network obtains a column vector with the same channel dimension as the high-level image features through a fully connected operation; the first normalization layer of the channel attention network normalizes the column vector with the same channel dimension as the high-level image features to obtain a first normalized column vector, where the values ​​in the first normalized column vector represent channel importance coefficients; the second perceptual layer of the spatial attention network in the image feature fusion model performs a fully connected operation on the high-level image features to obtain a column vector with the same spatial dimension as the underlying image features; the second normalization layer of the spatial attention network normalizes the column vector with the same spatial dimension as the underlying image features to obtain a second normalized column vector, where the values ​​in the second normalized column vector represent spatial importance coefficients.

6. The apparatus according to claim 5, wherein, The acquisition module is specifically used to acquire the image to be processed, input the image to be processed into a model for image feature fusion, and extract image features of different receptive fields of the image to be processed through the feature extraction network in the model for image feature fusion. The feature enhancement module is specifically used to enhance the channel information of the high-level image features in the image features of different receptive fields by using the channel attention network in the model for image feature fusion and utilizing the low-level image features in the image features of different receptive fields, thereby obtaining enhanced high-level features after channel information enhancement. By using the spatial attention network in the model for image feature fusion, the high-level image features in the image features of different receptive fields are used to enhance the spatial information of the low-level image features in the image features of different receptive fields, thereby obtaining the enhanced low-level features after spatial information enhancement. The fusion module is used to fuse the enhanced high-level features and the enhanced low-level features through the fusion network in the model for image feature fusion to obtain fused features.

7. A model training device for image feature fusion, comprising: The acquisition module is used to acquire multiple sample images and the corresponding labels for each sample image. The feature fusion acquisition module is used to input the sample images into a model for image feature fusion to obtain fused features for each sample image. The image processing result acquisition module is used to perform image processing based on the fused features to obtain the image processing result. The calculation module is used to calculate the difference between the image processing result and the label corresponding to the sample image; A training module is used to adjust the model parameters of the model for image feature fusion based on the differences; Based on the adjusted model parameters and the multiple sample images, the model parameter adjustment process continues until the preset iteration termination condition is met. The model parameters obtained when the preset iteration termination condition is met are used as the trained model parameters, and the model for image feature fusion including the trained model parameters is used as the trained model for image feature fusion. The model for image feature fusion includes a channel attention network, a spatial attention network, and a fusion network. The channel attention network is used to enhance the channel information of the high-level image features in the image features with different receptive fields by utilizing the low-level image features in the image features with different receptive fields, thereby obtaining enhanced high-level features with enhanced channel information; wherein the receptive field of the high-level image features is larger than that of the low-level image features. The spatial attention network is used to enhance the spatial information of the low-level image features in the image features of different receptive fields by utilizing the high-level image features in the image features of different receptive fields, thereby obtaining enhanced low-level features with enhanced spatial information. The fusion network is used to fuse the enhanced high-level features and the enhanced low-level features to obtain fused features; The channel attention network includes a global pooling layer, a first perception layer, and a first normalization layer; The global pooling layer is used to perform global pooling operations on the underlying image features; The first perception layer is used to obtain a column vector with the same channel dimension as the high-level image features by using a fully connected operation based on the result of the global pooling operation. The first normalization layer is used to normalize the column vector that has the same channel dimension as the high-level image features to obtain the first normalized column vector. The value in the first normalized column vector represents the channel importance coefficient. The spatial attention network includes a second perception layer and a second normalization layer; The second perception layer is used to perform a fully connected operation on the high-level image features to obtain a column vector with the same spatial dimension as the low-level image features; The second normalization layer is used to normalize the column vectors that have the same spatial dimension as the underlying image features to obtain a second normalized column vector. The values ​​in the second normalized column vector represent spatial importance coefficients.

8. The apparatus according to claim 7, wherein, For each sample image, the label corresponding to the sample image includes the category label, detection label, and / or segmentation label of the sample image; The image processing result acquisition module is specifically used to perform image classification, image detection and / or image segmentation based on the fusion result, so as to obtain the classification result, detection result and / or segmentation result for the sample image; The calculation module is specifically used to calculate the difference between the classification result and the category label, the difference between the detection result and the detection label, and / or the difference between the segmentation result and the segmentation label.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4.

Images