Method, device, electronic equipment and storage medium for processing feature map

Abstract

The present disclosure provides a method, device, electronic equipment and storage medium for processing feature maps. The specific implementation scheme is: based on the first feature map, generating a second feature map to be fused with the first feature map, and a third feature map different from the second feature map; fusing the first feature map, the second feature map and the third feature map to obtain an intermediate fusion feature map; based on the first feature map and the intermediate fusion feature map, generating first fusion weight data; based on the second feature map and the intermediate fusion feature map, generating second fusion weight data; based on the first fusion weight data and the second fusion weight data, fusing the first feature map and the second feature map to obtain a target fusion feature map.

Description

Technical Field

[0001] This disclosure relates to neural network modeling techniques, and more particularly to a method, apparatus, electronic device, and storage medium for processing feature maps. Background Technology

[0002] Feature maps are a common data structure in deep learning. In some cases, it is necessary to fuse feature maps of different sizes. The fusion scheme is generally to perform a weighted average of the feature maps using predefined fusion weights. Summary of the Invention

[0003] According to one aspect of this disclosure, a method for processing feature maps is provided, comprising: generating a second feature map to be fused with the first feature map and a third feature map different from the second feature map based on a first feature map; fusing the first feature map, the second feature map, and the third feature map to obtain an intermediate fused feature map; generating first fused weight data based on the first feature map and the intermediate fused feature map; generating second fused weight data based on the second feature map and the intermediate fused feature map; and fusing the first feature map and the second feature map based on the first fused weight data and the second fused weight data to obtain a target fused feature map.

[0004] According to another aspect of this disclosure, an apparatus for processing feature maps is provided, comprising: a first generation module for generating a second feature map to be fused with the first feature map and a third feature map different from the second feature map, based on a first feature map; a first fusion module for fusing the first feature map, the second feature map, and the third feature map to obtain an intermediate fused feature map; a second generation module for generating first fusion weight data based on the first feature map and the intermediate fused feature map; a third generation module for generating second fusion weight data based on the second feature map and the intermediate fused feature map; and a second fusion module for fusing the first feature map and the second feature map based on the first fusion weight data and the second fusion weight data to obtain a target fused feature map.

[0005] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program for performing the above-described method for processing feature maps.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the method described above for processing feature maps. Attached Figure Description

[0007] Figure 1 This is an architecture diagram of a neural network model in some exemplary embodiments of this disclosure.

[0008] Figure 2 This is a flowchart illustrating a method for processing feature maps provided by some exemplary embodiments of this disclosure.

[0009] Figure 3 This is a flowchart illustrating a method for generating first fused weight data provided by some exemplary embodiments of this disclosure.

[0010] Figure 4 This is a flowchart illustrating a method for generating first fused weight data provided by some other exemplary embodiments of this disclosure.

[0011] Figure 5 This is a flowchart illustrating a method for transforming a first feature map into a first key tensor and a first value tensor, provided by some exemplary embodiments of this disclosure.

[0012] Figure 6 This is a flowchart illustrating a method for generating second fusion weight data provided by some exemplary embodiments of this disclosure.

[0013] Figure 7 This is a flowchart illustrating a method for transforming a second feature map into a second key tensor and a second value tensor, provided by some exemplary embodiments of this disclosure.

[0014] Figure 8 This is a flowchart illustrating a method for obtaining an intermediate fused feature map provided by some exemplary embodiments of this disclosure.

[0015] Figure 9 This is a flowchart illustrating a feature map generation method provided by some exemplary embodiments of this disclosure.

[0016] Figure 10 This is a flowchart illustrating a model parameter optimization method provided by some exemplary embodiments of this disclosure.

[0017] Figure 11 This is a schematic diagram of the structure of the twelfth subnetwork in some exemplary embodiments of this disclosure.

[0018] Figure 12-1 This is a schematic diagram of the structure of the weighted feature fusion module in some exemplary embodiments of this disclosure.

[0019] Figure 12-2 This is a schematic diagram of the structure of the feature interaction module in some exemplary embodiments of this disclosure.

[0020] Figure 12-3 This is a schematic diagram of the feature balancing module in some exemplary embodiments of this disclosure.

[0021] Figure 13This is a schematic diagram of the structure of an apparatus for processing feature maps provided by some exemplary embodiments of this disclosure.

[0022] Figure 14 This is a schematic diagram of the structure of the second generation module in some exemplary embodiments of this disclosure.

[0023] Figure 15 This is a schematic diagram of the structure of the second generation submodule in some exemplary embodiments of this disclosure.

[0024] Figure 16 This is a schematic diagram of the structure of the first transformation submodule in some exemplary embodiments of this disclosure.

[0025] Figure 17 This is a schematic diagram of the structure of the third generation module in some exemplary embodiments of this disclosure.

[0026] Figure 18 This is a schematic diagram of the structure of the third transformation submodule in some exemplary embodiments of this disclosure.

[0027] Figure 19 This is a schematic diagram of the structure of the first fusion module in some exemplary embodiments of this disclosure.

[0028] Figure 20-1 This is a schematic diagram of a module used to assist in the generation of feature maps in some exemplary embodiments of this disclosure.

[0029] Figure 20-2 This is a schematic diagram of a module used to assist in the optimization of model parameters in some exemplary embodiments of this disclosure.

[0030] Figure 21 This is a schematic diagram of the structure of an electronic device provided by some exemplary embodiments of this disclosure. Detailed Implementation

[0031] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0032] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0033] Exemplary Overview

[0034] In some cases, it is necessary to fuse feature maps of different sizes. For example, a neural network model generates high-resolution and low-resolution feature maps during operation. These high-resolution and low-resolution feature maps can be fused, and the model's results can be obtained based on the fused feature map.

[0035] Optionally, the neural network model can be a semantic segmentation model, and correspondingly, the model's output can be a semantic segmentation result. Since high-resolution feature maps retain rich shallow texture information, which is beneficial for accurately identifying object boundaries, and low-resolution feature maps possess deep semantic information, which can determine the category of the segmented object, fusing high-resolution and low-resolution feature maps and generating semantic segmentation results based on the resulting fused feature map helps ensure the accuracy and reliability of the semantic segmentation results.

[0036] Of course, the types of neural network models are not limited to this. For example, neural network models can also be object detection models, object tracking models, etc., and correspondingly, the model's output can be object detection results, object tracking results, etc.

[0037] In related technologies, when fusing feature maps of different sizes, the typical fusion scheme involves using manually set fusion weights to perform a weighted average of the feature maps. The inventors have found that this fusion scheme has poor fusion performance, and how to improve the fusion effect is a problem worthy of attention for those skilled in the art.

[0038] Exemplary System

[0039] Figure 1 This is an architectural diagram of a neural network model in some exemplary embodiments of this disclosure. For example... Figure 1 As shown, a neural network model can include a backbone network and a prediction network. The prediction network can include a recovery branch (RB) and a control branch (CB).

[0040] Backbone networks can extract features from the original image input to a neural network model. As an example, a backbone network can be a residual neural network (ResNet), which can extract features from the original image at multiple scales.

[0041] The recovery branch can use upsampling operations to obtain an image with the same size as the original image that also contains the model's results. As an example, the recovery branch can first perform three 2x upsampling operations, followed by one 4x upsampling operation, to obtain an image with the same size as the original image that also contains the model's results.

[0042] It should be noted that the backbone network can generate a first feature map during operation, and the recovery branch can generate a second feature map to be fused with the first feature map during operation. Alternatively, the input of the recovery branch can be used as the second feature map to be fused with the first feature map. One of the first and second feature maps can be a high-resolution feature map, and the other can be a low-resolution feature map.

[0043] During operation, the control branch can generate a third feature map. Based on the third feature map, the control branch can generate first fusion weight data corresponding to the first feature map and second fusion weight data corresponding to the second feature map. Based on the first and second fusion weight data, the control branch can fuse the first and second feature maps to obtain the target fused feature map.

[0044] As can be seen, in the embodiments of this disclosure, the fusion of feature maps of different sizes is not done by applying pre-defined fusion weight data, but by applying fusion weight data determined according to the actual fusion scenario. The fusion result can be adapted to the actual fusion scenario, which helps to ensure the fusion effect.

[0045] Exemplary methods

[0046] Figure 2 This is a flowchart illustrating a method for processing feature maps provided by some exemplary embodiments of this disclosure. Figure 2 The method shown may include steps 210, 220, 230, 240 and 250.

[0047] Step 210: Based on the first feature map, generate a second feature map to be fused with the first feature map, and a third feature map that is different from the second feature map.

[0048] In some optional embodiments of this disclosure, the first feature map may be a feature map generated by the backbone network during operation. The second feature map may be a feature map to be fused with the first feature map, generated by the recovery branch during operation. Alternatively, the second feature map may be the input of the recovery branch. The size of the second feature map may be different from the size of the first feature map. For example, one of the first and second feature maps may be a high-resolution feature map, and the other may be a low-resolution feature map. Assuming the size of the first feature map is represented as H1*W1*C1, and the size of the second feature map is represented as H2*W2*C2, then we have: H2 / H1 = W2 / W1. The third feature map may be a feature map different from the second feature map, generated by the control branch during operation. As an example, the size of the third feature map may be the same as the size of the first feature map.

[0049] It should be noted that both the second and third feature maps can be generated based on the first feature map. Generally, neural network models can involve various computational logics, including but not limited to convolution, pooling, element-wise operations, batch normalization (BN), rectified linear unit (ReLU) operations, upsampling, and downsampling. The set of various computational logics involved in a neural network model can be referred to as the computational logic set. As an example, the second feature map can be a feature map obtained by performing various computational logics from a subset of the computational logic set on the first feature map, and the third feature map can be a feature map obtained by performing various computational logics from another subset of the computational logic set on the first feature map.

[0050] Step 220: Fuse the first feature map, the second feature map, and the third feature map to obtain an intermediate fused feature map.

[0051] In some optional embodiments of this disclosure, the first feature map, the second feature map, and the third feature map can be processed first. Figure 3 The dimensions of the three feature maps are standardized to obtain three feature maps of the same size. Then, these three feature maps are fused using feature concatenation or element-wise operations to obtain an intermediate fused feature map. For example, the size of the intermediate fused feature map can be the same as the size of the first feature map. Element-wise operations can include, but are not limited to, element-wise multiplication and element-wise addition.

[0052] Step 230: Generate first fusion weight data based on the first feature map and the intermediate fusion feature map.

[0053] In some optional embodiments of this disclosure, the first fusion weight data may be a weight matrix obtained after performing various computational logics on a subset of the computational logic set on the first feature map and the intermediate fusion feature map. The first fusion weight data can be considered as the fusion weight data corresponding to the first feature map. As an example, the size of both the first feature map and the size of the first fusion weight data may be H1*W1*C1. Then, the first feature map may include H1*W1*C1 feature values, and the first fusion weight data may include H1*W1*C1 weight values. Furthermore, the H1*W1*C1 weight values ​​included in the first fusion weight data can correspond one-to-one with the H1*W1*C1 feature values ​​included in the first feature map.

[0054] Step 240: Generate second fusion weight data based on the second feature map and the intermediate fusion feature map.

[0055] In some optional embodiments of this disclosure, the second fusion weight data can be a weight matrix obtained after performing various computational logics on a subset of the computational logic set on the first feature map and the intermediate fusion feature map. The second fusion weight data can be considered as the fusion weight data corresponding to the second feature map. As an example, the size of the second feature map can be H2*W2*C2, and the size of the second fusion weight data can be H1*W1*C1. Then, the second feature map can include H2*W2*C2 feature values, and the second fusion weight data can include H1*W1*C1 weight values. In this case, the H1*W1*C1 weight values ​​included in the second fusion weight data cannot correspond one-to-one with the H2*W2*C2 feature values ​​included in the second feature map. However, by adjusting the size of the second feature map, another feature map with a size of H1*W1*C1 can be obtained, and the H1*W1*C1 weight values ​​included in the second fusion weight data can correspond one-to-one with the H1*W1*C1 feature values ​​included in this other feature map.

[0056] Step 250: Based on the first fusion weight data and the second fusion weight data, fuse the first feature map and the second feature map to obtain the target fusion feature map.

[0057] In some optional embodiments of this disclosure, the sizes of the first feature map and the second feature map can be unified first to obtain two feature maps with the same size, for example, two feature maps with the same size of H1*W1*C1. Then, the two feature maps with the same size are weighted and averaged using the first fusion weight data and the second fusion weight data to obtain the target fused feature map.

[0058] In the embodiments of this disclosure, an intermediate fused feature map can be obtained by fusing the first feature map, the second feature map, and the third feature map. It should be noted that the first and second feature maps are two feature maps to be fused, and the third feature map is a feature map generated based on the first feature map. Therefore, the first, second, and intermediate fused feature maps are all closely related to the actual fusion scenario. Since the intermediate fused feature map and the first feature map serve as the basis for generating the first fusion weight data, and the intermediate and second feature maps serve as the basis for generating the second fusion weight data, both the first and second fusion weight data can be adapted to the actual fusion scenario. Thus, using the first and second fusion weight data for the fusion of the first and second feature maps can better guarantee the fusion effect. It is evident that in the embodiments of this disclosure, for the fusion of feature maps of different sizes, the fusion weight data applied is not a pre-set fusion weight data, but rather fusion weight data adapted to the actual fusion scenario, which is beneficial for ensuring the fusion effect.

[0059] Figure 3This is a flowchart illustrating a method for generating first fused weight data provided by some exemplary embodiments of this disclosure. Figure 3 The method shown may include steps 310, 320, 330, and 340. Optionally, a combination of steps 310 to 340 may be used as an alternative implementation of step 230 of this disclosure.

[0060] Steps 310 to 340 will be described in detail below.

[0061] Step 310: Transform the first feature map into a first key tensor and a first value tensor.

[0062] In some optional embodiments of this disclosure, the control branch may include a Weighted Feature Fusion Module (WFFM). The WFFM module can be used to generate fusion weight data. There may be multiple WFFM modules; the following mainly describes the structure and working principle of a single WFFM module. The WFFM module may include a first transformation network, which, based on the first transformation network and several preset weight matrices, can transform the first feature map into a first key tensor and a first value tensor.

[0063] Step 320: Transform the intermediate fused feature map into the first query tensor.

[0064] In some optional embodiments of this disclosure, a preset weight matrix can be used to transform the intermediate fused feature map into a first query tensor. For example, the preset weight matrix can be multiplied by the intermediate fused feature map to obtain the first query tensor.

[0065] Step 330: Based on the first key tensor, the first value tensor, and the first query tensor, the first query result is generated via the first cross-attention network.

[0066] In some optional embodiments of this disclosure, the first key tensor, the first value tensor, and the first query tensor can all be used as inputs to the first cross-attention network. The first cross-attention network can perform calculations based on these inputs to generate a first query result. The first query result can also be in tensor form.

[0067] Step 340: Based on the first query result and the intermediate fusion feature map, generate the first fusion weight data via the first generation network.

[0068] In some optional embodiments of this disclosure, the weighted feature fusion module may include a first generator network in addition to the first transform network. After generating the first query result using the first cross-attention network, both the first query result and the intermediate fusion feature map can be used as inputs to the first generator network, which can then perform calculations based on these inputs to generate the first fusion weight data.

[0069] In some optional embodiments of this disclosure, the first generating network may include multiple network layers, such as a first network layer, a second network layer, and a third network layer. Accordingly, as... Figure 4 As shown, generating first fusion weight data based on the first query result and intermediate fusion feature map via the first generation network may include steps 510, 520 and 530.

[0070] Step 510: The first query result and the intermediate fused feature map are added element by element through the first network layer in the first generator network to obtain the fourth feature map.

[0071] Step 520: Convolve the fourth feature map through the second network layer in the first generator network to obtain the fifth feature map.

[0072] Step 530: The fifth feature map is non-linearly mapped through the third network layer in the first generator network to obtain the first fusion weight data.

[0073] Optionally, the first network layer may include an element-wise addition layer. If the size of the first query result is the same as the size of the intermediate fused feature map, the first query result and the intermediate fused feature map can be directly added element-wise through the element-wise addition layer to obtain the fourth feature map. If the size of the first query result is different from the size of the intermediate fused feature map, the sizes of the first query result and the intermediate fused feature map can be unified first, and then element-wise addition can be performed to obtain the fourth feature map.

[0074] Optionally, the second network layer may include a convolutional layer. This convolutional layer may use, for example, a 1x1 convolutional kernel. This convolutional layer can convolve the fourth feature map to obtain the fifth feature map.

[0075] Optionally, the third network layer may include an activation layer. This activation layer could be, for example, a sigmoid layer, a softmax layer, etc. This activation layer can perform a non-linear mapping on the fifth feature map. As an example, this activation layer can non-linearly map each feature value in the fifth feature map to the range of 0 to 1, thereby obtaining the first fusion weight data.

[0076] In this way, based on the first query result and the intermediate fusion feature map, the first fusion weight data can be generated efficiently and reliably through operations such as element-wise addition, convolution, and nonlinear mapping.

[0077] Of course, the structure of the first generator network is not limited to this. For example, the element-wise addition layer in the first network layer can be replaced with an element-wise multiplication layer. Furthermore, other networks besides element-wise addition layers, convolutional layers, and activation layers can be introduced into the first generator network.

[0078] In the embodiments of this disclosure, by transforming the first feature map into a first key tensor and a first value tensor, and transforming the intermediate fused feature map into a first query tensor, and then combining this with the application of an attention mechanism, the first query result can be generated efficiently and reliably. Based on this, by combining the intermediate fused feature map and the first generative network, the first fused weight data can be generated efficiently and reliably. Because the generation process of the first fused weight data effectively utilizes the first feature map and the intermediate fused feature map that are closely related to the actual fusion scenario, the first fused weight data can have a high degree of matching with the actual fusion scenario.

[0079] Figure 5 This is a flowchart illustrating a method for transforming a first feature map into a first key tensor and a first value tensor, provided by some exemplary embodiments of this disclosure. Figure 5 The method shown may include steps 610, 620, 630 and 640. Figure 5 In the method shown, the first transformation network may include a first sub-network and a second sub-network.

[0080] Step 610: Encode the first feature map through the first sub-network in the first transformation network to obtain the sixth feature map.

[0081] Step 620: The sixth feature map is sequentially convolved and pooled through the second sub-network in the first transformation network to obtain the seventh feature map.

[0082] In some optional embodiments of this disclosure, the first sub-network may include an attention network, a feed-forward neural network (FFN), and two element-wise additive layers. For ease of description, one of the two element-wise additive layers in the first sub-network may be referred to as the first element-wise additive layer, and the other as the second element-wise additive layer. The attention network can update the first feature map. The first element-wise additive layer can perform element-wise addition on the first feature map before and after the update to obtain an element-wise additive result. This element-wise additive result can be input into the feed-forward neural network. The second element-wise additive layer can perform element-wise addition on the element-wise additive result and the output of the feed-forward neural network to obtain another element-wise additive result. This other element-wise additive result can be used as a sixth feature map obtained after encoding the first feature map.

[0083] In some optional embodiments of this disclosure, the second sub-network may include a convolutional layer and a pooling layer. The sixth feature map can be sequentially convolved and pooled using the convolutional layer and the pooling layer to obtain the seventh feature map.

[0084] Step 630: Using the first preset weight matrix, transform the seventh feature map into the first key tensor.

[0085] In some optional embodiments of this disclosure, the first preset weight matrix can be multiplied by the seventh feature map to obtain the first key tensor transformed from the seventh feature map.

[0086] Step 640: Using the second preset weight matrix, the seventh feature map is transformed into a first value tensor.

[0087] In some optional embodiments of this disclosure, the second preset weight matrix can be multiplied by the seventh feature map to obtain the first value tensor transformed from the seventh feature map.

[0088] In the embodiments of this disclosure, based on the first feature map, the seventh feature map can be obtained efficiently and reliably through operations such as encoding, convolution, and pooling. By combining the application of the first preset weight matrix and the second preset weight matrix, the transformation of the seventh feature map can be realized, thereby obtaining the first key tensor and the first value tensor as the transformation result of the first feature map efficiently and reliably.

[0089] Figure 6 This is a flowchart illustrating a method for generating second fusion weight data provided by some exemplary embodiments of this disclosure. Figure 6The method shown may include steps 710, 720, 730, and 740. Optionally, a combination of steps 710 to 740 may be used as an alternative implementation of step 240 of this disclosure.

[0090] Step 710: Transform the second feature map into a second bond tensor and a second value tensor.

[0091] In some optional embodiments of this disclosure, the weighted feature fusion module may include a second transformation network, which, based on the second transformation network and several preset weight matrices, can transform the second feature map into a second key tensor and a second value tensor.

[0092] Step 720: Transform the intermediate fused feature map into the second query tensor.

[0093] In some optional embodiments of this disclosure, a preset weight matrix can be used to transform the intermediate fused feature map into a second query tensor. For example, the preset weight matrix can be multiplied by the intermediate fused feature map to obtain the second query tensor. Here, the preset weight matrix used to obtain the second query tensor can be a different weight matrix from the preset weight matrix used to obtain the first query tensor in step 320.

[0094] Step 730: Based on the second key tensor, the second value tensor, and the second query tensor, the second query result is generated via the second cross-attention network.

[0095] Step 740: Based on the second query result and the intermediate fusion feature map, generate the second fusion weight data via the second generation network.

[0096] Optionally, the specific implementation of steps 730 to 740 can be referred to the relevant description of steps 330 to 340 above, and will not be repeated here.

[0097] In the embodiments of this disclosure, by transforming the second feature map into a second key tensor and a second value tensor, and transforming the intermediate fusion feature map into a second query tensor, and then combining this with the application of an attention mechanism, the second query result can be generated efficiently and reliably. Based on this, by combining the intermediate fusion feature map and the second generation network, the second fusion weight data can be generated efficiently and reliably. Because the generation process of the second fusion weight data effectively utilizes the second feature map and the intermediate fusion feature map that are closely related to the actual fusion scenario, the second fusion weight data can have a high degree of matching with the actual fusion scenario.

[0098] Figure 7 This is a flowchart illustrating a method for transforming a second feature map into a second key tensor and a second value tensor, provided by some exemplary embodiments of this disclosure. Figure 7The method shown may include steps 810, 820, 830, 840 and 850. Figure 7 In the method shown, the second transformation network may include a third sub-network, a fourth sub-network, and a fifth sub-network.

[0099] Step 810: The second feature map is sequentially convolved, batch normalized, and upsampled through the third sub-network in the second transformation network to obtain the eighth feature map.

[0100] Step 820: The eighth feature map is encoded through the fourth sub-network in the second transformation network to obtain the ninth feature map.

[0101] Step 830: The ninth feature map is sequentially convolved and pooled through the fifth sub-network in the second transformation network to obtain the tenth feature map.

[0102] In some optional embodiments of this disclosure, the third sub-network may include a convolutional layer, a batch normalization layer, and an upsampling layer. Through this convolutional layer, the batch normalization layer, and the upsampling layer, the second feature map can be sequentially convolved, batch normalized, and upsampled to obtain the eighth feature map. Here, the upsampling can be, for example, a doubling upsampling.

[0103] In some optional embodiments of this disclosure, the structure of the fourth sub-network can refer to the relevant description of the structure of the first sub-network above, and the structure of the fifth sub-network can refer to the relevant description of the structure of the second sub-network above, and will not be repeated here.

[0104] Step 840: Using the third preset weight matrix, the tenth feature map is transformed into the second key tensor.

[0105] In some optional embodiments of this disclosure, the third preset weight matrix can be multiplied by the tenth feature map to obtain the second key tensor transformed from the tenth feature map.

[0106] Step 850: Using the fourth preset weight matrix, the tenth feature map is transformed into a second value tensor.

[0107] In some alternative embodiments of this disclosure, the fourth preset weight matrix can be multiplied by the tenth feature map to obtain the second value tensor transformed from the tenth feature map.

[0108] In the embodiments of this disclosure, based on the second feature map, the tenth feature map can be obtained efficiently and reliably through operations such as convolution, batch normalization, upsampling, encoding, and pooling. Combined with the application of the third and fourth preset weight matrices, the transformation of the tenth feature map can be realized, thereby obtaining the second key tensor and the second value tensor as the transformation result of the second feature map efficiently and reliably.

[0109] Figure 8 This is a flowchart illustrating a method for obtaining an intermediate fused feature map provided by some exemplary embodiments of this disclosure. Figure 8 The method shown may include steps 910, 920, and 930. Optionally, a combination of steps 910 to 930 may be used as an alternative implementation of step 220 of this disclosure. Figure 8 In the method shown, the fusion network may include a sixth sub-network, a fourth network layer, and a seventh sub-network. Optionally, the fusion network may be located in a weighted feature fusion module.

[0110] Step 910: The first feature map, the second feature map, and the third feature map are processed through the sixth sub-network in the fusion network to obtain three processed feature maps of the same size.

[0111] Step 920: The three processed feature maps are added element-wise through the fourth network layer in the fusion network to obtain the eleventh feature map.

[0112] Step 930: Through the seventh sub-network in the fusion network, the eleventh feature map is sequentially subjected to convolution, batch normalization, and modified linear unit operations to obtain the intermediate fused feature map.

[0113] In some optional embodiments of this disclosure, the sixth sub-network may include a first part network, a second part network, and a third part network. The first part network may include two convolutional layers, two batch normalization layers, an element-wise addition layer, and a modified linear unit layer. For ease of description, one of the two convolutional layers in the first part network may be referred to as the first convolutional layer, and the other as the second convolutional layer. One of the two batch normalization layers in the first part network may be referred to as the first batch normalization layer, and the other as the second batch normalization layer. Through the first convolutional layer and the first batch normalization layer, the first feature map can be sequentially convolved and batch normalized. Through the element-wise addition layer, the output of the first batch normalization layer and the first feature map can be sequentially added element-wise to obtain an element-wise addition result. This element-wise addition result can be input into the modified linear unit layer. Through the second convolutional layer and the second batch normalization layer, the output of the modified linear unit layer can be sequentially convolved and batch normalized, thereby obtaining the processed feature map corresponding to the first feature map. The second part of the network can include a convolutional layer and a batch normalization layer. Through this convolutional layer and batch normalization layer, the second feature map can be sequentially convolved and batch normalized to obtain the processed feature map corresponding to the second feature map. The third part of the network is similar in composition to the first part, with the main difference being that it additionally includes an upsampling layer. After performing the various operations involved in the first part of the network on the third feature map, this upsampling layer can upsample the final result, for example, by a factor of two, to obtain the processed feature map corresponding to the third feature map. Thus, the first, second, and third feature maps each correspond to a processed feature map, and these processed feature maps can have the same size.

[0114] In some optional embodiments of this disclosure, the fourth network layer may include an element-wise addition layer. This element-wise addition layer can be used to add the processed feature maps corresponding to the first, second, and third feature maps element-wise to obtain the eleventh feature map.

[0115] In some optional embodiments of this disclosure, the seventh sub-network may include a convolutional layer, a batch normalization layer, and a modified linear unit layer. Through the convolutional layer, the batch normalization layer, and the modified linear unit layer, the eleventh feature map can be sequentially subjected to convolution, batch normalization, and modified linear unit operations to obtain an intermediate fused feature map.

[0116] In the embodiments of this disclosure, by unifying the size of the first feature map, the second feature map, and the third feature map, and then through operational logic such as element-wise addition, convolution, batch normalization, and modified linear unit operations, the fusion of the first feature map, the second feature map, and the third feature map can be achieved efficiently and reliably.

[0117] Figure 9 This is a flowchart illustrating a feature map generation method provided by some exemplary embodiments of this disclosure. Figure 9 The method shown may include steps 1010, 1020, 1030, 1040, and 1050. Optionally, steps 1010 and 1020 may be performed before step 210 of this disclosure. A combination of steps 1030 to 1050 may be an optional implementation of step 210 of this disclosure. Figure 9 In the method shown, the backbone network may include an eighth and a ninth subnetwork. The control branch may include a tenth subnetwork. The recovery branch may include an eleventh subnetwork.

[0118] Step 1010: Through the eighth sub-network in the backbone network, feature extraction is performed on the original image to obtain multiple feature extraction images of different sizes.

[0119] Step 1020: Determine the first feature map from multiple feature extraction maps.

[0120] Step 1030: Through the ninth sub-network in the backbone network, at least some of the feature extraction maps in the multiple feature extraction maps are aggregated to obtain the twelfth feature map.

[0121] Step 1040: Using the tenth subnetwork in the prediction network, upsampling is performed step by step, with the feature extraction map generated most recently among the multiple feature extraction maps as the starting point.

[0122] Step 1050: Using the eleventh sub-network in the prediction network, upsampling is performed step by step, starting with the twelfth feature map. Among the feature maps obtained by the tenth sub-network through step-by-step upsampling, the feature map with the same size as the first feature map is used as the third feature map. Among the feature maps obtained by the twelfth feature map and the eleventh sub-network through step-by-step upsampling, the feature map with the size second only to the first feature map is used as the second feature map. Furthermore, the first feature map, the second feature map, and the third feature map are used as the input for one level of upsampling in the eleventh sub-network, and the target fusion feature map is used as the output for that level of upsampling in the eleventh sub-network.

[0123] In some optional embodiments of this disclosure, the eighth sub-network may include a multi-scale feature extraction network. This multi-scale feature extraction network allows for multi-scale feature extraction of the original image, resulting in multiple feature extraction maps of varying sizes. Figure 1For example, by performing multi-scale feature extraction on the original image, we can obtain feature extraction maps with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image. The feature extraction map with a size of 1 / 4 can be used as the first feature map. Alternatively, the feature extraction map with a size of 1 / 8 or 1 / 16 can be used as the first feature map.

[0124] In some optional embodiments of this disclosure, the ninth sub-network may include a Feature Aggregation Module (FAM). This module can aggregate features from at least a portion of multiple feature extraction maps to obtain a twelfth feature map. For example, multiple feature extraction maps can be pooled separately, and the resulting pooled feature extraction maps can have the same size. Then, the pooled feature extraction maps can be aggregated together through feature concatenation, element-wise operations, etc., to obtain the twelfth feature map. The size of the twelfth feature map can be 1 / 32 of the original image. Specific implementations of feature aggregation may include, but are not limited to, Spatial Pyramid Pooling (SPP), Pyramid Pooling Module (PPM), and Deep Aggregation Pyramid Pooling Module (DAPPM).

[0125] In some optional embodiments of this disclosure, the feature extraction map generated most recently among multiple feature extraction maps can be... Figure 1 The shown size is a feature map extracted at 1 / 32 of the original image. Through the tenth sub-network, this 1 / 32 feature map can be used as the starting point for upsampling, and upsampling can be performed step-by-step. For example, the first level of upsampling at the tenth sub-network yields a feature map at 1 / 16 of the original image. The second level of upsampling at the tenth sub-network yields a feature map at 1 / 8 of the original image. The third level of upsampling at the tenth sub-network yields a feature map at 1 / 4 of the original image. It should be noted that the size of the third feature map can be the same as the size of the first feature map. For example, if the first feature map is a feature map extracted at 1 / 16 of the original image, the third feature map could be the feature map obtained through the first level of upsampling at the tenth sub-network. As another example, if the first feature map is a feature map extracted at 1 / 8 of the original image, the third feature map could be the feature map obtained through the second level of upsampling at the tenth sub-network.

[0126] In some optional embodiments of this disclosure, the eleventh sub-network can use the twelfth feature map as the starting point for upsampling and perform progressive upsampling. As an example, through the first level of upsampling at the eleventh sub-network, a feature map with a size 1 / 16 of the original image can be obtained. Through the second level of upsampling at the eleventh sub-network, a feature map with a size 1 / 8 of the original image can be obtained. Through the third level of upsampling at the eleventh sub-network, a feature map with a size 1 / 4 of the original image can be obtained. It should be noted that the second feature map can be the feature map whose size is second only to the first feature map among the feature maps obtained by progressive upsampling from the twelfth feature map and the eleventh sub-network. For example, if the first feature map is a feature extraction map with a size 1 / 16 of the original image, then the second feature map can be the twelfth feature map. As another example, if the first feature map is a feature extraction map with a size 1 / 8 of the original image, then the second feature map can be the feature map obtained through the first level of upsampling. For the case where the first feature map is a feature extraction map with a size of 1 / 16 of the original image, the second feature map can be a feature map with a size of 1 / 32 of the original image, and the third feature map can be a feature map with a size of 1 / 16 of the original image. By fusing the first feature map, the second feature map, and the third feature map, a target fused feature map with a size of 1 / 16 of the original image can be obtained. The target fused feature map with a size of 1 / 16 of the original image can be regarded as the output of the first-level upsampling at the eleventh sub-network, and can also be regarded as the input of the second-level upsampling at the eleventh sub-network. Next, we can consider the feature extraction map, which is 1 / 8 the size of the original image, as the new first feature map; the target fusion feature map, which is 1 / 16 the size of the original image, as the new second feature map; and the feature map obtained by second-level upsampling at the tenth sub-network, which is also 1 / 8 the size of the original image, as the new third feature map. By fusing the new first, second, and third feature maps, we can obtain the target fusion feature map, which is 1 / 8 the size of the original image. This target fusion feature map can be considered as the output of the second-level upsampling at the eleventh sub-network, and also as the input of the third-level upsampling at the eleventh sub-network. Subsequent processes follow the same logic and will not be elaborated further here.

[0127] In the embodiments of this disclosure, feature aggregation is performed, which increases the receptive field of the network and yields better global information. The twelfth feature map obtained through feature aggregation is used as the upsampling starting point at the eleventh sub-network, and the second feature map and fusion weight data are determined based on this. This helps ensure the rationality and reliability of the fusion weight data. Furthermore, in the embodiments of this disclosure, since stepwise upsampling is performed at both the tenth and eleventh sub-networks, multiple feature map fusions can be performed during this process. Each fusion uses fusion weight data adapted to the actual fusion scenario, which helps ensure the accuracy and reliability of the neural network model's operating results.

[0128] Figure 10 This is a flowchart illustrating a model parameter optimization method provided by some exemplary embodiments of this disclosure. Figure 10 The method shown may include steps 1110, 1120, 1130, 1140 and 1150. Figure 10 In the method shown, both the backbone network and the prediction network can be located within the semantic segmentation model.

[0129] Step 1110: By using the twelfth subnetwork in the prediction network, the feature map obtained by upsampling the last level in the tenth subnetwork is upsampled to obtain the first semantic segmentation result of the original image.

[0130] Step 1120: By using the thirteenth subnetwork in the prediction network, the feature map obtained by upsampling the last level in the eleventh subnetwork is upsampled to obtain the second semantic segmentation result of the original image.

[0131] In some alternative embodiments of this disclosure, the control branch may include a twelfth subnetwork. The structure of the twelfth subnetwork can be referred to... Figure 11 Includes two batch normalization layers (corresponding to) Figure 11 In BN), two modified linear unit layers (corresponding to) Figure 11 ReLU in the middle), two convolutional layers (corresponding to Figure 11 Conv in the middle) and a four-fold upsampling layer (corresponding to Figure 11 (UP 4x in the example). For ease of description, one of the two batch normalization layers in the twelfth sub-network can be called the third batch normalization layer, and the other can be called the fourth batch normalization layer. One of the two modified linear unit (MRU) layers in the twelfth sub-network can be called the first modified linear unit (MRU) layer, and the other can be called the second modified linear unit (MRU) layer. One of the two convolutional layers in the twelfth sub-network can be called the third convolutional layer, and the other can be called the fourth convolutional layer. The kernel size used in the third convolutional layer can be, for example, 3*3. The kernel size used in the fourth convolutional layer can be, for example, 1*1. Through the third batch normalization layer, the first modified linear unit (MRU) layer, the third convolutional layer, the fourth batch normalization layer, the second modified linear unit (MRU) layer, the fourth convolutional layer, and the four-fold upsampling layer, the feature map obtained by upsampling at the last level in the tenth sub-network can be sequentially subjected to batch normalization, modified linear unit operation, convolution, batch normalization, modified linear unit operation, convolution, and four-fold upsampling to obtain the first semantic segmentation result of the original image. The first semantic segmentation result can include the predicted category of each pixel in the original image.

[0132] In some optional embodiments of this disclosure, the structure of the thirteenth subnetwork can be similar to that of the twelfth subnetwork. The method for obtaining the second semantic segmentation result is the same as the method for obtaining the first semantic segmentation result described above, and will not be repeated here. Optionally, the twelfth subnetwork can be represented as follows: Figure 1 In a SegHead, the thirteenth subnetwork can be represented as Figure 1 Another SegHead in the series.

[0133] Step 1130: Determine the first loss value based on the semantic segmentation annotation information of the original image and the first semantic segmentation result.

[0134] In some optional embodiments of this disclosure, the original image can be pre-annotated to include semantic segmentation annotation information. The semantic segmentation annotation information may include the annotation category of each pixel in the original image. Here, a preset loss function can be used to calculate the loss between the semantic segmentation annotation information and the first semantic segmentation result to obtain a first loss value. The preset loss function may include, but is not limited to, the Mean Absence Error (MAE) loss function, the Mean Square Error (MSE) loss function, etc.

[0135] Step 1140: Determine the second loss value based on the semantic segmentation annotation information of the original image and the second semantic segmentation result.

[0136] In some optional embodiments of this disclosure, a preset loss function can be used to calculate the loss on the semantic segmentation annotation information and the second semantic segmentation result to obtain a second loss value.

[0137] Step 1150: Optimize the parameters of the semantic segmentation model based on the first loss value and the second loss value.

[0138] In some optional embodiments of this disclosure, the first loss value and the second loss value can be averaged or weighted averaged, and the resulting average loss value or weighted average loss value can be used as the model loss value of the semantic segmentation model. Based on the model loss value, stochastic gradient descent, steepest gradient descent, or other methods can be used to optimize the parameters of the semantic segmentation model.

[0139] In the embodiments of this disclosure, during the model training phase, after obtaining the feature map obtained by upsampling at the last level in the tenth sub-network, the first semantic segmentation result of the original image can be obtained efficiently and reliably through the operation of the twelfth sub-network. After obtaining the feature map obtained by upsampling at the last level in the eleventh sub-network, the second semantic segmentation result of the original image can be obtained efficiently and reliably through the operation of the thirteenth sub-network. By comparing the semantic segmentation annotation information of the original image with each semantic segmentation result, the corresponding loss value can be obtained efficiently and reliably for parameter optimization of the semantic segmentation model. In this way, the semantic segmentation model can be optimized to a better state, which is beneficial to ensuring the accuracy of the semantic segmentation model during the model inference phase.

[0140] In some optional examples, Figure 1 The diagram shown may depict the architecture of a semantic segmentation model. The backbone network can perform multi-scale feature extraction on the original image to obtain multiple feature extraction maps. The backbone network may include a Feature Amplifier (FAM). Through the FAM, at least some of the feature extraction maps from the multiple feature extraction maps can be aggregated to obtain a twelfth feature map. The twelfth feature map can be used as input to the recovery branch. The feature extraction map generated most recently among the multiple feature extraction maps can be used as input to the control branch. From the multiple feature extraction maps, a first feature map can be reasonably determined. From the feature maps obtained through progressive upsampling from the recovery branch and the twelfth feature map, a second feature map can be reasonably determined. From the feature maps obtained through progressive upsampling from the control branch, a third feature map can be reasonably determined. Optionally, the first feature map can be represented as a Backbone Feature, the second feature map as an RB Feature, and the third feature map as a CB Feature.

[0141] Control branches may include Figure 12-1 The WFFM shown. A WFFM can include... Figure 12-2 The Feature Interaction Module (FIM) shown is... Figure 12-3The Feature Balance Module (FBM) shown is used. The FBM can include the sixth sub-network, the fourth network layer, and the seventh sub-network mentioned above. Through the FBM, the Backbone Feature, RB Feature, and CB Feature can be fused to obtain an intermediate fused feature map. This intermediate fused feature map can be represented as FIMFeature. The FBM can include a first cross-attention network, a first generator network, a second cross-attention network, and a second generator network. Through the FBM, fusion weight data corresponding to the Backbone Feature and RB Feature can be generated. The fusion weight data corresponding to the Backbone Feature can be represented as... Figure 12-1 , Figure 12-3 The fusion weight data corresponding to α.RB Feature can be represented as: Figure 12-1 , Figure 12-3 In the context of β, optionally, at the FBM, the Backbone Feature and RB Feature can be first passed through an attention-based encoding layer. This attention-based encoding layer could be, for example, a Transformer Encoder Layer. The Transformer Encoder Layer can include an External Attention network and an FFN. Each Transformer Encoder Layer can generate its own key tensor (corresponding to...). Figure 12-3 K) and value tensor (corresponding to Figure 12-3 (V in the original text). Additionally, two query tensors (corresponding to...) can be generated. Figure 12-3 The generated key tensor, value tensor, and query tensor can then be input into the corresponding cross-attention network. Following this, a linear layer (which can be a 1x1 convolutional layer) and a sigmoid layer are used to generate the fused weight data corresponding to the Backbone Feature and RB Feature, i.e., α and β. In some embodiments, the introduction of a softmax layer can ensure that the sum of the weights at corresponding positions of α and β is 1.

[0142] Optionally, during the model training phase, the control branch can generate semantic segmentation results to participate in training. During the model inference phase, the control branch may not generate semantic segmentation results; that is, the SegHead in the control branch may not run.

[0143] It should be noted that real-world images can be complex and diverse, and can be used as the original image. Based on the original image, the semantic segmentation model can generate feature maps of different sizes to be fused. If the feature maps are weighted and averaged according to the set fusion weight data, the differences and specificities of real-world images cannot be reflected. In the embodiments of this disclosure, fusion weight data closely related to the actual fusion scenario can be used to perform weighted averaging of feature maps, which is beneficial to improving the fusion effect of feature maps and reflecting the differences and specificities of real-world images.

[0144] Any of the methods for processing feature maps provided in the embodiments of this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the methods for processing feature maps provided in the embodiments of this disclosure can be executed by a processor, such as by a processor executing any of the methods for processing feature maps mentioned in the embodiments of this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0145] Exemplary device

[0146] Figure 13 This is a schematic diagram of the structure of an apparatus for processing feature maps provided by some exemplary embodiments of this disclosure. Figure 13 The apparatus shown includes: a first generation module 1410, used to generate a second feature map to be fused with the first feature map, and a third feature map different from the second feature map, based on the first feature map; a first fusion module 1420, used to fuse the first feature map, the second feature map, and the third feature map to obtain an intermediate fused feature map; a second generation module 1430, used to generate first fusion weight data based on the first feature map and the intermediate fused feature map; a third generation module 1440, used to generate second fusion weight data based on the second feature map and the intermediate fused feature map; and a second fusion module 1450, used to fuse the first feature map and the second feature map based on the first fusion weight data and the second fusion weight data to obtain a target fused feature map.

[0147] In some optional examples, such as Figure 14 As shown, the second generation module 1430 includes: a first transformation submodule 1510, used to transform the first feature map into a first key tensor and a first value tensor; a second transformation submodule 1520, used to transform the intermediate fused feature map into a first query tensor; a first generation submodule 1530, used to generate a first query result based on the first key tensor, the first value tensor, and the first query tensor via a first cross-attention network; and a second generation submodule 1540, used to generate first fused weight data based on the first query result and the intermediate fused feature map via a first generation network.

[0148] In some optional examples, such as Figure 15 As shown, the second generation submodule 1540 includes: a first processing unit 1610, used to perform element-wise addition of the first query result and the intermediate fusion feature map through the first network layer in the first generation network to obtain a fourth feature map; a second processing unit 1620, used to perform convolution on the fourth feature map through the second network layer in the first generation network to obtain a fifth feature map; and a third processing unit 1630, used to activate the fifth feature map through the third network layer in the first generation network to obtain first fusion weight data.

[0149] In some optional examples, such as Figure 16 As shown, the first transformation submodule 1510 includes: a fourth processing unit 1710, used to encode the first feature map through the first subnetwork in the first transformation network to obtain a sixth feature map; a fifth processing unit 1720, used to perform convolution and pooling on the sixth feature map through the second subnetwork in the first transformation network to obtain a seventh feature map; a first transformation unit 1730, used to transform the seventh feature map into a first key tensor using a first preset weight matrix; and a second transformation unit 1740, used to transform the seventh feature map into a first value tensor using a second preset weight matrix.

[0150] In some optional examples, such as Figure 17 As shown, the third generation module 1440 includes: a third transformation submodule 1810, used to transform the second feature map into a second key tensor and a second value tensor; a fourth transformation submodule 1820, used to transform the intermediate fused feature map into a second query tensor; a third generation submodule 1830, used to generate a second query result based on the second key tensor, the second value tensor, and the second query tensor via a second cross-attention network; and a fourth generation submodule 1840, used to generate second fused weight data based on the second query result and the intermediate fused feature map via a second generation network.

[0151] In some optional examples, such as Figure 18 As shown, the third transformation submodule 1810 includes: a sixth processing unit 1910, used to sequentially perform convolution, batch normalization, and upsampling on the second feature map through the third subnetwork in the second transformation network to obtain an eighth feature map; a seventh processing unit 1920, used to encode the eighth feature map through the fourth subnetwork in the second transformation network to obtain a ninth feature map; an eighth processing unit 1930, used to sequentially perform convolution and pooling on the ninth feature map through the fifth subnetwork in the second transformation network to obtain a tenth feature map; a third transformation unit 1940, used to transform the tenth feature map into a second key tensor using a third preset weight matrix; and a fourth transformation unit 1950, used to transform the tenth feature map into a second value tensor using a fourth preset weight matrix.

[0152] In some optional examples, such as Figure 19 As shown, the first fusion module 1420 includes: a first processing submodule 2010, used to process the first feature map, the second feature map, and the third feature map through the sixth subnetwork in the fusion network to obtain three processed feature maps of the same size; a second processing submodule 2020, used to perform element-wise addition of the three processed feature maps through the fourth network layer in the fusion network to obtain the eleventh feature map; and a third processing submodule 2030, used to perform convolution, batch normalization, and modified linear unit operations on the eleventh feature map sequentially through the seventh subnetwork in the fusion network to obtain an intermediate fused feature map.

[0153] In some optional examples, such as Figure 20-1 As shown, the apparatus provided in the embodiments of this disclosure further includes: a feature extraction module 2110, used to extract features from the original image through an eighth sub-network in the backbone network before the first generation module 1410 generates the first feature map, the second feature map, and the third feature map, to obtain multiple feature extraction maps of different sizes; a first determination module 2120, used to determine the first feature map from the multiple feature extraction maps; the first generation module 1410 includes: a fourth processing sub-module 2130, used to perform feature aggregation on at least a portion of the feature extraction maps in the multiple feature extraction maps through a ninth sub-network in the backbone network to obtain a twelfth feature map; and a fifth processing sub-module 2140, used to perform feature aggregation on the multiple feature extraction maps through a tenth sub-network in the prediction network. The feature map generated latest in the figure is used as the starting point for upsampling, and upsampling is performed step by step. The sixth processing submodule 2150 is used to perform step by step upsampling through the eleventh subnetwork in the prediction network, with the twelfth feature map as the starting point for upsampling. Among the feature maps obtained by the tenth subnetwork through step-by-step upsampling, the feature map with the same size as the first feature map is used as the third feature map. Among the feature maps obtained by the twelfth feature map and the eleventh subnetwork through step-by-step upsampling, the feature map with the size second only to the first feature map is used as the second feature map. Furthermore, the first feature map, the second feature map, and the third feature map are used as the input for one level of upsampling in the eleventh subnetwork, and the target fusion feature map is used as the output for that level of upsampling in the eleventh subnetwork.

[0154] In some optional examples, both the backbone network and the prediction network are located within the semantic segmentation model; such as Figure 20-2 As shown, the apparatus provided in the embodiments of this disclosure further includes:

[0155] The first processing module 2160 is used to upsample the feature map obtained by upsampling the last level of the tenth sub-network through the twelfth sub-network in the prediction network to obtain the first semantic segmentation result of the original image.

[0156] The second processing module 2170 is used to upsample the feature map obtained by upsampling the last level of the eleventh sub-network through the thirteenth sub-network in the prediction network to obtain the second semantic segmentation result of the original image.

[0157] The second determining module 2180 is used to determine the first loss value based on the semantic segmentation annotation information of the original image and the first semantic segmentation result;

[0158] The third determining module 2190 is used to determine the second loss value based on the semantic segmentation annotation information of the original image and the second semantic segmentation result;

[0159] The parameter optimization module 2195 is used to optimize the parameters of the semantic segmentation model based on the first loss value and the second loss value.

[0160] In the apparatus disclosed herein, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve the corresponding functions and effects, and this disclosure does not list them all.

[0161] Exemplary electronic devices

[0162] Figure 21 The illustration shows a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 2200 includes one or more processors 2210 and memory 2220.

[0163] The processor 2210 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 2200 to perform desired functions.

[0164] The memory 2220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 2210 may execute one or more computer program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions.

[0165] In one example, the electronic device 2200 may also include an input device 2230 and an output device 2240, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0166] The input device 2230 may also include, for example, a keyboard, a mouse, etc.

[0167] The output device 2240 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0168] Of course, for the sake of simplicity, Figure 21 Only some of the components of the electronic device 2200 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 2200 may include any other suitable components depending on the specific application.

[0169] Exemplary computer program products and computer-readable storage media

[0170] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0171] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0172] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0173] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0174] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. The specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0175] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for processing feature maps, comprising: Based on the first feature map, a second feature map to be fused with the first feature map and a third feature map different from the second feature map are generated. The first feature map, the second feature map, and the third feature map are fused to obtain an intermediate fused feature map. Based on the first feature map and the intermediate fusion feature map, first fusion weight data is generated; Based on the second feature map and the intermediate fusion feature map, second fusion weight data is generated; Based on the first fusion weight data and the second fusion weight data, the first feature map and the second feature map are fused to obtain the target fusion feature map.

2. The method according to claim 1, wherein, The step of generating first fusion weight data based on the first feature map and the intermediate fusion feature map includes: Transform the first feature map into a first key tensor and a first value tensor; Transform the intermediate fused feature map into a first query tensor; Based on the first key tensor, the first value tensor, and the first query tensor, a first query result is generated via a first cross-attention network; Based on the first query result and the intermediate fusion feature map, the first fusion weight data is generated via the first generation network.

3. The method according to claim 2, wherein, The step of generating the first fusion weight data based on the first query result and the intermediate fusion feature map via a first generation network includes: The first query result and the intermediate fused feature map are added element by element through the first network layer in the first generation network to obtain the fourth feature map. The fourth feature map is convolved through the second network layer in the first generator network to obtain the fifth feature map; The fifth feature map is nonlinearly mapped through the third network layer in the first generation network to obtain the first fused weight data.

4. The method according to claim 2, wherein, The step of transforming the first feature map into a first key tensor and a first value tensor includes: The first feature map is encoded by the first sub-network in the first transformation network to obtain the sixth feature map; The sixth feature map is sequentially convolved and pooled through the second sub-network in the first transformation network to obtain the seventh feature map. Using the first preset weight matrix, the seventh feature map is transformed into the first key tensor; The seventh feature map is transformed into the first value tensor using the second preset weight matrix.

5. The method according to claim 1, wherein, The step of generating second fusion weight data based on the second feature map and the intermediate fusion feature map includes: Transform the second feature map into a second key tensor and a second value tensor; Transform the intermediate fused feature map into a second query tensor; Based on the second key tensor, the second value tensor, and the second query tensor, a second query result is generated via a second cross-attention network; Based on the second query result and the intermediate fusion feature map, the second fusion weight data is generated via the second generation network.

6. The method according to claim 5, wherein, The step of transforming the second feature map into a second key tensor and a second value tensor includes: The second feature map is sequentially convolved, batch normalized, and upsampled through the third sub-network in the second transformation network to obtain the eighth feature map. The eighth feature map is encoded through the fourth sub-network in the second transformation network to obtain the ninth feature map; The ninth feature map is sequentially convolved and pooled through the fifth sub-network in the second transformation network to obtain the tenth feature map. Using the third preset weight matrix, the tenth feature map is transformed into the second key tensor; The tenth feature map is transformed into the second value tensor using the fourth preset weight matrix.

7. The method according to any one of claims 1-6, wherein, The process of fusing the first feature map, the second feature map, and the third feature map to obtain an intermediate fused feature map includes: By using the sixth sub-network in the fusion network, the first feature map, the second feature map, and the third feature map are processed to obtain three processed feature maps of the same size; The eleventh feature map is obtained by adding the three processed feature maps element by element through the fourth network layer of the fusion network. The eleventh feature map is obtained by sequentially performing convolution, batch normalization, and modified linear unit operations on the eleventh feature map through the seventh sub-network in the fusion network.

8. The method according to any one of claims 1-6, wherein, Before generating a second feature map to be fused with the first feature map based on the first feature map, and a third feature map different from the second feature map, the method further includes: The original image is feature extracted by the eighth sub-network in the backbone network, resulting in multiple feature extraction images of different sizes. The first feature map is determined from the multiple feature extraction maps; The step of generating a second feature map to be fused with the first feature map, and a third feature map different from the second feature map, based on the first feature map, includes: The twelfth feature map is obtained by performing feature aggregation on at least a portion of the feature extraction maps in the multiple feature extraction maps through the ninth sub-network in the backbone network. By using the tenth subnetwork in the prediction network, the feature extraction map with the latest generation time in the multiple feature extraction maps is used as the upsampling starting point, and upsampling is performed step by step. Through the eleventh subnetwork in the prediction network, and using the twelfth feature map as the upsampling starting point, step-by-step upsampling is performed; In this context, among the feature maps obtained by the tenth sub-network through step-by-step upsampling, the feature map with the same size as the first feature map is used as the third feature map; among the feature maps obtained by the twelfth and eleventh sub-networks through step-by-step upsampling, the feature map with a size second only to the first feature map is used as the second feature map; and the first feature map, the second feature map, and the third feature map serve as the input for one level of upsampling in the eleventh sub-network, while the target fusion feature map serves as the output for that level of upsampling in the eleventh sub-network.

9. The method according to claim 8, wherein, Both the backbone network and the prediction network are located in the semantic segmentation model; The method further includes: The feature map obtained by upsampling the last level of the tenth sub-network is upsampled through the twelfth sub-network of the prediction network to obtain the first semantic segmentation result of the original image. The feature map obtained by upsampling the last level of the eleventh sub-network is upsampled through the thirteenth sub-network of the prediction network to obtain the second semantic segmentation result of the original image. Based on the semantic segmentation annotation information of the original image and the first semantic segmentation result, a first loss value is determined; Based on the semantic segmentation annotation information of the original image and the second semantic segmentation result, a second loss value is determined; Based on the first loss value and the second loss value, the parameters of the semantic segmentation model are optimized.

10. An apparatus for processing a feature map, comprising: The first generation module is used to generate a second feature map to be fused with the first feature map, and a third feature map different from the second feature map, based on the first feature map. The first fusion module is used to fuse the first feature map, the second feature map, and the third feature map to obtain an intermediate fused feature map; The second generation module is used to generate first fusion weight data based on the first feature map and the intermediate fusion feature map; The third generation module is used to generate second fusion weight data based on the second feature map and the intermediate fusion feature map; The second fusion module is used to fuse the first feature map and the second feature map based on the first fusion weight data and the second fusion weight data to obtain a target fusion feature map.

11. An electronic device, comprising: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method for processing feature maps as described in any one of claims 1 to 9.

12. A computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the method for processing a feature map according to any one of claims 1 to 9.

Images