Image label generation method and device based on feature fusion, equipment and medium

Abstract

The application relates to the fields of artificial intelligence and medical health, and discloses an image label generation method based on feature fusion, which comprises the following steps: acquiring image data, generating a multi-scale local feature map through multi-scale feature extraction, extracting category attention information and constructing a category matrix; generating a gating signal through a gating unit, adjusting the feature map, extracting global category features and local category features, and fusing the two to generate a category attention guide feature map; and finally generating a category label map through weighted processing of the fused feature map. Through multi-scale feature extraction, dynamic adjustment of category attention information, and fusion of global and local features, the application improves the precision and adaptability of image segmentation, can more effectively identify target regions with large intra-class changes and small inter-class differences, and is suitable for medical image analysis, disaster assessment, financial claims and other fields.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and the field of medical health, and in particular to an image label generation method and device based on feature fusion, equipment and a storage medium. BACKGROUND

[0002] With the continuous progress of deep learning technology, the Transformer architecture has been widely used in semantic segmentation tasks due to its powerful self-attention mechanism. Semantic segmentation is an important task in computer vision, aiming to assign class labels to each pixel in an image, such as buildings, roads, vegetation, etc. This task can be considered as a pixel-level classification problem. However, existing semantic segmentation methods are mostly designed for natural scene images and are difficult to effectively handle the unique properties of remote sensing images, especially in scenarios with large intra-class variation and small inter-class variation, resulting in limited precision and generalization ability of image segmentation. Therefore, in practical applications, existing technologies have many shortcomings in image processing tasks in the fields of medical health and finance.

[0003] In the field of medical health, semantic segmentation technology is often used for automatic identification and analysis of medical images, such as lesion region labeling in CT, MRI, X-ray, etc. Existing technologies mainly rely on convolutional neural networks (CNN) to extract image features, but due to the variable shape and fuzzy boundary of lesion regions in medical images, it is difficult to accurately identify lesion regions relying solely on local features. Medical images often exhibit large intra-class variation and small inter-class variation, i.e., the same lesion may exhibit multiple morphologies, while the boundary between the lesion region and normal tissue is not obvious. In addition, existing segmentation methods lack effective combination of global information and local features, making it difficult to identify the scope of the lesion in a global perspective, while also being difficult to take into account the detailed changes in the image. These shortcomings make it difficult for existing image segmentation technology to meet the needs of precise identification and real-time performance in clinical diagnosis.

[0004] In the field of finance, image segmentation technology has a wide range of applications in disaster assessment and asset claims scenarios. For example, insurance companies often use images of disaster sites taken by drones to quickly identify buildings, roads, vegetation, and other targets in disaster-affected areas during the claims process after natural disasters. However, existing technologies have low recognition accuracy and large errors in practical applications, making it difficult to accurately extract detailed features of disaster-affected areas. Image data in disaster scenarios are often affected by factors such as occlusion, lighting changes, and environmental interference, making it difficult for traditional segmentation models to accurately identify target classes in complex scenarios. In addition, target objects in disaster scenarios may exhibit multi-scale distribution, including large-scale disaster-affected areas and small-scale detailed changes. Existing technologies are inadequate in handling these multi-scale features, making it difficult to effectively extract complete disaster information, affecting the efficiency of insurance companies' claims and the accuracy of loss assessment. SUMMARY

[0005] The main purpose of the present application is to provide a feature fusion-based image label generation method, device, equipment and storage medium, aiming at solving the technical problems of lack of effective extraction of class difference information and fusion of global and local features when the prior art processes images with large intra-class variation and small inter-class variation, resulting in insufficient image segmentation precision and adaptability.

[0006] To achieve the above purpose, the present application provides a feature fusion-based image label generation method, comprising:

[0007] Obtaining image data, performing multi-scale local feature extraction on the image data through a multi-scale feature extraction module to generate a multi-scale local feature map;

[0008] Extracting class attention information from the multi-scale local feature map, and constructing a class matrix based on the class weight and class index in the class attention information;

[0009] Generating a gating signal through a gating unit, adjusting the multi-scale local feature map through the gating signal, and performing channel-by-channel weighting processing on the adjusted multi-scale local feature map through the class matrix to generate a gated class feature map;

[0010] Extracting global class features from the gated class feature map through a global class attention guiding module;

[0011] Extracting local class features from the gated class feature map through a local class attention guiding module;

[0012] Fusing the global class features and the local class features to generate a class attention guiding feature map;

[0013] Fusing the class attention guiding feature map and the multi-scale local feature map, performing weighting processing on the fused feature map through a feature fusion head module to generate a final fused feature map;

[0014] Generating a class label map according to the final fused feature map.

[0015] Further, to achieve the above purpose, the present application provides a feature fusion-based image label generation device, comprising:

[0016] A multi-scale feature extraction module for obtaining image data, performing multi-scale local feature extraction on the image data through a multi-scale feature extraction module to generate a multi-scale local feature map;

[0017] The category attention extraction module is configured to extract category attention information from the multi-scale local feature map and construct a category matrix based on a category weight and a category index in the category attention information.

[0018] The gating unit module is configured to generate a gating signal by a gating unit, adjust the multi-scale local feature map by the gating signal, and perform channel-by-channel weighting processing on the adjusted multi-scale local feature map by the category matrix to generate a gated category feature map.

[0019] The global category attention guiding module is configured to extract a global category feature from the gated category feature map by a global category attention guiding module.

[0020] The local category attention guiding module is configured to extract a local category feature from the gated category feature map by a local category attention guiding module.

[0021] The category attention fusion module is configured to fuse the global category feature and the local category feature to generate a category attention guiding feature map.

[0022] The feature fusion module is configured to fuse the category attention guiding feature map and the multi-scale local feature map, perform weighting processing on the fused feature map by a feature fusion head module, and generate a final fused feature map.

[0023] The label generation module is configured to generate a category label map according to the final fused feature map.

[0024] Further, to achieve the above object, the present application further provides a computer device, which comprises a memory, a processor, and a feature fusion-based image label generation program stored in the memory and executable on the processor, and the feature fusion-based image label generation program, when executed by the processor, implements the steps of the feature fusion-based image label generation method.

[0025] Further, to achieve the above object, the present application further provides a computer-readable storage medium, which stores a feature fusion-based image label generation program, and the feature fusion-based image label generation program, when executed by a processor, implements the steps of the feature fusion-based image label generation method.

[0026] Beneficial effects: The application relates to the fields of artificial intelligence and medical health, and discloses an image label generation method based on feature fusion, which comprises the following steps: acquiring image data, generating a multi-scale local feature map through multi-scale feature extraction, extracting category attention information and constructing a category matrix; generating a gate signal through a gate unit, adjusting the feature map, extracting global category features and local category features, fusing the global category features and the local category features to generate a category attention guide feature map, and finally generating a category label map through weighted processing of the fused feature map. Through multi-scale feature extraction, dynamic adjustment of category attention information and fusion of global and local features, the precision and adaptability of image segmentation are improved, the target region with large intra-class variation and small inter-class difference can be more effectively recognized, and the method is suitable for medical image analysis, disaster assessment and financial claim settlement and the like. BRIEF DESCRIPTION OF DRAWINGS

[0027] The application will be further described below in combination with the drawings and examples, wherein:

[0028] Figure 1 An application environment diagram of the image label generation method based on feature fusion in an embodiment of the application;

[0029] Figure 2 A flowchart of the image label generation method based on feature fusion in an embodiment of the application;

[0030] Figure 3 A functional module diagram of a preferred embodiment of the image label generation device based on feature fusion of the application;

[0031] Figure 4 A structural diagram of a computer device in an embodiment of the application;

[0032] Figure 5 Another structural diagram of a computer device in an embodiment of the application. DETAILED DESCRIPTION

[0033] It should be understood that the specific embodiments described herein are merely intended to explain the application, and are not intended to limit the application.

[0034] The image label generation method based on feature fusion provided by the embodiments of the application can be applied to the fields of medical image analysis, disaster assessment, financial claim settlement and the like. Figure 1In an application environment of the present application, a user terminal communicates with a server terminal through a network. The server terminal can obtain image data through the user terminal, generate a multi-scale local feature map through multi-scale feature extraction, extract category attention information and construct a category matrix, generate a gating signal through a gating unit, adjust the feature map, extract global and local category features, fuse the two to generate a category attention guided feature map, and finally generate a category label map. Through multi-scale feature extraction, dynamic adjustment of category attention information, and fusion of global and local features, the present application improves the accuracy and adaptability of image segmentation, can more effectively identify target regions with large intra-class variation and small inter-class difference, and is suitable for medical image analysis, disaster assessment, and financial claims, etc. The user terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server terminal can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0035] Please refer to Figure 2 , Figure 2 The flowchart of an embodiment of the image label generation method based on feature fusion provided by the present application is shown. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] As shown in Figure 2 , the image label generation method based on feature fusion provided by the present application includes the following steps:

[0037] S10, obtaining image data, performing multi-scale local feature extraction on the image data through a multi-scale feature extraction module to generate a multi-scale local feature map

[0038] In this embodiment, the image data can come from various sources, including but not limited to satellite remote sensing, unmanned aerial vehicle shooting, medical imaging devices (such as CT, MRI), surveillance cameras, etc. In the implementation process, appropriate devices can be selected according to the actual application scenario. For example, in the disaster assessment scenario, an unmanned aerial vehicle can quickly capture high-resolution image data of the disaster area; in the medical and health field, medical imaging devices can obtain image data of the lesion area.

[0039] The obtained image data is usually an RGB three-channel or grayscale image. In the implementation, the image data can be collected using the API interface of the device, and the data can be stored in common formats such as JPEG, PNG or TIFF. For medical imaging, the data can be stored in DICOM format and needs to be decoded by special software tools.

[0040] After acquiring the images, basic preprocessing of the data is required to ensure the quality of subsequent analysis. The preprocessing steps may include image denoising, grayscale conversion, resizing, and format conversion. During implementation, open-source image processing libraries such as OpenCV and Pillow can be used for fast batch processing of images.

[0041] The collected image data needs to be stored on local storage devices or cloud storage platforms. To ensure data security, hierarchical storage and encryption techniques can be used. In the medical and health field, the storage of image data needs to comply with data privacy regulations.

[0042] For example, using a drone to fly over the disaster area and take high-resolution ground images. The drone records and stores image data of the disaster area automatically through the equipped camera module. The flight height, camera focal length, and other parameters of the drone can be adjusted according to the size of the target area. The collected images are transmitted wirelessly to the ground station for further processing.

[0043] In medical imaging, CT or MRI devices acquire image data of internal tissues or organs according to preset scanning parameters such as slice thickness and resolution. The data is stored in DICOM format and analyzed in subsequent image processing software. To improve image quality, denoising and artifact removal algorithms can be used to optimize imaging results.

[0044] Example: In the agricultural insurance scenario, insurance companies can collect image data of farmland disasters using drones. The affected areas of crops can be quickly extracted, and the severity of different crops can be identified using category attention networks, providing more accurate disaster assessment data for insurance companies to assist in developing claims solutions. Through automated image processing and analysis techniques, insurance companies can significantly shorten the claims cycle and improve customer satisfaction.

[0045] In the diagnosis of lung diseases, CT scanners are used to acquire image data of patients' lungs. Abnormal areas in the lung image can be quickly extracted, and lesion types such as lung nodules and inflammation can be identified using category attention networks. Combining image data and other patient diagnosis and treatment information, doctors can more accurately determine the patient's condition and develop targeted treatment plans, thereby improving treatment outcomes and reducing misdiagnosis rates.

[0046] By acquiring high-quality image data, a reliable data foundation is provided for subsequent multi-scale feature extraction and category attention analysis. It can flexibly adapt to various scenarios, both quickly acquiring image data required for disaster assessment and meeting the high-precision imaging needs of the medical and health field, improving the accuracy and timeliness of data analysis.

[0047] The core of the multi-scale feature extraction module is to build a pre-trained base convolutional network model. This network can be based on mainstream deep learning models such as ResNet, VGG, etc., and extract the basic features of image data through convolution operations. The basic convolutional network mainly identifies the edges, textures and color distributions of the image to generate a basic feature map. This feature map contains low-level information of the image, providing input for subsequent multi-scale feature extraction.

[0048] Select a suitable base network architecture, such as ResNet-50, ResNet-101, or VGG-16, etc. Initialize the network using the weights of the pre-trained model. Input the image data into the basic convolutional network to extract the basic feature map.

[0049] The key step of the multi-scale feature extraction module is to perform multi-scale convolution operations on the basic feature map. Multi-scale convolution refers to using different sizes of convolution kernels (such as 1x1, 3x3, 5x5, etc.) to extract features from different regions of the image. Small convolution kernels can capture detailed information, and large convolution kernels can capture global context information.

[0050] Convolution operations are performed on the basic feature map using different sizes of convolution kernels. Different sizes of convolution kernels can be implemented through parallel branches, i.e., each convolution kernel is set up with a separate convolution branch. The feature map output by each convolution operation needs to be processed through batch normalization (Batch Normalization) and activation functions (such as ReLU, LeakyReLU) to improve the expression ability of the features.

[0051] Multi-scale feature extraction can also enhance the recognition ability of long strip-shaped targets by introducing striped convolution operations. Striped convolution uses convolution kernels with different aspect ratios (such as 1x7, 7x1, 1x11, 11x1, etc.) to extract features of linear targets. This is of great significance for recognizing roads, boundary lines, etc. in images.

[0052] Convolution operations are performed on the output feature map of the multi-scale convolution operation using long strip-shaped convolution kernels. The aspect ratio of the strip convolution kernel can be adjusted according to the application scenario. For example, in remote sensing images, longer convolution kernels can be used to extract elongated targets such as rivers and roads.

[0053] After completing the multi-scale convolution and striped convolution operations, the output feature maps of these convolution operations need to be aggregated to generate the final multi-scale local feature map. The aggregation operation can use splicing or addition to fuse feature maps of different scales together to form a feature map containing multiple scale information.

[0054] The output feature maps of different scale convolution operations are spliced to form a multi-channel feature map. The spliced feature map is subjected to channel weighting processing to adjust the weight of each scale feature map, thereby improving the expression of key features. The final multi-scale local feature map can be further compressed by a convolution layer to reduce the dimension of the feature map and improve the calculation efficiency.

[0055] For example, the extraction of local features can be performed by a CNN-based remote sensing (RS) image multi-scale local feature extraction module (CNN). The CNN can be divided into scale and strip feature extraction. The CNN introduces a pre-trained deep learning model ResNet in the scale feature extraction step, receives the ResNet features of the corresponding stage, and then enters a branch composed of 3 convolution layers (1x1, 3x3 and 5x5) to obtain context information of different scales. Batch normalization (BN) and LeakyReLU activation function appear after each convolution layer. After extracting local features at three scales, the local features are aggregated by summing them. The formula can be:

[0056]

[0057] The strip feature extraction step involves 3 groups of strip convolutions (1x7 and 7x1, 1x11 and 11x1, 1x21 and 21x1). The strip convolution has fewer parameters and can provide better strip target recognition results, such as roads and water areas. The above local features are aggregated by a convolution layer (3x3), and residual connection and multi-scale feature fusion are used, ⊕ represents the corresponding position addition of the matrix, to obtain the CNN output

[0058] X 7×7 =BN(Conv 7×1 (Conv 7×1 (X scale ))

[0059] X 11×11 =BN(Conv 7×1 (Conv 7×1 (X scale ))

[0060] X 21×21 =BN(Conv 7×1 (Conv 7×1 (X scale ))

[0061]

[0062] wherein, X Ris the input feature map, usually a preliminary feature map extracted by a basic convolutional network. Its dimension is X B×C×H×W , B represents the batch size, C represents the number of channels, H represents the height of the image, and W represents the width of the image; Conv 1×1 , Conv 3×3 , and Conv 5×5 represent convolution kernels of different sizes, respectively; BN() represents Batch Normalization, which normalizes the convolution output to speed up model convergence and improve stability; represents a concatenation or summation operation by channel, used to integrate multi-scale feature maps; X 7×7 , X 11×11 , and X 21×21 represent different scale feature maps after using the strip convolution operation. These feature maps are generated by using convolution kernels of different sizes on the input feature map. scale represents the generated multi-scale feature map, which contains multi-scale local features obtained by 1x1, 3x3, and 5x5 convolution operations.

[0063] Example: In the financial field, insurance companies can use drones to obtain image data of disaster areas. Through the multi-scale feature extraction module, it can simultaneously identify the damage of different targets such as houses, roads, and farmland. For example, in a flood scenario, multi-scale convolution operations can extract damage features of houses and farmland, and strip convolution operations can extract morphological features of roads or rivers destroyed by floods. The final multi-scale local feature map provides accurate data support for the insurance company to generate a damage assessment report.

[0064] By using the multi-scale feature extraction module to extract multi-scale local features from image data, global and local features in the image can be extracted, and the recognition ability of linear targets and multi-scale targets can be enhanced. It can improve the accuracy and robustness of image segmentation and has significant application effect in asset damage assessment in the financial field and lesion area recognition in the medical field. By extracting multi-scale local feature maps, high-quality input features can be provided for subsequent class attention networks, thereby improving the overall performance of the entire system.

[0065] S20, extracting class attention information from the multi-scale local feature map, and constructing a class matrix based on the class weight and class index in the class attention information;

[0066] In this embodiment, the class attention information is the core information extracted from the multi-scale local feature map, aiming to obtain the attribution probability of each pixel point to different classes. The multi-scale local feature map contains multi-level feature information of the image, and by analyzing these features, the class probability distribution of each pixel point is generated.

[0067] The generation process of class attention information can map the input feature map to probability values for each class based on a convolutional neural network or a multi-layer perceptron (MLP). The class attention information contains both class weights and class indices. The class weights represent the degree of belonging of each pixel point to each class, and the class indices identify the most likely class of each pixel point.

[0068] The multi-scale local feature map is input into the classification network, and the class attention information is generated by calculating the class probability distribution of each pixel point. In the implementation process, the Softmax function can be used to standardize the class probability, ensuring that the sum of the probabilities of each pixel point in all classes is 1.

[0069] The class matrix is an important data structure for storing class features. By extracting class weights and class indices from class attention information, a two-dimensional matrix containing class features can be constructed. The class weight is the belonging probability value of each pixel point, representing the likelihood of each pixel point belonging to each class.

[0070] The class index is the predicted class label of each pixel point, representing the most likely class of each pixel point. By combining the class weight map and the class index map, a class matrix can be constructed for subsequent gating operations or attention mechanism calculations.

[0071] The class weight and class index are extracted from the class attention information to generate the class weight map and the class index map, respectively. By combining the class weight map and the class index map, a class matrix is generated. The class matrix is used in subsequent steps to generate a gating signal, focusing and optimizing the feature map on class features.

[0072] For example, the process of extracting class attention information from the feature map X CNN output by CNN, specifically including:

[0073]

[0074] where B is the batch size, C is the number of channels, H is the height, and W is the width. The class information is then strengthened through batch normalization (BN) and multi-layer perceptron (MLP) layers.

[0075] Then, a 1x1 convolutional layer is used to change the dimension of the feature map to the number of classes N c of the dataset. Finally, the class matrix Y is obtained through the Softmax and Argmax functions.

[0076] XC = Argmax(Softmax(Conv 1×1 (X MLP ))

[0077] The formula represents the application of a 1x1 convolution layer, a Softmax function, and an Argmax operation to extract the class weight and class index of each pixel point from the feature map. The Softmax calculates the class weight, and the Argmax extracts the class index, finally generating the class matrix X C .

[0078] Example explanation: In the construction of customer portraits in the financial field, the behavior data of each customer can be regarded as a multi-scale feature map of the image, and different categories of behavior features can be extracted through category attention information extraction. For example, the behavior patterns of "high-risk", "medium-risk", and "low-risk" customers can be extracted, and through the construction of the category matrix, the attention and management optimization of different customer categories can be realized.

[0079] In medical image analysis, local feature maps of different anatomical structures can be obtained through the multi-scale feature extraction module. The extraction of category attention information can be used to identify the category labels of the lesion area (such as tumors, normal tissues, inflammation, etc.). Through the construction of the category matrix, accurate lesion positioning and classification results can be provided for doctors, and the accuracy of diagnosis can be improved.

[0080] By extracting category attention information from multi-scale local feature maps and constructing a category matrix based on category weights and category indexes, effective representation of category features is achieved. This construction method of the category matrix can more accurately capture the category differences of each pixel point in the image, thereby providing support for subsequent feature adjustment and attention calculation, improving the attention and recognition accuracy of the model to category features.

[0081] S30, a gating unit is used to generate a gating signal, the multi-scale local feature map is adjusted through the gating signal, and the adjusted multi-scale local feature map is processed through the category matrix for channel-by-channel weighting to generate a gated category feature map;

[0082] In this embodiment, the role of the gating unit is to generate a dynamic gating signal, which is used to control the transmission of information flow and the dynamic adjustment of feature weights. The input of the gating unit is the feature map of the current layer and the feature map of the previous layer. For this step, the feature map of the current layer is the multi-scale local feature map, and the feature map of the previous layer is the output of the previous stage of the network. The internal structure of the gating unit includes two convolution operations, which perform convolution calculation on the current layer feature map and the previous layer feature map respectively to obtain preliminary adjustment signals. After adding these two adjustment signals, a nonlinear transformation is performed through an activation function (such as a Sigmoid function) to generate a dynamic gating signal.

[0083] In the actual implementation process, first, the input multi-scale local feature map is processed through the convolution layer to generate a preliminary feature map; second, the feature map of the previous layer is also processed through the convolution operation to obtain another feature map. After element-wise addition of the two feature maps, an activation function is applied for nonlinear transformation, so that the generated gating signal changes between 0 and 1. This nonlinear processing method enables the network to control the transmission intensity of the information flow.

[0084] After generating the gating signal, the multi-scale local feature map of the current layer needs to be adjusted. The adjustment process includes channel weighting and spatial weighting. The gating signal acts as a weight coefficient and directly affects the feature value of each channel, and is adjusted by weighting according to the feature weight information of different pixel positions, so that the network can better focus on the pixel area with feature difference when processing different types of information.

[0085] In the specific implementation process, the channel weighting processing is to multiply each channel weight of the gating signal by the corresponding feature map channel value, and the spatial weighting processing is to perform pixel-by-pixel multiplication operation on the weight information of each pixel position. The adjusted multi-scale local feature map can more accurately represent the class information in the subsequent network calculation, improving the classification performance.

[0086] The adjusted multi-scale local feature map needs to be further combined with the class weight information in the class matrix for channel-by-channel weighting processing. The class matrix is constructed from class attention information, which contains the class weight corresponding to each pixel point. The channel-by-channel weighting process is to weight the value of each channel of the adjusted feature map according to the weight value of different classes in the class matrix, so as to enhance the discrimination ability of class features.

[0087] In the implementation process, first, the class weight information of each channel is extracted from the class matrix, and then these weight values are multiplied element by element with the corresponding channel values of the adjusted multi-scale local feature map to generate a new feature map. This channel-by-channel weighting operation enables the network to strengthen or suppress different channel features according to the class information, so as to more accurately represent the class features.

[0088] After the adjustment of the dynamic gating signal and the channel-by-channel weighting processing of the class matrix, the gated class feature map is finally generated. This feature map contains more accurate class feature information, which can provide input for the subsequent global class attention guiding module and local class attention guiding module.

[0089] The generation process of the gated category feature map includes two main steps: the first step is to adjust the feature map using a dynamic gating signal, so that the network can dynamically adjust the transmission of feature information; the second step is to weight the feature map channel by channel using the category weight information of the category matrix, so that the network can better focus on important category features.

[0090] For example, in order to make more accurate category feature adjustment on multi-scale local feature maps, a gating unit (Gate Unit) needs to be introduced, which plays a core role in generating dynamic gating signals through the information of the category matrix to adapt to the weighting adjustment of different category features.

[0091] Specifically, the core calculation process of the gating unit is:

[0092] G = σ(Conv(X R )+Conv(X prev ))

[0093] Where: G represents the generated gating signal matrix; σ is the Sigmoid function, and Conv represents the convolution operation. Assuming X prev is the output feature of the previous layer, its dimension is the same as X R .

[0094] Adjust the category matrix CM using the gating signal G:

[0095]

[0096] The gating signal G is a dynamic weight signal generated from the feature map by the gating unit. The category matrix CM is constructed after extracting category attention information from the multi-scale local feature map. When the gating signal G is applied to the category matrix CM, the category weight can be dynamically adjusted, thereby enhancing the category difference in the feature map.

[0097] Where CM' represents the adjusted category matrix, G is the gating signal, and represents the element-wise multiplication operation. Each element in the gating signal G is multiplied by the corresponding element in the category matrix CM pixel by pixel, thereby adjusting the weight value in the category matrix according to the dynamic change of the category weight.

[0098] X′ R =CM′·X R

[0099] Where X' R represents the adjusted feature map, i.e., the gated category feature map, CM' is the adjusted category matrix, and X R is the original multi-scale local feature map. This step is completed by matrix multiplication, which weights the category features in the multi-scale local feature map channel by channel according to the adjusted category weight, generating a more accurate category feature map.

[0100] By introducing the gating unit and the channel-wise weighting processing of the category matrix, dynamic adjustment of the feature map and fine representation of the category features are realized. The category feature information can be more accurately extracted and strengthened, especially when dealing with image data with large intra-class differences and small inter-class differences, thereby improving the accuracy and robustness of image classification, and being suitable for medical health, finance and other fields.

[0101] S40, extracting global category features from the gated category feature map through a global category attention guiding module;

[0102] In this embodiment, in the global category attention guiding module, the gated category feature map is first divided into windows. The window division operation aims to decompose the entire feature map into multiple small window feature blocks, each of which contains information of a local region in the feature map. The size of the window division can be set according to the specific application requirements, such as using a fixed size k x k window or dynamically adjusting the window size.

[0103] In the implementation process, the sliding window method is used to slide the feature map by rows and columns, extracting each local region in the feature map to form a non-overlapping window feature block. This operation helps to reduce the computational complexity of global feature extraction while preserving the local structure information.

[0104] In order to better adapt to the subsequent global category attention mechanism, each window feature block needs to be unfolded into a one-dimensional vector form through linear transformation operation. This process can be realized through matrix flattening (Flatten) and fully connected layer (Fully Connected Layer).

[0105] The flattening operation converts the two-dimensional matrix form of each window feature block into a one-dimensional vector, thereby constituting the input sequence of the attention mechanism. Subsequently, the dimension of the feature is adjusted through linear transformation to match the dimensions of the query vector, key vector and value vector, facilitating subsequent attention calculation.

[0106] When constructing the global category attention, the input sequence unfolded by linear expansion needs to be mapped into query vector (Query), key vector (Key) and value vector (Value) respectively. The query vector is used to determine the attention direction of the current window feature block, the key vector is used to represent the association between the feature blocks, and the value vector contains the feature representation information of each feature block.

[0107] The mapping process uses convolution operation or linear layer to complete the transformation of each input sequence into query vector, key vector and value vector. This operation ensures that the model can calculate the correlation between window feature blocks through different feature representations.

[0108] The attention score function is constructed using the scaled dot-product attention mechanism of the query vector Q, the key vector K, and the value vector V. By calculating the attention score function, a weight matrix of class attention can be generated for the extraction of global class features. First, the attention score function is calculated as follows:

[0109]

[0110] where ⊕ denotes the element-wise addition operation, and CA represents the class weight map.

[0111] Finally, the calculated attention output vector is concatenated to restore the same dimension as the original feature map. After concatenation, a linear transformation is performed on the concatenated feature vector to generate the final global class feature.

[0112] The linear transformation process can be implemented through convolution operation or fully connected layer to ensure that the output global class feature maintains consistent shape and dimension with the input feature map. The generated global class feature contains class information and global context information in different regions of the image, which provides more rich global feature representation for subsequent feature fusion.

[0113] For example, the attention weight matrix is applied to the value vector V to extract the global class feature:

[0114]

[0115] where, denotes the element-wise multiplication operation, and X GCA denotes the global class feature.

[0116] The global class feature is extracted from the gated class feature map through the global class attention guidance module, realizing the global modeling ability of the class information in the image. It effectively improves the attention and resolution of the model to the global class information. By integrating the class weight information into the attention score function, this module can dynamically adjust the weights of each window feature block, making the model more robust and accurate in identifying different classes.

[0117] S50, through the local class attention guidance module, the local class feature is extracted from the gated class feature map;

[0118] In this embodiment, the gated class feature map is generated by the gating unit of the previous step, containing the fusion information of global and class features. In order to better extract local class features of different scales, multi-scale convolution operation is needed, that is, different size convolution kernels such as 3x3, 5x5, 7x7, etc. are applied to the same feature map. This multi-scale convolution operation can capture feature information of different levels and scales in the image, thereby improving the perception ability of local class features in the image.

[0119] A multi-scale convolution module is constructed, and different size convolution kernels are used to perform convolution operations on the gated class feature map respectively; the output results of each convolution operation are subjected to batch normalization and activation processing to ensure that the feature values are within a reasonable range and avoid gradient explosion or disappearance; finally, the results of multi-scale convolution operations are spliced or superimposed to generate a preliminary local class feature map.

[0120] In remote sensing images, linear targets such as roads and rivers are common class features, but these linear targets may not be effectively extracted by traditional square convolution kernels. Therefore, a strip convolution operation is introduced to extract linear target features by using convolution kernels of different lengths and widths. For example, 1x7, 1x11, and 1x21 convolution kernels are used to extract horizontal linear features, and 7x1, 11x1, and 21x1 convolution kernels are used to extract vertical linear features.

[0121] The strip convolution operation is applied to the local class feature map to extract horizontal and vertical linear target features respectively; the convolution results of the horizontal and vertical directions are superimposed to generate a feature map containing linear target information.

[0122] In order to further enhance the local class features, channel weighting processing is needed for the results of the convolution operation. The core idea of channel weighting processing is to dynamically adjust the weights of each channel in the feature map according to the distribution of each class in the image, emphasizing the features of important classes and suppressing irrelevant information.

[0123] The weight value of each channel is calculated and weighted according to the importance of the class feature; each channel of the feature map is multiplied by the corresponding weight value to obtain the weighted feature map.

[0124] Depthwise separable convolution is an efficient convolution operation that can significantly reduce the number of convolution operation parameters without losing feature extraction capability. It decomposes the standard convolution into depthwise convolution and pointwise convolution, which extracts features in the spatial dimension and channel dimension respectively.

[0125] The depthwise separable convolution operation is applied to the local class feature map to extract more fine-grained local class features; the feature information of different channels is fused together through pointwise convolution to generate the final local class feature map.

[0126] In order to ensure that the value range of the feature map is within a reasonable range, and to enhance the nonlinear expression ability of the network, activation and normalization processing are needed for the local class feature. The activation function can be ReLU or Leaky ReLU, and the normalization operation can be Batch Normalization (Batch Normalization).

[0127] The activation function is applied to the local class feature map to enhance the nonlinear expression ability of the network. The normalized feature map after activation avoids gradient explosion or gradient disappearance. The final output of the local class feature map contains rich local class information, which provides an important input for subsequent feature fusion.

[0128] For example, the local class attention guide module (LCAG) is a core module for extracting local class features. It mainly realizes the precise extraction of local class information by performing dot product attention calculation and convolution operation on the gated class feature map. The whole process can be divided into the following steps:

[0129]

[0130] The formula represents the local attention weight matrix Atten LCA , which is calculated by the local class attention mechanism, where CA is the class feature in the class matrix, and V is the value vector representing the actual value of the class feature. represents the dot product operation of the tensor; softmax is a normalization function used to normalize the weight value to the range of [0, 1].

[0131] Calculate the local class feature:

[0132]

[0133] The formula represents the local class feature calculation process based on the local attention weight matrix Atten LCA and the gated class feature map X CNN , where X CNN is the input gated class feature map. represents the element-level product operation. represents the residual connection (i.e., directly adding the input feature map to the output feature map).

[0134] Convolution and normalization processing are performed on the local class feature:

[0135] X LCA =BN(Conv 3×3 (X′ LCA ))

[0136] The formula represents the local class feature map X' LCA A 3x3 convolution operation is performed, and the convolution result is normalized by batch normalization (BN), where: Conv 3×3 represents a convolution operation using a 3x3 convolution kernel; and BN represents a batch normalization operation.

[0137] Example: In medical image analysis, the local class attention guide module can be used to accurately extract local features of the lesion area. For example, in lung CT images, the local class attention mechanism can effectively distinguish the boundaries between normal tissue and diseased tissue, helping doctors more accurately identify the location and type of lesions, thereby improving the accuracy and efficiency of diagnosis.

[0138] In financial risk assessment, the local class attention guide module can be used to accurately identify key features in image data. For example, in unmanned aerial vehicle image analysis, the local class attention mechanism can accurately identify crop types and growth conditions in farmland, providing more accurate risk assessment basis for agricultural insurance companies. At the same time, the module can also help banks identify specific information of assets such as real estate and land when auditing mortgage assets, improving the accuracy and efficiency of the audit.

[0139] By processing the gated class feature map through the local class attention guide module, local class features in the image can be effectively extracted, especially for class information with significant boundaries or linear targets. The network can dynamically adjust the weight of the features during local feature extraction, thereby improving the perception ability of local class information and classification accuracy.

[0140] S60, fusing the global class feature and the local class feature to generate a class attention guide feature map;

[0141] In this embodiment, after generating the global class feature and the local class feature, the two need to be fused to generate a more comprehensive class attention guide feature map.

[0142] First, the feature fusion is completed by element-wise weighted summation of the global class feature and the local class feature matrix. This process ensures that the class feature map contains both global context information and local detail information, thereby improving class representation ability and classification accuracy.

[0143] The mathematical formula of the fusion operation is as follows:

[0144]

[0145] where: X CAGM represents the fused class attention guide feature map; and X GCAX LCA represents the global class feature map generated by the global class attention guiding module. represents the local class feature map generated by the local class attention guiding module.

[0146] After the fusion operation is completed, the generated class attention guiding feature map has the following characteristics:

[0147] It can capture global information and local detail information in the image; it improves the recognition accuracy of the model for class features, especially in complex background or uneven lighting conditions.

[0148] Through the fusion of global and local class features, a class attention guiding feature map is generated, thereby improving the feature representation ability and classification performance of the model. In practical applications, this scheme can effectively improve the recognition accuracy in complex image scenes and has wide applicability.

[0149] S70, fuse the class attention guiding feature map with the multi-scale local feature map, and perform weighted processing on the fused feature map through the feature fusion head module to generate the final fused feature map.

[0150] In this embodiment, after the class attention guiding feature map and the multi-scale local feature map are generated, the two feature maps need to be fused, and the fused feature map is weighted processed through the feature fusion head module to generate the final fused feature map.

[0151] The class attention guiding feature map (denoted as X CAGM ) contains the feature representation of global information and local information of different classes in the image, while the multi-scale local feature map (denoted as X CNN ) contains the local detail features of different scales in the image.

[0152] In order to better utilize the advantages of the two feature maps, the two feature maps are fused. The fusion operation is completed by element-wise addition, ensuring that the feature value of each pixel point contains class information and local detail information. The mathematical representation of fusion is as follows:

[0153] X fusion = X CAGM ⊕ X CNN

[0154] Where: X fusion represents the fused feature map; X CAGM represents the class attention guiding feature map; X CNN represents the multi-scale local feature map; and ⊕ represents the element-wise addition operation.

[0155] This operation ensures that the fused feature map has both global class information and local detail information, providing high-quality input data for subsequent weighted processing.

[0156] Fused Feature Map X fusion After generation, it needs to be weighted by the feature fusion head module. The feature fusion head module is a convolutional network structure that can assign different weights to different channels and spatial positions of the fused feature map, achieving more detailed feature representation.

[0157] The mathematical representation of weighted processing is:

[0158] X FFH = Conv 1×1 (BN(Conv 3×3 (X fusion )))

[0159] Where: X FFH represents the final fused feature map after weighted processing; Conv 3×3 represents a 3x3 convolution operation to extract local context information; BN represents batch normalization to stabilize the training process; Conv 1×1 represents a 1x1 convolution operation for channel weighting.

[0160] Through the weighted processing of the feature fusion head module, the model can dynamically adjust each channel of the feature map according to the importance of different feature channels, generating more discriminative feature maps.

[0161] After processing by the feature fusion head module, the final fused feature map X FFH contains a global, local, and class information fusion representation. This feature map can more comprehensively describe the target class features in the image, providing high-quality input data for the subsequent class label map generation step.

[0162] After generating the final fused feature map X FFH , to optimize the performance of the model, a loss function needs to be introduced for supervision and optimization. The design of the loss function covers primary loss, scale auxiliary loss, class auxiliary loss, and disaster-related loss. These loss terms can jointly act on the network training process, thereby improving the model's expression ability and prediction accuracy.

[0163] Primary Loss

[0164] The primary loss optimizes the generated class label map of the final fused feature map X FFH by combining cross-entropy loss and Dice loss :

[0165]

[0166] wherein:

[0167] Dice loss:

[0168]

[0169] Cross-entropy loss:

[0170]

[0171] N denotes the number of samples, C denotes the number of classes, denotes the predicted probability that sample n belongs to class c, denotes the true label of sample n.

[0172] Scale auxiliary loss

[0173] The scale auxiliary loss is obtained by fusing multi-scale features X scale , which constrains the expression ability of multi-scale features:

[0174]

[0175] X scale denotes the feature map fused with multi-scale features; denotes the k-th feature map of the class attention guide module; UP 2× , UP 4× denote the up-sampling operations by two and four times, respectively, for adjusting the feature map to the same resolution.

[0176] Class auxiliary loss

[0177] In the calculation of the class auxiliary loss, a multi-stage fusion strategy is adopted to improve the supervision effect on the class features. First, the class features are extracted from the feature maps of each stage output by the class attention module. To ensure that these feature maps have consistent resolutions, up-sampling operations UP 2× , UP 4× are used to adjust the feature maps of different stages to a uniform resolution.

[0178] Subsequently, the adjusted class feature maps are fused in an element-wise addition manner to generate the final class auxiliary feature X category . This fused feature contains multi-level class information and can more comprehensively describe the class distribution in the image, providing high-quality input for the calculation of the class auxiliary loss. The formula is as follows:

[0179]

[0180] Disaster-related loss

[0181]

[0182] represents a disaster-related loss, used to measure the difference between predicted loss and actual loss; represents the predicted value of the nth sample for class c; represents the true value of the nth sample for class c.

[0183] Overall loss function (L):

[0184]

[0185] represents the overall loss function, used to comprehensively optimize the model; a, b, g represent the weight parameters of the loss term, which can be adjusted according to the actual scene.

[0186] Example description: Disaster scene images taken by drones usually contain various categories of features such as buildings, roads, and vegetation. Through the feature fusion head module, the local features of different scales (such as cracks and collapse details of damaged buildings) and global features (such as the extent and range of the overall disaster area) in the image data collected by the drone can be effectively combined. For example, in a flood disaster, the images taken by the drone may contain collapsed houses, flooded roads, and other disaster areas. These local details can be extracted to identify the range and extent of various damaged areas, and they can be analyzed in conjunction with the overall disaster range to generate a high-accuracy disaster map. These data not only help insurance companies quickly assess loss amounts and shorten claims processing times, but also provide scientific rescue decision-making basis for government and rescue departments.

[0187] By fusing the class attention guided feature map with the multi-scale local feature map and weighting the fused feature map through the feature fusion head module, the final fused feature map is generated.

[0188] S80, according to the final fused feature map, a class label map is generated.

[0189] In this embodiment, the final fused feature map integrates global, local, and class feature information, and has high-dimensional context semantic expression capability. In order to realize the generation of the class label map, the final fused feature map needs to be further decoded. The main purpose of the decoding operation is to restore the high-dimensional features to the same resolution as the original input image, and to predict a class label for each pixel.

[0190] The final fused feature map is upsampled using an upsampling operation such as deconvolution or interpolation to expand the spatial resolution. The upsampled feature map will align with the spatial dimensions of the input image while preserving the semantic information of the final fused features.

[0191] The upsampled feature map is fed into a classification layer such as a 1x1 convolutional layer to calculate the probability distribution of each pixel for each class. The core of this step is to generate a class probability distribution map through pixel-wise multi-class classification. The output of the classification layer is normalized using the Softmax function to obtain the class probability distribution map. The Softmax function ensures that the sum of all class probabilities for each pixel is 1, thus achieving pixel-level class prediction.

[0192] In the class probability distribution map, the class of each pixel is determined by the maximum class of its probability value. The core of this step is to map continuous probability distribution to discrete class label map. The Argmax operation is used to extract the class index with the maximum probability from the class probability distribution to generate the final class label map. Each pixel of the class label map represents the corresponding class at that location.

[0193] Example: In the medical health field, it can be applied to medical image analysis such as lesion segmentation and tissue recognition tasks. Taking lung CT images as an example, different lung structures (such as airways, alveoli, blood vessels, etc.) and lesion areas (such as lung nodules, tumors, inflammation, etc.) can be segmented at the pixel level to accurately identify different types of tissues and lesion areas.

[0194] In the specific application process, doctors do not need to manually label image data, but through the multi-scale feature extraction module, different scale features in medical images are automatically extracted, and through the global class attention guide module and the local class attention guide module, the class features of specific lesion areas are focused. For example, when identifying lung nodules, the global feature guide module can focus on the overall structure of the lung, while the local feature guide module can finely identify the boundaries, size and location of the nodules, thereby improving the segmentation accuracy and recognition effect.

[0195] In actual deployment, unmanned aerial vehicles or remote medical devices can be used to collect CT images of patients. After the image data is input, the system dynamically adjusts the multi-scale local feature map through the gating unit to generate a fused feature map containing global and local class information. Finally, the class label map generated based on the fused feature map can accurately identify different lung structures and lesion areas. For example, red areas represent lung nodules, yellow areas represent normal lung tissue, and blue areas represent blood vessel pathways. The following beneficial effects can be achieved:

[0196] Improving the accuracy of lesion recognition: By fusing global and local class information, the boundaries and internal structures of lung nodules can be more accurately identified.

[0197] Reducing misdiagnosis and missed diagnosis rates: The class attention mechanism effectively distinguishes different tissues and lesion areas, avoiding recognition errors due to similar classes.

[0198] Automated medical image analysis: From image data input to class label map output, the whole process of lesion area recognition is automated, providing an efficient and accurate auxiliary diagnostic tool for doctors.

[0199] For example, for a patient at risk of lung nodules, doctors can quickly identify suspicious areas in the lungs through automatically generated class label maps and further analyze the nature, size, and trend of the nodules to develop more accurate treatment plans.

[0200] Similarly, in the financial sector, it can be applied to disaster assessment and insurance claims scenarios, especially for geographic remote sensing image analysis tasks in agricultural insurance and property insurance. For example, after a disaster occurs (such as floods, fires, earthquakes, etc.), insurance companies usually need to quickly assess the damage in the affected area to provide data support for insurance claims. Through semantic segmentation analysis of remote sensing images, accurate identification of different classes such as buildings, roads, farmland, and forests in the disaster area is achieved, and a class label map of the damaged area is automatically generated, providing a basis for disaster loss assessment.

[0201] In specific applications, geographic remote sensing images of the affected area are obtained through drones or satellites, and the image data is input into the model. First, the multi-scale feature extraction module extracts different scale features from the remote sensing image, such as identifying the boundaries of buildings, the layout of roads, and the distribution of farmland. Then, through the gating unit and class attention guide module, the feature map is dynamically adjusted and optimized to generate a gated class feature map. This feature map can effectively distinguish different classes in the affected area, such as damaged buildings, destroyed roads, flooded areas, and unaffected farmland.

[0202] During the generation of the class label map, the system automatically performs global and local weighting fusion of the class features to ensure that the segmentation results have higher precision and robustness. The final output class label map can accurately identify the class and damage level of each region. For example, red areas represent completely destroyed buildings, yellow areas represent partially damaged roads, green areas represent unaffected farmland, and blue areas represent flooded areas. The following benefits can be achieved:

[0203] Improving the accuracy of disaster loss assessment: The automatically generated class label map covers multiple class information of the disaster area, ensuring that the identification of damaged areas is more comprehensive and accurate.

[0204] Shorten the claim settlement time: the automated disaster assessment process greatly shortens the manual verification time, and the insurance company can complete the disaster claim more quickly.

[0205] Reduce the assessment cost: without manual verification of disaster situations one by one, the personnel and time cost of disaster assessment is reduced.

[0206] For example, in a flood disaster, the insurance company obtains remote sensing image data of the disaster area through a drone, and the system automatically generates a category label map that identifies the area of buildings and farmland covered by the flood. The insurance company can quickly assess the area and severity of the damaged area according to the label map, and then provide fast claim settlement services for the disaster-stricken customers.

[0207] By upsampling, classifying and probability mapping the final fused feature map, not only a category label map consistent with the resolution of the input image can be generated, but also each category area can be accurately divided in pixel units.

[0208] The application relates to the fields of artificial intelligence technology and medical health, and discloses an image label generation method based on feature fusion, which comprises the following steps: acquiring image data, generating multi-scale local feature maps through multi-scale feature extraction, extracting category attention information and constructing a category matrix; generating a gating signal through a gating unit, adjusting the feature maps, extracting global and local category features, and fusing the two to generate a category attention guide feature map; and finally generating a category label map through weighted processing of the fused feature map. Through multi-scale feature extraction, dynamic adjustment of category attention information and fusion of global and local features, the accuracy and adaptability of image segmentation are improved, target areas with large intra-class variation and small inter-class difference can be more effectively recognized, and the method is suitable for medical image analysis, disaster assessment and financial claim settlement and the like.

[0209] In one embodiment, after S20, the method further comprises:

[0210] S301, performing window division on the category label of each pixel in the category matrix to obtain a plurality of window feature matrices;

[0211] S302, extracting category labels from each window feature matrix, and combining the category labels into a category label sequence according to a preset order;

[0212] S303, performing subtraction operation on the category label sequence corresponding to each window feature matrix to obtain a category difference matrix;

[0213] S304, adding a position index to the category difference matrix to obtain a category difference matrix containing position information;

[0214] S305, plane processing is performed on the category difference matrix containing position information, and negative values are offset to construct a relative category index;

[0215] S306, adjusting the category matrix based on the relative category index to optimize the feature representation of the category matrix.

[0216] In this embodiment, after extracting the category attention information from the multi-scale local feature map and constructing the category matrix based on the category weight and category index in the category attention information, in order to further optimize the feature representation of the category matrix, it is necessary to divide the category labels of each pixel in the category matrix into windows to obtain more refined category feature information. The window division operation aims to divide the category matrix into multiple small feature regions, and each region is a window feature matrix. The division process can be set according to the preset window size and sliding step, so as to ensure the local analysis of the category information of each pixel in the category matrix. This window division process can effectively capture the spatial locality of the category features, making the representation of the category features more accurate and complete.

[0217] After completing the window division, it is necessary to extract the category labels from each window feature matrix and combine these category labels into a category label sequence according to a preset order. The core of this process is to arrange the category labels in each window region into a linear sequence to facilitate subsequent serialization processing and feature analysis. During the generation of the category label sequence, the order and position information of each category label need to be accurately recorded to ensure correct mapping of the category features in subsequent processing.

[0218] For each category label sequence generated from a window feature matrix, a subtraction operation is performed to obtain a category difference matrix. The generation process of the category difference matrix aims to measure the degree of change of the category features within each window by calculating the difference between the category labels. In the subtraction operation process, the value of each category label is usually compared with the mean or median value in the window to obtain the difference value between the categories. The result of the category difference matrix can reflect the change trend of the category features in each window region, which is helpful to identify the boundary information and category conversion area in the image.

[0219] In order to further improve the expression ability of the category difference matrix, it is necessary to add position index to the category difference matrix to generate a category difference matrix containing position information. The addition of position index makes each category difference value not only contain the difference information between categories, but also include the specific position of the difference value, so that the spatial distribution information of the category features is preserved. The position index is usually represented in the form of two-dimensional coordinates, which can be generated by marking the pixel positions of the category matrix.

[0220] The category difference matrix containing position information is flattened and the negative values are offset to construct a relative category index. The purpose of this step is to flatten the multi-dimensional data of the category difference matrix into a one-dimensional data sequence to facilitate vectorization processing in subsequent calculations. In order to avoid the influence of negative values on feature representation during the flattening process, negative values need to be offset to convert negative values to positive values or zero values, thereby ensuring the effectiveness and accuracy of the category difference data. The implementation of flattening and negative value offsetting can be adjusted according to the needs of different application scenarios, such as normalization, standardization or threshold operation to optimize the representation effect of category features.

[0221] After completing the construction of the relative category index, the category matrix is adjusted based on the relative category index to optimize the feature representation of the category matrix. The core of this adjustment process is to modify the weight of each pixel in the category matrix through the relative category index, thereby improving the expression ability of the category features. The adjustment of the category matrix can be realized by matrix multiplication, weighted average and other operations, and the specific adjustment strategy needs to be dynamically optimized according to the distribution of the category features and the weight values of the relative category index.

[0222] This embodiment realizes the fine expression and differentiated recognition ability of category features through the optimized adjustment of the category matrix. Through the steps of window division, category label sequence generation, category difference matrix calculation, position index addition and flattening, the feature representation effect of the category matrix can be effectively improved, and the spatial distribution information and category difference information of the category features can be enhanced. The optimized category matrix can more accurately reflect the category features in the image, providing high-quality input data for subsequent feature fusion and category label map generation.

[0223] In one embodiment, the above S40 comprises:

[0224] S401, the gated category feature map is window segmented to generate a plurality of window feature blocks;

[0225] S402, each window feature block is linearly unfolded to construct an input sequence of multi-head attention;

[0226] S403, the input sequence is mapped to query vector, key vector and value vector to form an initial vector of multi-head attention;

[0227] S404, the initial vector is subjected to scaling dot product attention calculation, and the category attention information is incorporated into the attention score function as an additive term;

[0228] S405, the output vector of the attention score function is spliced and linearly transformed to generate the global category feature.

[0229] In this embodiment, the process of extracting global class features from the gated class feature map through the global class attention guidance module aims to enhance the representation ability of class features by utilizing global context information. This process is mainly achieved through steps such as window segmentation, multi-head attention mechanism, scaled dot-product attention calculation, and concatenation and linear transformation, to ensure that more comprehensive class features are extracted from the overall level.

[0230] First, the gated class feature map is window segmented, dividing the feature map into multiple window feature blocks. The purpose of window segmentation is to split large-sized feature maps into smaller regional blocks without losing global information, to facilitate subsequent attention calculation. Each window feature block contains feature information of a local region in the image while maintaining the overall spatial structure relationship. During window segmentation, the size and overlap ratio of the window can be adjusted according to specific task requirements, balancing the effect of calculation overhead and information preservation.

[0231] After completing window segmentation, each window feature block needs to be linearly expanded to construct the input sequence of multi-head attention. The linear expansion process converts the two-dimensional feature map into a one-dimensional feature sequence, so that the pixel information in each feature block can be processed one by one. This conversion operation not only preserves the spatial structure information of each feature block, but also provides input data format conversion for subsequent query, key, and value mapping.

[0232] Mapping the input sequence to query vectors, key vectors, and value vectors aims to provide basic feature representation for multi-head attention mechanism calculation. Query vectors represent the target information that needs to be obtained from the feature map, key vectors represent the feature information of each position in the feature map, and value vectors represent the output information of the corresponding position in the feature map. By mapping the input sequence, the original feature sequence can be converted into the three core vectors of the multi-head attention mechanism, thereby realizing the correlation modeling between features.

[0233] After obtaining the query vectors, key vectors, and value vectors, scaled dot-product attention calculation is performed on the initial vectors, and the class attention information is incorporated into the attention score function as an additive term. The core of this step is to generate an attention weight matrix by calculating the dot product of the query vectors and the key vectors, thereby measuring the correlation between input features. To avoid numerical instability caused by large feature values, scaling processing is needed for the dot product result. In the calculation of the attention score function, the class attention information is introduced as an additive term, thereby further enhancing the recognition ability of class features and enabling the model to pay more attention to the difference information between classes.

[0234] Finally, the output vector of the attention score function is spliced and linearly transformed to generate the global class feature. The splicing operation combines the output features of different attention heads to form a comprehensive representation containing multiple feature perspectives. The linear transformation adjusts the spliced feature vector to the same dimension as the original feature map through matrix multiplication of the weight matrix, ensuring that the generated global class feature maintains the same shape as the input feature map. The global class feature after linear transformation contains class information from different window regions and integrates global context information, providing high-quality input data for subsequent feature fusion and class label map generation.

[0235] The process of extracting global class features by the global class attention guide module in this embodiment effectively solves the problem of insufficient global context information in the prior art. Through the fusion of multi-head attention mechanism and class attention information, the model can focus on the class association between different regions in the image, thereby improving the representation ability of the class features.

[0236] In one embodiment, the above S50 comprises:

[0237] S501, respectively, the gated class feature map is subjected to branch convolution operation with different size convolution kernels to extract local features of different scales;

[0238] S502, the output results of each branch convolution operation are spliced or added to obtain multi-scale local responses;

[0239] S503, the multi-scale local responses are subjected to depth separable convolution operation, and the output feature map of the depth separable convolution operation is activated and normalized to generate the local class feature.

[0240] In this embodiment, the process of extracting local class features from the gated class feature map by the local class attention guide module is to effectively capture the detailed features of local regions in the image. Especially when dealing with targets with scale changes, local feature extraction is crucial. This process uses different size convolution kernels for branch convolution operation, multi-scale feature fusion, and depth separable convolution operation to achieve efficient local feature extraction, thereby generating local class features with stronger discriminative ability.

[0241] Firstly, different size convolution kernels are used for branch convolution operation on the gated category feature map to extract local features of different scales. In actual implementation, common convolution kernel size combinations can be selected, such as 1x1, 3x3, 5x5, etc. Each convolution kernel can focus on the feature extraction ability of different receptive fields. The 1x1 convolution kernel is used to extract pixel-level feature information, the 3x3 convolution kernel focuses on a smaller local area, and the 5x5 convolution kernel can capture more context information in a larger range. The design of this multi-branch convolution operation can effectively solve the representation problem of target objects in images at different scales, ensuring that the feature information of small and large targets can be extracted.

[0242] After completing the branch convolution operation, the output results of each branch convolution operation are spliced or added to obtain multi-scale local responses. The splicing operation can preserve the feature integrity of each branch convolution and directly integrate feature maps of different scales into a larger feature map. The addition operation fuses the features of different branch convolutions to reduce the redundant information of the feature map. These two operation methods can be selected according to the actual application scenario and the limitation of computing resources. For example, in a scenario with less computing resources, the addition operation can be used to reduce the computational complexity; while in a scenario that pays more attention to accuracy, the splicing operation can be used to preserve more feature details.

[0243] Next, a depth separable convolution operation is performed on the multi-scale local response, and an activation and normalization operation is performed on the output feature map of the depth separable convolution operation to generate the local category feature. The depth separable convolution operation decomposes the standard convolution into two steps of depth convolution and point convolution, thereby greatly reducing the complexity of convolution calculation. In the depth convolution process, each convolution kernel only acts on one input channel, so it can more efficiently extract local feature information; and the point convolution integrates the outputs of each channel to generate an output feature map with comprehensive information. In order to ensure that the feature map after the depth separable convolution operation can maintain numerical stability and fast convergence during the training process, the output feature map needs to be activated and normalized. Common activation functions include ReLU, Leaky ReLU, etc., which can effectively introduce nonlinear changes and improve the expression ability of the model; batch normalization (Batch Normalization) can make the distribution of the feature map more stable, thereby speeding up the convergence of the model.

[0244] After processing by the local category attention guide module, the generated local category feature map can more accurately capture the category feature information of the local area in the image. Whether it is a small target object or local area boundary detail information, it can be effectively extracted to provide high-quality input data for subsequent feature fusion and class label map generation.

[0245] The embodiment effectively solves the problem of insufficient local detail feature extraction in the prior art by extracting local class features from the gated class feature map through a local class attention guide module. The use of convolution kernels of different scales and depth separable convolution technology ensures that the model can accurately capture the class features of the local region when processing target objects with large scale changes. In addition, the splicing or addition operation of multi-scale local responses realizes the fusion between features of different scales, so that the model can capture local detail information and retain overall context information. Activation and normalization processing ensure the numerical stability of the feature map during the training process, thereby improving the convergence speed and classification accuracy of the model.

[0246] In one embodiment, in the S10, the multi-scale local feature extraction of the image data is performed by a multi-scale feature extraction module to generate a multi-scale local feature map, including:

[0247] S101, a pre-trained base convolutional network is constructed, and the image data is preliminarily feature-extracted by the base convolutional network to obtain a base feature map;

[0248] S102, multi-scale convolution operation is performed on the base feature map using convolution kernels of different sizes;

[0249] S103, strip convolution operation is performed on the output result of the multi-scale convolution operation to extract feature information in the form of long strips or boundaries;

[0250] S104, the output result of the strip convolution operation is aggregated to generate the multi-scale local feature map.

[0251] In the embodiment, the base convolutional network usually uses a pre-trained model such as ResNet, VGG or EfficientNet, which has been pre-trained on a large-scale dataset and has strong feature extraction capability. The input of the base convolutional network is image data, and a feature map containing preliminary features is generated through a series of convolution, activation and pooling operations. This step is mainly responsible for extracting the bottom features of the image, including edge, texture, color and other information, which provides a basis for subsequent multi-scale feature extraction.

[0252] The convolution layer in the base convolutional network extracts features from the input image through a sliding window. The size and step of the convolution kernel determine the receptive field range of the extracted features, and the activation function (such as ReLU) introduces non-linear changes, so that the feature extraction result can better describe the complex structure in the image.

[0253] On the basis of the base feature map, multi-scale convolution operations are performed using different sizes of convolution kernels to extract feature information of different scales in the image. The core of this step is to capture multi-level information in the image using convolution kernels with multiple receptive field ranges. For example, a 1x1 convolution kernel is used for pixel-level feature extraction, a 3x3 convolution kernel is used to capture local region features, and a 5x5 or larger convolution kernel is used to obtain larger range context information. Through multi-scale convolution operations, the scale variation problem in the image can be effectively addressed, so that small targets and large targets can be accurately recognized by the model.

[0254] The output of the multi-scale convolution operation contains multiple feature maps of different scales, each of which corresponds to the feature information extracted by a different size of convolution kernel. These feature maps together form a multi-scale feature space, providing input data for subsequent strip convolution operations.

[0255] Strip convolution is a special convolution operation designed to capture long strip or boundary features in the image. Unlike traditional square convolution kernels, strip convolution kernels are more elongated in shape, allowing them to focus more effectively on line, boundary, and texture features in the image. This operation is particularly important for identifying long strip-shaped objects such as roads, rivers, and borders in remote sensing images.

[0256] Strip convolution can use convolution kernels of different directions, such as horizontal strip convolution, vertical strip convolution, or diagonal strip convolution, to capture long strip features in different directions. Each type of strip convolution kernel can focus on boundary features in a specific direction, allowing the model to have stronger boundary recognition capabilities.

[0257] Aggregation processing is used to integrate the output of the strip convolution operation into a unified feature map. Common aggregation methods include feature concatenation, feature addition, or feature pooling operations. Through aggregation processing, strip features in different directions and multi-scale convolution features can be integrated into a multi-scale local feature map containing rich context information.

[0258] In actual implementation, convolution operations or pooling operations can be used to aggregate multiple feature maps. Convolution operations are used to generate feature maps containing multi-level information, while pooling operations can compress the size of the feature map and reduce computational complexity. The generated multi-scale local feature map can more comprehensively describe the class features in the image, providing high-quality input data for subsequent class attention information extraction and label map generation.

[0259] The embodiment extracts multi-scale local features from the image data through a multi-scale feature extraction module, and the generated multi-scale local feature map contains rich multi-scale information and boundary feature information, which can effectively solve the deficiencies in the image feature extraction process in the prior art. Through multi-scale convolution operation and strip convolution operation, it is ensured that the model can capture feature information of target objects of different scales and directions, thereby improving the robustness and recognition accuracy of the model.

[0260] In one embodiment, S20 described above comprises:

[0261] S201, extracting class attention information from the multi-scale local feature map, the class attention information including class probability distribution of each pixel point;

[0262] S202, applying a Softmax function to the class attention information to analyze the class weight of each pixel point and generate a class weight map;

[0263] S203, applying an Argmax function to the class attention information to extract the maximum probability class index of each pixel point from the class probability distribution and generate a class index map;

[0264] S204, combining the class weight map and the class index map to generate the class matrix.

[0265] In the embodiment, the class attention information is extracted from the multi-scale local feature map, and the class matrix is constructed based on the class weight and the class index in the class attention information, which is an important step to realize class feature enhancement. By extracting the class probability distribution of each pixel point, calculating the class weight and the class index, and finally generating the class matrix, the class matrix is used for subsequent feature guidance and adjustment. The class matrix contains the class information of each pixel in the image, and is the basis for class feature weight adjustment in the subsequent steps.

[0266] The multi-scale local feature map contains the local feature representation of each pixel in the image. In order to further obtain the pixel-level class information, class probability analysis needs to be performed on the multi-scale local feature map. The extraction process of class attention information usually uses a classifier or a fully connected layer to map each pixel in the feature map to a class probability distribution. This means that the feature vector of each pixel point will be converted into a class probability vector, the length of which is equal to the total number of classes, and each value represents the probability that the pixel belongs to the corresponding class.

[0267] In the implementation process, a simple linear layer or a multi-layer perception (MLP) can be used to complete the mapping of the class probability distribution. The feature vector X pixel of each pixel point is input into the linear layer, and a class probability vector P pixel is output, where each element The probability value indicates that the pixel belongs to class c.

[0268] To normalize the class probability distribution into comparable weight values, a Softmax function needs to be applied to the class probability distribution. The role of the Softmax function is to convert an arbitrary vector of real numbers into a probability distribution, so that the sum of the output values is 1. Through the Softmax function, the weight value of each class can represent the relative likelihood of the pixel belonging to that class.

[0269] For each pixel point, the class probability vector P pixel is obtained after applying the Softmax function to the class probability vector P pixel , where each value represents the weight of the pixel point to the corresponding class. These class weight values can be regarded as the importance degree of the class, which is used for subsequent feature weight adjustment and class matrix construction.

[0270] In addition to the class weight map, the maximum probability class index of each pixel point needs to be extracted to represent the class that the pixel point is most likely to belong to. By applying the Argmax function to the class probability distribution, the class index corresponding to the maximum probability value of each pixel point can be extracted, generating a class index map.

[0271] The role of the Argmax function is to find the index position of the maximum value from the input vector. For each pixel point, the class probability distribution P pixel is obtained after applying the Argmax function to the class probability distribution P pixel . Each element of the class index map represents the class index value of the corresponding pixel point. This index value can be used in the subsequent class feature guidance process, enabling the model to adjust the feature map according to the class information.

[0272] The construction process of the class matrix is completed by combining the class weight map and the class index map. The class weight map contains the weight values of each pixel point to different classes, and the class index map contains the class index information of each pixel point. Combining these two images can generate a class matrix containing class weights and class indexes.

[0273] Each element M pixel of the class matrix can be represented as a binary tuple (W pixel , I pixel ), where W pixel is the class weight value and I pixel is the class index value. The class matrix is used in the subsequent class feature adjustment step, which dynamically adjusts the weight of the class feature to optimize the representation ability of the feature map.

[0274] The embodiment can realize accurate capture of the class information of each pixel point in the image by extracting class attention information from the multi-scale local feature map and constructing a class matrix based on the class weight and the class index. The generated class matrix contains class weight and class index information, so that the model can better understand the class distribution in the image. The construction process of the class matrix can not only effectively reduce the confusion between classes, but also enhance the attention ability of the model to detailed features, thereby improving the overall classification accuracy and segmentation effect of the model.

[0275] In one embodiment, S30 described above comprises:

[0276] S307, using the multi-scale local feature map as the current layer feature map, and obtaining the previous layer feature map of the current layer feature map;

[0277] S308, performing convolution operation on the current layer feature map and the previous layer feature map respectively to generate a preliminary adjustment signal;

[0278] S309, processing the preliminary adjustment signal through an activation function to generate a gating signal;

[0279] S310, performing channel weighting and spatial weighting adjustment on the multi-scale local feature map through the gating signal;

[0280] S311, multiplying the adjusted multi-scale local feature map and the class weight map of the class matrix pixel by pixel to generate a gated class feature map.

[0281] In the embodiment, a gating signal is generated by a gating unit, the multi-scale local feature map is adjusted through the gating signal, and the adjusted multi-scale local feature map is processed through channel weighting by the class matrix to generate a gated class feature map. In this process, by dynamically controlling the weight values of different feature channels and adjusting the weight of each pixel point in the feature map according to the class information, a more accurate class feature map can be generated. This operation not only enhances the expression ability of class information, but also effectively reduces the redundant information in the feature map, thereby improving the recognition performance of the overall model.

[0282] First, the construction of the gating unit needs the feature map of the current layer and the feature map of the previous layer as input. The current layer feature map is usually a multi-scale local feature map, and the previous layer feature map is a feature map generated in the previous stage. These two feature maps provide different image information: the current layer feature map contains rich local detailed features, and the previous layer feature map contains higher level abstract information. The gating unit generates a gating signal by fusing these two features to dynamically control the weight distribution of the feature map.

[0283] In order to generate the gating signal, convolution operations need to be performed on the current layer feature map and the previous layer feature map respectively. The role of convolution operation is to extract key information in the feature map, and at the same time to reduce the dimension of the feature map to reduce the computational complexity. Through the convolution operation, the preliminary adjustment signal can be obtained. Next, the preliminary adjustment signal is processed through the activation function to obtain the gating signal. The activation function usually selects the Sigmoid function or the ReLU function to ensure that the output value of the gating signal is within a reasonable range. Each value of the gating signal represents the importance of the corresponding feature channel, and the larger the value, the more important the information of the channel.

[0284] After obtaining the gating signal, the important features in the feature map are enhanced and the unimportant features are suppressed through channel weighting and spatial weighting adjustment of the multi-scale local feature map. Channel weighting adjustment refers to the separate weight adjustment of each channel of the feature map, thereby dynamically adjusting the contribution value of each channel. Spatial weighting adjustment refers to weight adjustment of each pixel point of the feature map according to the importance of the features at different positions. Through such an adjustment process, the feature map can more accurately reflect the key information in the image.

[0285] The adjusted multi-scale local feature map also needs to be further processed in combination with the class weight information of the class matrix. The class matrix includes a class weight map and a class index map, wherein the class weight map includes class weight values of each pixel point. These weight values reflect the correlation degree of each pixel point to different classes. Multiplying the adjusted feature map with the class weight map pixel by pixel can further enhance the class information expression ability of the feature map, thereby generating the final gated class feature map. The class feature map not only contains the detailed information in the original feature map, but also strengthens the class-related features, significantly improving the model's ability to distinguish different classes.

[0286] The embodiment dynamically adjusts the multi-scale local feature map through the gating unit, and performs pixel-by-pixel weighting processing in combination with the class weight information of the class matrix, thereby effectively enhancing the different class features in the feature map. The introduction of the gating unit enables the model to dynamically adjust the weight value of each feature channel according to the feature information of the current layer and the previous layer, thereby better capturing the key features in the image. In combination with the class weight information of the class matrix, the pixel-by-pixel weighting processing of the adjusted feature map can further strengthen the class feature expression in the feature map and reduce the confusion phenomenon of the class features.

[0287] In an embodiment, a feature fusion-based image label generation device is provided, which corresponds one-to-one to the feature fusion-based image label generation method in the above embodiment. Referring to Figure 3 , Figure 3A function module schematic diagram of a preferred embodiment of the image label generation device based on feature fusion of the application. Multi-scale feature extraction module 10, class attention extraction module 20, gate unit module 30, global class attention guide module 40, local class attention guide module 50, class attention fusion module 60, feature fusion module 70 and label generation module 80. The detailed description of each function module is as follows:

[0288] Multi-scale feature extraction module 10, for obtaining image data, performing multi-scale local feature extraction on the image data through the multi-scale feature extraction module, and generating a multi-scale local feature map;

[0289] Class attention extraction module 20, for extracting class attention information from the multi-scale local feature map, and constructing a class matrix based on the class weight and class index in the class attention information;

[0290] Gate unit module 30, for generating a gating signal based on the class matrix through the gate unit, and adjusting the multi-scale local feature map through the gating signal to generate a gated class feature map;

[0291] Global class attention guide module 40, for extracting global class features from the gated class feature map through the global class attention guide module;

[0292] Local class attention guide module 50, for extracting local class features from the gated class feature map through the local class attention guide module;

[0293] Class attention fusion module 60, for fusing the global class features and the local class features to generate a class attention guide feature map;

[0294] Feature fusion module 70, for fusing the class attention guide feature map and the multi-scale local feature map, and performing weighted processing on the fused feature map through the feature fusion head module to generate a final fused feature map;

[0295] Label generation module 80, for generating a class label map according to the final fused feature map.

[0296] In an embodiment, the class attention extraction module 20 is specifically used for:

[0297] Windowing the class label of each pixel in the class matrix to obtain a plurality of window feature matrices;

[0298] Extracting class labels from each window feature matrix respectively, and combining the class labels into a class label sequence in a predetermined order;

[0299] Subtracting each window feature matrix corresponding to the category label sequence to obtain a category difference matrix;

[0300] Adding a position index to the category difference matrix to obtain a category difference matrix containing position information;

[0301] Flattening the category difference matrix containing position information and offsetting negative values to construct a relative category index;

[0302] Adjusting the category matrix based on the relative category index to optimize the feature representation of the category matrix.

[0303] In an embodiment, the global category attention guide module 40 is specifically configured to:

[0304] Window segmentation is performed on the gated category feature map to generate a plurality of window feature blocks;

[0305] Each window feature block is linearly unfolded to construct an input sequence of multi-head attention;

[0306] The input sequence is mapped to query vectors, key vectors, and value vectors to form initial vectors of multi-head attention;

[0307] The initial vectors are subjected to scaled dot-product attention calculation, and the category attention information is incorporated into the attention score function as an additive term;

[0308] The output vectors of the attention score function are spliced and linearly transformed to generate the global category features.

[0309] In an embodiment, the local category attention guide module 50 is specifically configured to:

[0310] Branch convolution operations are performed on the gated category feature map using convolution kernels of different sizes to extract local features of different scales;

[0311] The output results of each branch convolution operation are spliced or added to obtain multi-scale local responses;

[0312] The multi-scale local responses are subjected to depth separable convolution operations, and the output feature maps of the depth separable convolution operations are activated and normalized to generate the local category features.

[0313] In an embodiment, the multi-scale feature extraction module 10 is specifically configured to:

[0314] A pre-trained basic convolutional network is constructed, and the image data is subjected to preliminary feature extraction through the basic convolutional network to obtain a basic feature map;

[0315] performing multi-scale convolution operation on the base feature map using convolution kernels of different sizes;

[0316] performing strip convolution operation on the output result of the multi-scale convolution operation to extract feature information in long strip or boundary shape;

[0317] performing aggregation processing on the output result of the strip convolution operation to generate the multi-scale local feature map.

[0318] In an embodiment, the category attention extraction module 20 is specifically configured to:

[0319] extract category attention information from the multi-scale local feature map, the category attention information including a category probability distribution of each pixel point;

[0320] apply a Softmax function to the category attention information to analyze a category weight of each pixel point and generate a category weight map;

[0321] apply an Argmax function to the category attention information to extract a maximum probability category index of each pixel point from the category probability distribution and generate a category index map;

[0322] combine the category weight map and the category index map to generate the category matrix.

[0323] In an embodiment, the gating unit module 30 is specifically configured to:

[0324] use the multi-scale local feature map as a current layer feature map and obtain a previous layer feature map of the current layer feature map;

[0325] perform convolution operation on the current layer feature map and the previous layer feature map respectively to generate a preliminary adjustment signal;

[0326] process the preliminary adjustment signal through an activation function to generate a gating signal;

[0327] perform channel weighting and spatial weighting adjustment on the multi-scale local feature map through the gating signal;

[0328] multiply the adjusted multi-scale local feature map and the category weight map of the category matrix pixel by pixel to generate a gated category feature map.

[0329] In an embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 4As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media, internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external user terminal through the network connection. The computer program is executed by the processor to realize the functions or steps of the server side of the feature fusion-based image label generation method.

[0330] In one embodiment, a computer device is provided, which can be a user terminal, and its internal structure diagram can be as shown in the figure. Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media, internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the functions or steps of the user side of the feature fusion-based image label generation method

[0331] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the following steps:

[0332] Obtain image data, perform multi-scale local feature extraction on the image data through a multi-scale feature extraction module, and generate a multi-scale local feature map;

[0333] Extract class attention information from the multi-scale local feature map, and construct a class matrix based on the class weight and class index in the class attention information;

[0334] Generate a gating signal through a gating unit, adjust the multi-scale local feature map through the gating signal, and perform channel-by-channel weighting processing on the adjusted multi-scale local feature map through the class matrix to generate a gated class feature map;

[0335] Extract global class features from the gated class feature map through a global class attention guiding module;

[0336] Extract local class features from the gated class feature map through a local class attention guiding module;

[0337] fuse the global class feature and the local class feature to generate a class attention guide feature map;

[0338] fuse the class attention guide feature map and the multi-scale local feature map, and perform weighted processing on the fused feature map through a feature fusion head module to generate a final fused feature map;

[0339] generate a class label map according to the final fused feature map.

[0340] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium. The computer program is executed by a processor to implement the following steps:

[0341] obtain image data, perform multi-scale local feature extraction on the image data through a multi-scale feature extraction module to generate a multi-scale local feature map;

[0342] extract class attention information from the multi-scale local feature map, and construct a class matrix based on the class weight and the class index in the class attention information;

[0343] generate a gating signal through a gating unit, adjust the multi-scale local feature map through the gating signal, and perform channel-by-channel weighted processing on the adjusted multi-scale local feature map through the class matrix to generate a gated class feature map;

[0344] extract a global class feature from the gated class feature map through a global class attention guide module;

[0345] extract a local class feature from the gated class feature map through a local class attention guide module;

[0346] fuse the global class feature and the local class feature to generate a class attention guide feature map;

[0347] fuse the class attention guide feature map and the multi-scale local feature map, and perform weighted processing on the fused feature map through a feature fusion head module to generate a final fused feature map;

[0348] generate a class label map according to the final fused feature map.

[0349] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the descriptions of the server side and the user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0350] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0351] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0352] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A feature fusion-based image label generation method, characterized in that, The method comprises the following steps: obtaining image data, performing multi-scale local feature extraction on the image data through a multi-scale feature extraction module to generate a multi-scale local feature map; extracting category attention information from the multi-scale local feature map, and constructing a category matrix based on the category weight and the category index in the category attention information; generating a gating signal through a gating unit, adjusting the multi-scale local feature map through the gating signal, and performing channel-by-channel weighting processing on the adjusted multi-scale local feature map through the category matrix to generate a gated category feature map; extracting global category features from the gated category feature map through a global category attention guiding module; extracting local category features from the gated category feature map through a local category attention guiding module; fusing the global category features and the local category features to generate a category attention guiding feature map; fusing the category attention guiding feature map and the multi-scale local feature map, and performing weighting processing on the fused feature map through a feature fusion head module to generate a final fused feature map; generating a category label map according to the final fused feature map. 2.The feature fusion based image label generation method of claim 1, wherein, After extracting the category attention information from the multi-scale local feature map and constructing the category matrix based on the category weight and the category index in the category attention information, the method further comprises the following steps: performing window division on the category label of each pixel in the category matrix to obtain a plurality of window feature matrices; extracting category labels from each window feature matrix respectively, and combining the category labels into a category label sequence in a preset order; performing subtraction operation on the category label sequence corresponding to each window feature matrix to obtain a category difference matrix; adding a position index to the category difference matrix to obtain a category difference matrix containing position information; performing planarization processing on the category difference matrix containing position information, and offsetting negative values to construct a relative category index; adjusting the category matrix based on the relative category index to optimize the feature representation of the category matrix. 3.The feature fusion based image label generation method of claim 1, wherein, The method of extracting global category features from the gated category feature map through a global category attention guiding module comprises the following steps: performing window segmentation on the gated category feature map to generate a plurality of window feature blocks; linearly expanding each window feature block to construct an input sequence of multi-head attention; mapping the input sequence to a query vector, a key vector and a value vector to form an initial vector of multi-head attention; performing scaling dot product attention calculation on the initial vector, and incorporating the category attention information into the attention score function as an additive term; splicing and linearly transforming the output vector of the attention score function to generate the global category features. 4.The feature fusion based image label generation method of claim 1, wherein, The method of extracting local category features from the gated category feature map through a local category attention guiding module comprises the following steps: performing branch convolution operation on the gated category feature map using convolution kernels of different sizes respectively to extract local features of different scales; performing splicing or adding operation on the output results of each branch convolution operation to obtain multi-scale local responses; The depth separable convolution operation is performed on the multi-scale local response, and an activation and normalization operation is performed on an output feature map of the depth separable convolution operation to generate the local class feature. 5.The feature fusion based image label generation method of claim 1, wherein, The multi-scale local feature extraction module is used to perform multi-scale local feature extraction on the image data to generate a multi-scale local feature map, including: A pre-trained basic convolutional network is constructed, and the basic convolutional network is used to perform preliminary feature extraction on the image data to obtain a basic feature map; Multi-scale convolution operations are performed on the basic feature map using convolution kernels of different sizes; A strip convolution operation is performed on an output result of the multi-scale convolution operation to extract feature information in a long strip or boundary shape; The multi-scale local feature map is generated by performing aggregation processing on an output result of the strip convolution operation. 6.The feature fusion based image label generation method of claim 1, wherein, Class attention information is extracted from the multi-scale local feature map, and a class matrix is constructed based on a class weight and a class index in the class attention information, including: The class attention information includes a class probability distribution of each pixel point; A Softmax function is applied to the class attention information to analyze a class weight of each pixel point to generate a class weight map; An Argmax function is applied to the class attention information to extract a maximum probability class index of each pixel point from the class probability distribution to generate a class index map; The class weight map and the class index map are combined to generate the class matrix. 7.The feature fusion based image tag generation method of claim 1, wherein, A gating signal is generated by a gating unit, the multi-scale local feature map is adjusted by the gating signal, and the adjusted multi-scale local feature map is processed by channel-by-channel weighting based on the class matrix to generate a gated class feature map, including: The multi-scale local feature map is used as a current layer feature map, and a previous layer feature map of the current layer feature map is obtained; Convolution operations are performed on the current layer feature map and the previous layer feature map to generate a preliminary adjustment signal; The preliminary adjustment signal is processed by an activation function to generate the gating signal; The multi-scale local feature map is adjusted by channel weighting and spatial weighting based on the gating signal; The adjusted multi-scale local feature map is multiplied by a class weight map of the class matrix pixel by pixel to generate the gated class feature map.

8. An image label generation apparatus based on feature fusion, characterized by, The image label generation device based on feature fusion includes: A multi-scale feature extraction module is used to obtain image data, perform multi-scale local feature extraction on the image data by the multi-scale feature extraction module, and generate a multi-scale local feature map. A class attention extraction module is used to extract class attention information from the multi-scale local feature map, and construct a class matrix based on a class weight and a class index in the class attention information. A gating unit module is used to generate a gating signal by a gating unit, adjust the multi-scale local feature map by the gating signal, and perform channel-by-channel weighting processing on the adjusted multi-scale local feature map based on the class matrix to generate a gated class feature map. The global class attention guiding module is configured to extract global class features from the gated class feature map; The local class attention guiding module is configured to extract local class features from the gated class feature map; The class attention fusion module is configured to fuse the global class features and the local class features to generate a class attention guiding feature map; The feature fusion module is configured to fuse the class attention guiding feature map and the multi-scale local feature map, perform weighted processing on the fused feature map through a feature fusion head module, and generate a final fused feature map; The label generation module is configured to generate a class label map according to the final fused feature map.

9. A computer device, comprising: The computer device comprises a memory, a processor, and a feature fusion-based image label generation program stored on the memory and capable of running on the processor. When the feature fusion-based image label generation program is executed by the processor, the steps of the feature fusion-based image label generation method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The storage medium stores a feature fusion-based image label generation program. When the feature fusion-based image label generation program is executed by the processor, the steps of the feature fusion-based image label generation method according to any one of claims 1-7 are implemented.

Images