A semantic understanding method based on cross-modal image target detection

Abstract

The application discloses a semantic understanding method based on cross-modal image target detection, relates to the technical field of target detection, and extracts features of visible light remote sensing images and infrared remote sensing images through a multi-branch convolutional neural network; performs multi-scale fusion operations on the extracted features respectively through a feature pyramid network; inputs a feature map obtained through the fusion operation into a region proposal network to generate visible light remote sensing image feature candidate regions and infrared remote sensing image feature candidate regions, and extracts features from the candidate regions; matches the candidate region features through a cross-modal similarity measurement network to obtain an associated matching result; determines visible light semantic feature maps and infrared semantic feature maps according to the associated matching result; and obtains target detection results and target semantic descriptions based on the visible light semantic feature maps and the infrared semantic feature maps. In this way, the target detection accuracy and the target description accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of object detection technology, and in particular to a semantic understanding method based on cross-modal image object detection. Background Technology

[0002] With the rapid development of aerospace remote sensing technology, multimodal remote sensing images are playing an increasingly important role in areas such as Earth / sea observation, disaster early warning, regional governance, and environmental protection. Visible light remote sensing images can provide rich texture details and color information, but they are significantly limited by lighting conditions, resulting in poor imaging quality at night or in adverse weather conditions. Infrared remote sensing images, based on the thermal radiation characteristics of targets, have all-weather operation capabilities and strong penetration, but their spatial resolution is relatively low and they lack detailed information. Therefore, fusing visible light and infrared image data to achieve cross-modal target association detection and semantic understanding is of great significance for improving intelligent image analysis capabilities, such as remote sensing interpretation.

[0003] In related technologies, deep learning-based image detection methods have achieved significant results in single-modal image target detection tasks, but these methods are difficult to directly apply to cross-modal scenes. In recent years, some studies have attempted to extend deep learning methods to the field of cross-modal image matching, such as metric learning-based methods that achieve image matching by learning similarity measures in cross-modal feature spaces. However, these methods usually require a large number of paired samples for training and are mainly aimed at image-level matching, making it difficult to achieve accurate target-level association in large-scale images. In terms of image semantic understanding, image description generation technology has made significant progress. Typical methods adopt an encoder-decoder architecture, extracting image features through convolutional neural networks and then using recurrent neural networks to generate natural language descriptions. However, existing methods are mainly aimed at natural scene images, and their application in fields such as remote sensing images with large scenes, multiple targets, and high specialization is limited. They also lack organic integration with target detection. In addition, existing cross-modal image processing methods mostly adopt multi-stage serial processing flows, such as performing image registration first, then target detection, and finally semantic analysis. This serial architecture leads to error accumulation and low computational efficiency. There is a lack of a unified framework that can perform cross-modal association detection and semantic understanding end-to-end. Summary of the Invention

[0004] In view of this, this application provides a semantic understanding method based on cross-modal image target detection to improve the accuracy of target detection and target description.

[0005] The objective of this application can be achieved through the following technical solutions:

[0006] The first aspect of this application is to provide a semantic understanding method for cross-modal image object detection. The object detection model for object detection includes a multi-branch convolutional neural network, a feature pyramid network, a region proposal network, a cross-modal similarity measurement network, a hierarchical convolutional encoder, and a detection head, comprising:

[0007] Acquire visible light and infrared remote sensing images;

[0008] Visible light remote sensing images and infrared remote sensing images are input into the target detection model. Feature extraction is performed on the visible light remote sensing images and infrared remote sensing images through a multi-branch convolutional neural network to obtain feature maps of visible light remote sensing images and infrared remote sensing images.

[0009] The feature pyramid network is used to perform multi-scale fusion operations on the feature maps of visible light remote sensing images and infrared remote sensing images respectively to obtain the enhanced feature maps of visible light remote sensing images and infrared remote sensing images.

[0010] The enhanced feature maps of visible light remote sensing images and infrared remote sensing images are respectively input into the region proposal network to generate candidate regions for visible light remote sensing image features and candidate regions for infrared remote sensing image features. Features are then extracted from the candidate regions for visible light remote sensing image features and candidate regions for infrared remote sensing image features, respectively, to obtain the visible light candidate region features and the infrared candidate region features.

[0011] The visible light candidate region features and infrared candidate region features are matched by a cross-modal similarity measurement network to obtain the association matching results, which include target localization information and confidence information.

[0012] Based on the association matching results, the RoI features of the target region are extracted from the visible light remote sensing image and the infrared remote sensing image respectively, thus obtaining the visible light RoI features and the infrared RoI features;

[0013] Visible light RoI features and infrared RoI features are input into a hierarchical convolutional encoder, and features are extracted by the hierarchical convolutional encoder to obtain visible light semantic feature maps and infrared semantic feature maps.

[0014] The visible light semantic feature map and the infrared semantic feature map are input into the detection head to obtain the target detection result, which includes the predicted target category and the predicted target bounding box.

[0015] Based on visible light semantic feature maps and infrared semantic feature maps, a semantic description of the target is generated.

[0016] In an optional embodiment, the target detection model further includes a two-layer LSTM decoder and a fully connected layer to generate a target semantic description based on visible light semantic feature maps and infrared semantic feature maps, including:

[0017] Based on the hidden state, visible light semantic feature map and infrared semantic feature map of the two-layer LSTM decoder in the previous time step, the attention weight is calculated, and the visible light semantic feature map and infrared semantic feature map are weighted based on the attention weight to obtain the weighted feature vector.

[0018] The word embedding vector and weighted feature vector predicted in the previous time step are used as input to the two-layer LSTM decoder, and the current hidden state is output.

[0019] The current hidden state and the weighted feature vector are input into the fully connected layer to generate a semantic description of the target.

[0020] In one optional embodiment, the attention weights include visible light spatial attention weights and infrared spatial attention weights. The attention weights are calculated based on the hidden state of the dual-layer LSTM decoder at the previous time step, the visible light semantic feature map, and the infrared semantic feature map, including:

[0021] Global average pooling is performed on the visible light semantic feature map to obtain the average feature of visible light spatial description;

[0022] Performing a global max pooling operation on the visible light semantic feature map yields the maximum feature for the visible light spatial description.

[0023] Global average pooling is performed on the infrared semantic feature map to obtain the average feature of the infrared spatial description.

[0024] Global max pooling is performed on the infrared semantic feature map to obtain the maximum feature describing the infrared space.

[0025] Convolutional processing is performed on the average feature and the maximum feature of the visible light spatial description respectively to obtain the average enhanced feature and the maximum enhanced feature of the visible light spatial description.

[0026] The average enhancement feature and the maximum enhancement feature of the visible light spatial description are fused to obtain the initial weights for visible light spatial attention;

[0027] Convolutional processing is performed on the average feature and the maximum feature of infrared spatial description respectively to obtain the average enhanced feature and the maximum enhanced feature of infrared spatial description.

[0028] The average enhancement feature and the maximum enhancement feature of the infrared spatial description are fused to obtain the initial weights for infrared spatial attention.

[0029] Based on the hidden state of the previous time step, the initial weights of visible light spatial attention and infrared spatial attention are adjusted using the dual-layer LSTM decoder to obtain the visible light spatial attention weights and infrared spatial attention weights.

[0030] In one optional embodiment, the attention weights include visible light channel attention weights and infrared channel attention weights. The attention weights are calculated based on the hidden state of the dual-layer LSTM decoder at the previous time step, the visible light semantic feature map, and the infrared semantic feature map, including:

[0031] Global average pooling is performed on the visible light semantic feature map to obtain the average features describing the visible light channels;

[0032] Performing a global max pooling operation on the visible light semantic feature map yields the maximum feature describing the visible light channel.

[0033] Global average pooling is performed on the infrared semantic feature map to obtain the average feature describing the infrared channels;

[0034] Global max pooling is performed on the infrared semantic feature map to obtain the maximum feature describing the infrared channel;

[0035] The average feature and the maximum feature of the visible light channel description are mapped to obtain the average enhancement feature and the maximum enhancement feature of the visible light channel description, respectively.

[0036] The average enhancement feature and the maximum enhancement feature of the visible light channel are fused to obtain the initial weights for attention in the visible light channel.

[0037] Convolutional processing is performed on the average feature and the maximum feature of the infrared channel description respectively to obtain the average enhanced feature and the maximum enhanced feature of the infrared channel description.

[0038] The average enhancement feature and the maximum enhancement feature of the infrared channel are fused to obtain the initial weights for infrared channel attention.

[0039] The initial attention weights of the visible light channel and the infrared channel are adjusted based on the hidden state of the previous time step using a dual-layer LSTM decoder, thus obtaining the attention weights of the visible light channel and the infrared channel.

[0040] In one optional embodiment, the attention weights include inter-modal attention weights for visible light and inter-modal attention weights for infrared light. The attention weights are calculated based on the hidden state of the dual-layer LSTM decoder at the previous time step, the visible light semantic feature map, and the infrared semantic feature map, including:

[0041] Global average pooling is performed on the visible light semantic feature map to obtain the overall average feature description of visible light;

[0042] Global average pooling is performed on the infrared semantic feature map to obtain the average feature of the overall infrared description;

[0043] Initial weights for attention between visible light modalities are determined based on the average features of the overall visible light description.

[0044] Initial weights for intermodal attention in infrared mode are determined based on the average features of the overall infrared description.

[0045] Based on the hidden state of the previous time step, the initial attention weights between visible light modes and between infrared modes are adjusted using the dual-layer LSTM decoder to obtain the attention weights between visible light modes and between infrared modes.

[0046] In one optional embodiment, the method for training the object detection model includes:

[0047] Obtain the training sample set, which includes sample visible light remote sensing images, sample visible light remote sensing images, sample target detection box labels, and sample target semantic description labels corresponding to each sample target;

[0048] Input the sample visible light remote sensing image and the sample infrared remote sensing image into the target detection model to be trained to obtain the sample target detection results and sample target semantic description information;

[0049] The total loss is calculated based on the sample target detection results, sample target semantic description information, sample target detection box labels, and sample target semantic description labels corresponding to each sample target.

[0050] The model parameters of the target detection model to be trained are updated based on the total loss.

[0051] In one alternative embodiment, the total loss includes: contrast loss, triplet loss, and regularization loss.

[0052] In one alternative embodiment, the multi-branch convolutional neural network includes shared convolutional layers, visible light-specific convolutional layers, and infrared-specific convolutional layers.

[0053] In one alternative embodiment, the cross-modal similarity measurement network includes two weight-sharing branches, each branch comprising multiple convolutional layers and normalization layers.

[0054] In an optional embodiment, the target detection model further includes a residual encoder. Before inputting visible light RoI features and infrared RoI features into a hierarchical convolutional encoder to extract features and obtain visible light semantic feature maps and infrared semantic feature maps, the model further includes:

[0055] The visible light RoI features and infrared RoI features are residual encoded by a residual encoder to obtain the visible light RoI enhanced features and infrared RoI enhanced features.

[0056] Visible light RoI features and infrared RoI features are input into a hierarchical convolutional encoder. The encoder extracts features to obtain visible light semantic feature maps and infrared semantic feature maps, including:

[0057] The visible light RoI enhancement features and infrared RoI enhancement features are input into a hierarchical convolutional encoder, and the features are extracted by the hierarchical convolutional encoder to obtain the visible light semantic feature map and the infrared semantic feature map.

[0058] Compared with existing technologies, the semantic understanding method for cross-modal image target detection provided in this application extracts features from visible light and infrared remote sensing images using a multi-branch convolutional neural network; performs multi-scale fusion operations on the extracted features using a feature pyramid network; inputs the fused feature maps into a region proposal network to generate candidate feature regions for both visible light and infrared remote sensing images, and extracts features from these candidate regions; matches the candidate region features using a cross-modal similarity measurement network to obtain association matching results; determines visible light and infrared semantic feature maps based on the association matching results; and obtains target detection results and target semantic descriptions based on the visible light and infrared semantic feature maps. This improves the accuracy of target detection and target description. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A flowchart illustrating a semantic understanding method based on cross-modal image target detection provided in an embodiment of this application;

[0061] Figure 2 This is a schematic diagram of a structure for extracting features from visible light remote sensing images and infrared remote sensing images using a multi-branch convolutional neural network, as provided in an embodiment of this application, to obtain feature maps of visible light remote sensing images and infrared remote sensing images.

[0062] Figure 3 This is a schematic diagram of a feature pyramid network provided in an embodiment of this application;

[0063] Figure 4 This is a schematic diagram illustrating how a semantic feature map is first weighted by channel attention weights and then by spatial attention weights, as provided in an embodiment of this application.

[0064] Figure 5 A schematic diagram of the target association matching detection result provided in the embodiments of this application;

[0065] Figure 6 A schematic diagram of a training loss curve provided in an embodiment of this application;

[0066] Figure 7 A schematic diagram of the verification loss curve provided in the embodiments of this application;

[0067] Figure 8 A schematic diagram of a BLEU-1 score curve provided in an embodiment of this application;

[0068] Figure 9 A schematic diagram illustrating the semantic description of infrared remote sensing images for various scenarios and types of targets provided in the embodiments of this application;

[0069] Figure 10 A schematic diagram of the structure of the target detection model provided in the embodiments of this application;

[0070] Figure 11 This is a schematic diagram of a training process for a recurrent neural network provided in an embodiment of this application;

[0071] Figure 12 A structural block diagram of a semantic understanding device based on cross-modal image target detection provided in an embodiment of this application;

[0072] Figure 13 This is a structural block diagram of an electronic device for implementing a semantic understanding method based on cross-modal image target detection, as provided in an embodiment of this application. Detailed Implementation

[0073] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0074] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0075] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0076] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B", or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0077] To address the technical problems existing in related technologies, embodiments of this application provide a semantic understanding method based on cross-modal image target detection.

[0078] The semantic understanding method based on cross-modal image target detection provided in this application can be executed by an electronic device, such as a terminal or a server. The terminal can be a smartphone, tablet, laptop, or other similar device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. It is understood that this application does not specifically limit the executing entity of the semantic understanding method based on cross-modal image target detection.

[0079] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments described below are used to explain the technical solution of this application and are not intended to limit actual use.

[0080] To address the technical problems existing in related technologies, embodiments of this application provide a semantic understanding method based on cross-modal image target detection, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating a semantic understanding method based on cross-modal image target detection provided in an embodiment of this application. It should be noted that the steps shown may be executed in a different logical order than those shown in the flowchart. The method may include the following steps S101 to S109.

[0081] Step S101: Acquire visible light remote sensing images and infrared remote sensing images.

[0082] It should be noted that details on acquiring visible light remote sensing images and infrared remote sensing images can be found in relevant technologies, and will not be elaborated upon here.

[0083] Step S102: Input the visible light remote sensing image and the infrared remote sensing image into the target detection model, and extract features from the visible light remote sensing image and the infrared remote sensing image through a multi-branch convolutional neural network to obtain the visible light remote sensing image feature map and the infrared remote sensing image feature map.

[0084] In one alternative embodiment, the multi-branch convolutional neural network includes shared convolutional layers, visible light-specific convolutional layers, and infrared-specific convolutional layers.

[0085] In one specific embodiment, the visible light remote sensing image and the infrared remote sensing image are respectively passed through shared convolutional layers Conv1-Conv3, where the weights of the two ends of the shared convolutional layers are shared. Then, the visible light remote sensing image enters the visible light-specific convolutional layers Conv4-Conv6, and the infrared remote sensing image enters the infrared-specific convolutional layers Conv4-Conv5, to obtain the visible light remote sensing image feature map and the infrared remote sensing image feature map.

[0086] In a more specific embodiment, Figure 2 This application provides a schematic diagram of a structure for extracting features from visible light and infrared remote sensing images using a multi-branch convolutional neural network, resulting in visible light and infrared remote sensing image feature maps. Figure 2 As shown, the size of a visible light remote sensing image can be 224. 224 3. The size of the infrared remote sensing image can be 224. 224 1. Shared Convolutional Layers: The same three-layer convolutional layer with shared weights is used for both visible light and infrared remote sensing images to extract common features such as edges and contours. The shared convolutional layers include: the first convolutional layer (Conv1) with 64 3×3 kernels, stride = 1, and ReLU activation function; the first max pooling layer (MaxPool1) using a 2×2 sliding window with stride = 2; the second convolutional layer (Conv2) with 128 3×3 kernels, stride = 1, and ReLU activation function; the second pooling layer (MaxPool2) using a 2×2 sliding window with stride = 2; and the third convolutional layer (Conv3) with 256 3×3 kernels, stride = 1, and ReLU activation function.

[0087] To address the unique characteristics of both visible light and infrared remote sensing images, an asymmetric twin structure was employed to extract more effective features from each image. The visible light-specific convolutional layers include: a first convolutional layer (Conv4_v) with 512 3×3 kernels, capturing details using small-scale kernels; a second convolutional layer (Conv5_v) with 512 3×3 kernels; a third convolutional layer (Conv6_v) with 512 3×3 kernels; a pooling layer (MaxPool) using a 2×2 sliding window; and a fully connected layer (FC1_v) with 4096 dimensions. These visible light-specific layers are used to extract deep, fine-grained features.

[0088] The infrared-specific convolutional layer comprises: a first convolutional layer, denoted as Conv4_ir, containing 256 5×5 convolutional kernels, employing large-scale kernels to capture thermal radiation distribution; a second convolutional layer, denoted as Conv5_ir, containing 512 5×5 convolutional kernels; a pooling layer, denoted as MaxPool, using a 2×2 sliding window for pooling operations; and a fully connected layer, denoted as FC1_ir, with a dimension of 2048. The infrared-specific convolutional layer is used to extract coarse-grained features from shallower layers.

[0089] In a more specific embodiment, the visible light remote sensing image feature map is represented by FC2_v, which has a dimension of 2048, and the infrared remote sensing image feature map is represented by FC2_ir, which also has a dimension of 2048.

[0090] In a more specific embodiment, the shared convolutional layer can be a VGG (Visual Geometry Group) 16.

[0091] In this step, the shared layer learns cross-modal general features, while the modality-specific layer learns unique features specific to its respective modality.

[0092] Step S103: Perform multi-scale fusion operations on the visible light remote sensing image feature map and the infrared remote sensing image feature map respectively through the feature pyramid network to obtain the visible light remote sensing image enhancement feature map and the infrared remote sensing image enhancement feature map.

[0093] In one alternative embodiment, Figure 3 This is a schematic diagram of a feature pyramid network provided in an embodiment of this application, such as... Figure 3 As shown, Layer 1 (Feature Pyramid Network Layer 1): Acquires high-resolution feature maps, using 1 / 4 of the initial resolution of the input image, enabling the extraction of details and small targets. Layer 2 (Feature Pyramid Network Layer 2): Acquires medium-to-high-resolution feature maps, with a resolution half that of Layer 1, handling larger targets. Layer 3 (Feature Pyramid Network Layer 3): Acquires medium-resolution feature maps, with a resolution 1 / 4 that of Layer 1, handling even larger targets. Layer 4 (Feature Pyramid Network Layer 4): Acquires low-resolution feature maps, with a resolution 1 / 8 that of Layer 1, handling larger targets. Layer 5 (Feature Pyramid Network Layer 5): Acquires the lowest-resolution feature maps, with a resolution 1 / 16 that of Layer 1, primarily handling large global targets.

[0094] A five-layer pyramid structure is employed, with resolution decreasing progressively from 1 / 4 to 1 / 64, generating low-resolution feature maps. The output of each layer represents a feature map at a different scale. Simultaneously, features from different scales are fused through top-down paths and lateral connections, thus adapting to targets of varying sizes and distributions. This multi-scale pyramid structure simultaneously extracts shallow detail features and deep semantic features.

[0095] Step S104: Input the enhanced feature map of the visible light remote sensing image and the enhanced feature map of the infrared remote sensing image into the region proposal network to generate candidate regions for features of the visible light remote sensing image and the infrared remote sensing image. Extract features from the candidate regions for features of the visible light remote sensing image and the infrared remote sensing image respectively to obtain the visible light candidate region features and the infrared candidate region features.

[0096] It should be noted that the introduction of the Region Proposal Network (RPN) enables the generation of candidate regions on feature maps of different scales. For small targets, recommendations are primarily generated on high-resolution feature maps; for large targets, recommendations are primarily generated on low-resolution feature maps. Anchor boxes of different sizes and proportions are set for each scale, covering target scales ranging from 16×16 to 512×512 pixels.

[0097] In an optional embodiment, the target detection model further includes a weighted shared convolutional encoder, which inputs the visible light remote sensing image feature candidate region and the infrared remote sensing image feature candidate region into the weighted shared encoder to obtain visible light candidate region features and infrared candidate region features.

[0098] Step S105: Match the visible light candidate region features and the infrared candidate region features through a cross-modal similarity measurement network to obtain the association matching results.

[0099] In one alternative embodiment, the association matching result includes target location information and confidence information.

[0100] It should be noted that target localization information describes the spatial location and geometric extent of the target in the image, and may include the coordinates of the detection box. Confidence information measures the degree of certainty that a region contains the target and the certainty of the target's category.

[0101] In another alternative embodiment, the cross-modal similarity measurement network includes two weight-sharing branches, each of which includes multiple convolutional layers and normalization layers.

[0102] In one specific embodiment, the visible light candidate region features and the infrared candidate region features are processed by two weight-sharing branches to obtain visible light candidate region embedding vectors and infrared candidate region embedding vectors respectively; the visible light candidate region embedding vectors and infrared candidate region embedding vectors are concatenated or subtracted to obtain a fused embedding vector; the fused embedding vector is input into the discriminant network to output the probability that the candidate regions belong to the same target.

[0103] This step enables precise association and location of the target.

[0104] Step S106: Extract the RoI features of the target region from the visible light remote sensing image and the infrared remote sensing image respectively based on the association matching results to obtain the visible light RoI features and the infrared RoI features.

[0105] It should be noted that the target area is the area that contains the target.

[0106] In another optional embodiment, the target detection model further includes a residual encoder. Before the visible light RoI (Region of Interest) features and infrared RoI features are input into a hierarchical convolutional encoder to extract features and obtain the visible light semantic feature map and infrared semantic feature map, the model further includes:

[0107] The visible light RoI features and infrared RoI features are residual encoded by a residual encoder to obtain the visible light RoI enhanced features and infrared RoI enhanced features.

[0108] Step S107: Input the visible light RoI features and infrared RoI features into the hierarchical convolutional encoder, extract the features through the hierarchical convolutional encoder, and obtain the visible light semantic feature map and the infrared semantic feature map.

[0109] In one optional embodiment, visible light RoI features and infrared RoI features are input into a hierarchical convolutional encoder, and features are extracted by the hierarchical convolutional encoder to obtain visible light semantic feature maps and infrared semantic feature maps, including:

[0110] The visible light RoI enhancement features and infrared RoI enhancement features are input into a hierarchical convolutional encoder, and the features are extracted by the hierarchical convolutional encoder to obtain the visible light semantic feature map and the infrared semantic feature map.

[0111] In one specific embodiment, the hierarchical convolutional encoder is a VGG19 convolutional neural network with fully connected layers and pooling layers removed.

[0112] In another specific embodiment, a batch normalization layer is added between the convolutional layer and the nonlinear layer. That is, in stochastic gradient descent, the input data is normalized on a batch basis, which speeds up the fitting speed of the convolutional neural network during the training process and reduces the sensitivity of the data to the initial values ​​of the network.

[0113] Step S108: Input the visible light semantic feature map and the infrared semantic feature map into the detection head to obtain the target detection result.

[0114] It should be noted that the object detection results include the predicted object category and the predicted object bounding box.

[0115] Step S109: Generate a target semantic description based on the visible light semantic feature map and the infrared semantic feature map.

[0116] In an optional embodiment, the target detection model further includes a two-layer LSTM (Long Short-Term Memory) decoder and a fully connected layer to generate a target semantic description based on visible light semantic feature maps and infrared semantic feature maps, including:

[0117] Based on the hidden state, visible light semantic feature map, and infrared semantic feature map of the previous time step in the dual-layer LSTM decoder, attention weights are calculated. The visible light semantic feature map and infrared semantic feature map are weighted based on the attention weights to obtain a weighted feature vector. The word embedding vector predicted in the previous time step and the weighted feature vector are used as input to the dual-layer LSTM decoder to output the current hidden state. The current hidden state and the weighted feature vector are input into the fully connected layer to generate the target semantic description.

[0118] In one specific embodiment, the attention weights include visible light spatial attention weights and infrared spatial attention weights. Based on the hidden state of the dual-layer LSTM decoder at the previous time step, the visible light semantic feature map, and the infrared semantic feature map, the attention weights are calculated, including:

[0119] Global average pooling is performed on the visible light semantic feature map to obtain the visible light spatial description average feature; global max pooling is performed on the visible light semantic feature map to obtain the visible light spatial description maximum feature; global average pooling is performed on the infrared semantic feature map to obtain the infrared spatial description average feature; global max pooling is performed on the infrared semantic feature map to obtain the infrared spatial description maximum feature; convolution is performed on the visible light spatial description average feature and the visible light spatial description maximum feature respectively to obtain the visible light spatial description average enhanced feature and the visible light spatial description maximum enhanced feature; the visible light spatial description average enhanced feature and the visible light spatial description maximum enhanced feature are fused to obtain the visible light spatial attention initial weights; convolution is performed on the infrared spatial description average feature and the infrared spatial description maximum feature respectively to obtain the infrared spatial description average enhanced feature and the infrared spatial description maximum enhanced feature; the infrared spatial description average enhanced feature and the infrared spatial description maximum enhanced feature are fused to obtain the infrared spatial attention initial weights; the visible light spatial attention initial weights and the infrared spatial attention initial weights are adjusted based on the hidden state of the two-layer LSTM decoder in the previous time step to obtain the visible light spatial attention weights and the infrared spatial attention weights.

[0120] In a more specific embodiment, the semantic feature map (which can be a visible light semantic feature map and / or an infrared semantic feature map) can be used... This indicates that C is the number of channels, H is the height, and W is the width.

[0121] Spatial descriptive average features (which can be visible light spatial descriptive average features or infrared spatial descriptive average features) can be used This indicates that the maximum spatial description feature (which can be either the maximum spatial description feature in visible light or the maximum spatial description feature in infrared light) can be used... It means that, among them, It is the first The semantic feature map of the channel, where C is the number of channels.

[0122] Spatial description average enhancement features (spatial description average enhancement features can be visible light spatial description average enhancement features or infrared spatial description average enhancement features) can be used This indicates that the spatial description maximum enhancement feature (which can be the visible light spatial description average enhancement feature or the infrared spatial description maximum enhancement feature) can be used... This indicates that k is the kernel size of the convolutional layer. This is the activation function used for smooth normalization.

[0123] For example, the activation function can be sigmoid (logistic-sigmoid) or ReLU (rectified linear units).

[0124] In a more specific embodiment, the spatial attention weights are calculated using the following formula:

[0125]

[0126] in, The sigmoid function represents the spatial attention weights.

[0127] In another, more specific embodiment, the attention weights include visible light channel attention weights and infrared channel attention weights. Based on the hidden state of the dual-layer LSTM decoder at the previous time step, the visible light semantic feature map, and the infrared semantic feature map, the attention weights are calculated, including: performing a global average pooling operation on the visible light semantic feature map to obtain the average feature describing the visible light channel; performing a global max pooling operation on the visible light semantic feature map to obtain the maximum feature describing the visible light channel; performing a global average pooling operation on the infrared semantic feature map to obtain the average feature describing the infrared channel; performing a global max pooling operation on the infrared semantic feature map to obtain the maximum feature describing the infrared channel; and mapping the average feature describing the visible light channel and the maximum feature describing the visible light channel to obtain... The average enhancement feature and the maximum enhancement feature of the visible light channel are described. These two features are then fused to obtain the initial attention weights for the visible light channel. Convolutional processing is performed on the average feature and the maximum enhancement feature of the infrared channel to obtain the average enhancement feature and the maximum enhancement feature of the infrared channel. These two features are then fused to obtain the initial attention weights for the infrared channel. Based on the hidden state of the two-layer LSTM decoder at the previous time step, the initial attention weights for the visible light channel and the infrared channel are adjusted to obtain the attention weights for the visible light channel and the infrared channel.

[0128] In a more specific embodiment, the channel description average feature can be used This indicates that the most significant feature of a channel description can be represented by... It means that, among them, , Let h represent the feature value of the c-th channel at position (h, w), where h represents the row index and w represents the column index. This represents the average value of the c-th channel. This represents a vector consisting of all channel GAP values. A vector consisting of all channel GMP values.

[0129] Channel description average enhancement features can be used This indicates that the channel description of the maximum enhancement feature can be used... express, It is the ReLU activation function. It is the sigmoid activation function. This is a learnable weight matrix.

[0130] In a more specific embodiment, the channel attention weights are calculated using the following formula:

[0131]

[0132] In a more specific embodiment, Figure 4 This is a schematic diagram illustrating how a semantic feature map is first weighted by channel attention weights and then by spatial attention weights, as provided in an embodiment of this application. Figure 4 Including (a) and (b), as shown in (a), max pooling and average pooling operations are performed on the feature map (the semantic feature map mentioned above), and the max pooling and average pooling results are obtained through a weight-sharing network. The two results are XORed and then activated by a Sigmoid function to obtain a channel attention-enhanced selected feature map (channel attention feature map). The shared network includes an input layer, a fully connected layer, and an output layer. As shown in (b), the selected feature map obtained in the first step is subjected to averaging and max pooling operations in the spatial domain, and the average output and the max output are concatenated. The concatenated result is input into a convolutional layer and then activated by a Sigmoid function to finally obtain a spatial attention-enhanced feature map (spatial attention feature map).

[0133] In one specific embodiment, the attention weights include inter-modal attention weights for visible light and inter-modal attention weights for infrared light. Based on the hidden state of the dual-layer LSTM decoder at the previous time step, the visible light semantic feature map, and the infrared semantic feature map, the attention weights are calculated, including:

[0134] Global average pooling is performed on the visible light semantic feature map to obtain the overall visible light descriptive average feature; global average pooling is performed on the infrared semantic feature map to obtain the overall infrared descriptive average feature; initial weights for inter-modal attention in visible light are determined based on the overall visible light descriptive average feature; initial weights for inter-modal attention in infrared are determined based on the overall infrared descriptive average feature; the initial weights for inter-modal attention in visible light and infrared are adjusted based on the hidden state of the two-layer LSTM decoder in the previous time step to obtain the inter-modal attention weights for visible light and infrared.

[0135] In a more specific embodiment, the overall visible light descriptive average feature can be used This indicates that the average feature of the overall infrared description can be used... express.

[0136] In a more specific embodiment, the initial attention weights between visible light modalities and between infrared modalities are determined using an MLP (Multi-Layer Perceptron) with the following formula:

[0137]

[0138]

[0139] in, This represents the initial attention weights between visible light modes, used to represent the contribution weights of visible light modes to infrared modes. This represents the initial attention weights between infrared modes, used to indicate the contribution weights of infrared modes to visible light modes. and Represents the learnable weight coefficients. and Indicates the bias term. () represents the normalization function.

[0140] It's important to note that the LSTM receives word embedding vectors and attention-weighted visual features, outputting intermediate hidden states; the second LSTM receives the output of the first layer and produces the final word prediction. This two-layer structure enhances the network's expressive power, enabling it to capture more complex semantic dependencies. Regarding semantically guided word generation, when generating each word, in addition to considering the previously generated word sequence, a semantic topic vector is introduced as additional input. This semantic topic vector is obtained by global average pooling of the encoder features, representing the high-level semantics of the entire target, ensuring that the generated description remains semantically coherent and consistent.

[0141] In an alternative embodiment, during semantic description generation, this application uses a linear layer to convert the output of the two-layer LSTM decoder into a score for each word in the vocabulary dictionary. Directly selecting the word with the highest score and using it to predict the next word is not necessarily optimal, as subsequent words in the semantic sequence are influenced by previously selected words. Therefore, this model employs beam search (sequence search) to find the overall optimal semantic sequence.

[0142] The specific steps of beam search are as follows: In the decoding stage, first select the K highest-scoring first words, and then select the K highest-scoring second words for each of these K highest-scoring first words. Combine these two to form a first word plus a second word combination. Then, select the K highest-scoring third words for this combination to form a first, second, and third word combination, and so on. Finally, select the sequence with the highest overall score as the output semantic sequence of the model.

[0143] To measure the accuracy of semantic understanding and description generation, this application uses the BLEU (Bilingual Evaluation Understudy)-1 metric. This method is used to determine the similarity of sentences; the closer the translation is to the reference translation, the better the result. In image understanding problems, the description generated by the neural network based on image features is generally compared with several annotations corresponding to the original image, and then word fragment scores of different lengths are obtained. The higher the score, the better the translation, meaning the image description is more consistent with the image content. BLEU uses an N-gram (word fragment, where N represents the word length, usually 1 to 4, corresponding to BLEU-1 to BLEU-4 respectively) matching rule, which compares similar words between the machine translation and the reference translation, and then calculates the proportion of N similar words in the machine translation. The BLEU score ranges from 0 to 1, with a higher score indicating better machine translation quality. In practical applications, the average of the BLEU scores of multiple N-grams is usually calculated to obtain a more comprehensive evaluation.

[0144] In another optional embodiment, the training method for the target detection model includes: acquiring a training sample set, which includes sample visible light remote sensing images, sample visible light remote sensing images, sample target detection box labels, and sample target semantic description labels corresponding to each sample target; inputting the sample visible light remote sensing images and sample infrared remote sensing images into the target detection model to be trained to obtain sample target detection results and sample target semantic description information; calculating the total loss based on the sample target detection results, sample target semantic description information, sample target detection box labels, and sample target semantic description labels corresponding to each sample target; and updating the model parameters of the target detection model to be trained based on the total loss.

[0145] In one specific embodiment, the total loss includes contrast loss, triplet loss, and regularization loss.

[0146] In a more specific embodiment, the total loss is calculated using the following formula:

[0147] ;

[0148] in, Indicates the total loss. Indicates comparative loss, Indicates the loss of the triplet. Indicates the regularization loss. and This is a hyperparameter.

[0149] The comparative loss is:

[0150]

[0151] Where D represents the feature distance between the enhanced feature map of the visible light remote sensing image and the enhanced feature map of the infrared remote sensing image. Indicates a positive sample pair. Indicates negative sample pairs; The boundary parameter is set to 2.0.

[0152] The loss of the triplet is:

[0153]

[0154] in, For anchor point samples, As a positive sample, Negative samples; This is the interval parameter, which can be set to 0.5. This represents the feature distance between the anchor point and the positive sample. This represents the feature distance between the anchor point and the negative sample.

[0155] The regularization loss is expressed as:

[0156]

[0157] in, and Here, L is the regularization hyperparameter, representing the number of network layers. This represents the learnable sharing coefficient. Represents specific parameters of visible light. Indicates specific infrared parameters; the first item encourages... Converging toward 0 (completely independent) or 1 (completely shared), the second term when When the difference is large, the penalty is applied to the difference in weights between the two branches;

[0158] in, To share weight parameters, These are the branch-specific weight parameters.

[0159] It should be noted that the hyperparameters can be initially set as follows: .

[0160] In a more specific embodiment, the training process is as follows:

[0161] All values ​​are initialized to 0.5. Shared layers use pre-trained VGG16 weights, while private layers are randomly initialized.

[0162] Training steps:

[0163] Step 1: Forward propagation, calculate the mixed weights based on the current calculation;

[0164] Step 2: Calculate the total loss;

[0165] Step 3: Backpropagation: Simultaneous updates;

[0166] Step 4: Use the Adam (Adaptive Moment Estimation) optimizer with a learning rate;

[0167] Step 5: Apply softmax every 10 epochs to promote convergence.

[0168] Shared coefficient update strategy:

[0169] The first 50 epochs are updated with normal gradients; from 50 to 100 epochs, the gradient is increased to 0.2 to strengthen binarization; after 100+ epochs, the gradient is fixed, and only the network weights are fine-tuned.

[0170] It should be noted that this application, based on the visible light and infrared remote sensing image registration work based on deep Siamese networks, breaks through the limitations of target detection association methods based on cross-domain knowledge transfer and adaptive learning by finding the relative position coordinates of the template image in a large reference image and utilizing end-to-end adaptive network learning. The accuracy of target detection association can be obtained by calculating the error between the test detection box and the ideal labeled coordinate box. Target detection association experiments were conducted using a multi-source remote sensing image registration and fusion dataset. Visible light and infrared remote sensing image pairs were divided into training, validation, and test sets in a 3:1:1 ratio. All images were set to grayscale and scaled to a fixed size before being input into the neural network. The results of the target association matching detection are shown below. Figure 5 As shown, Figure 5 This diagram illustrates a target association matching detection result provided in an embodiment of this application. In each matched image pair, the left image is the infrared template target, and the right image is a large-scale reference image. The solid-lined box in the reference image represents the true location of the template target, and the dashed-lined box represents the actual matching detection location of the model. This method facilitates the understanding of multi-source situations and the high-precision detection and association of cross-domain target characteristic information. It enables the transfer and utilization of high-level knowledge in cross-domain images and provides a basis for the indication, judgment, and subsequent semantic understanding and expression of multiple targets in remote sensing images.

[0171] By semantically annotating a subset of visible light samples, this application constructs a cross-modal semantic description dataset for visible light and infrared remote sensing images. The semantic descriptions of the images in this dataset are manually annotated, with the minimum Chinese description set to five characters and the maximum to 30 characters. Furthermore, each image has five semantic descriptions, written by different individuals to provide diverse interpretations of the image content.

[0172] The dataset was randomly divided into training, validation, and test sets at ratios of 5%, 5%, and 90%, respectively. The algorithm of this invention was then used to train, validate, and test the dataset. Figure 6 This is a schematic diagram of a training loss curve provided in an embodiment of this application. The horizontal axis represents the number of rounds, ranging from 0 to 40, and the vertical axis represents the loss value, ranging from 1.5 to 4.0. Figure 7 This is a schematic diagram of a verification loss curve provided in an embodiment of this application. The horizontal axis represents the number of rounds, ranging from 0 to 40, and the vertical axis represents the loss value, ranging from 1.5 to 3. Figure 8 This is a schematic diagram of a BLEU-1 score curve provided in an embodiment of this application. The vertical axis represents the number of rounds, ranging from 0 to 40, and the vertical axis represents the BLEU-1 score, ranging from 0.65 to 0.95.

[0173] To evaluate the algorithm's performance in semantic understanding and representation under cross-domain knowledge transfer from visible light and infrared remote sensing images, a set of infrared remote sensing image test samples was constructed using semantic standardization of infrared simulated images as the test sample set for cross-domain semantic understanding and representation. Based on this, a semantic description experiment was conducted on the test sample set using the network trained on the visible light simulated images from the previous experiment. Ultimately, through training on visible light remote sensing images, the semantic description score (BLEU-1) for the infrared remote sensing image test was 86.0057%. Figure 9 The image shown is an example of semantic description of infrared remote sensing images with various scenarios and target types.

[0174] It is known that the algorithm of this application embodiment has a high BLEU-1 score for single-source images on the simulation dataset, and the semantic description accuracy of cross-domain knowledge transfer in visible light remote sensing image training and infrared remote sensing image testing scenarios also exceeds the index agreed upon in this application.

[0175] The embodiments of this application can effectively realize the semantic description of targets in different scenarios, which helps to understand the overall situation and detect and describe the characteristics of targets in real time. It can provide more efficient target indication and more direct semantic prompts, and improve the level of intelligent decision-making.

[0176] This application also provides a semantic understanding method based on cross-modal image target detection. Figure 10 This is a schematic diagram of the structure of the target detection model provided in the embodiments of this application, such as... Figure 10 As shown, firstly, an infrared-visible remote sensing image is input, and the image encoding features are obtained by the encoder. Based on the image encoding features and the hidden state of the decoder, the attention weight is calculated by the attention network to obtain the attention-weighted image encoding. Then, the decoder combines the semantic encoding and the weighted image encoding, and performs decoding through a recurrent neural network. Finally, the semantic sequence is obtained through semantic sequence search to obtain the semantic sequence about the target information, and finally, word prediction is generated.

[0177] The encoder in the above architecture uses a VGG19 convolutional neural network (CNN) with fully connected and pooling layers removed as its feature extraction layer. This CNN effectively preserves the commonalities and differences between visible light and infrared remote sensing images. The encoder takes a visible light or infrared remote sensing image as input and outputs an encoded feature vector of the image after processing by the pre-trained CNN. The decoder generates a semantic description based on the image features extracted by the encoder. Its input is the encoded feature vector processed by the encoder, and its output is a text semantic sequence. First, the decoder semantically encodes each word or phrase in the input semantics, converting the semantic description into an embedding vector. Then, it uses an LSTM recurrent neural network for iterative decoding, aligning the image feature vector with the word vector, thus completing the matching of image features and semantic information, i.e., the encoding and decoding process.

[0178] In a more specific embodiment, Figure 11 This is a schematic diagram of the training process of a recurrent neural network provided in an embodiment of this application, as shown below. Figure 11 As shown, during training, the input image is first passed through the CNN. θc Convolutional neural networks extract visual features, which are then processed by fully connected layers before being input into recurrent neural network (RNN) modules. In the RNN part, the network starts with a "START" marker and progressively generates a sequence of target descriptions (e.g., "red" → "hat" → "END"). The training process employs a teacher-forcing strategy: at each time step t, the previous true word Xt and image features ht are used as input. The hidden state ht is calculated using weight matrices Whi, Whh, Woh, and Whx, ultimately predicting the output word Yt. The network optimizes the parameters of each layer by minimizing the cross-entropy loss between the predicted and true labeled words, thereby learning the ability to map image content to accurate natural language descriptions.

[0179] Corresponding to the semantic understanding method based on cross-modal image target detection provided in this application embodiment, this application embodiment also provides a semantic understanding device based on cross-modal image target detection. The target detection model for target detection includes a multi-branch convolutional neural network, a feature pyramid network, a region proposal network, a cross-modal similarity measurement network, a hierarchical convolutional encoder, and a detection head, such as... Figure 12 As shown, the semantic understanding device based on cross-modal image target detection includes:

[0180] Module 1201 acquires visible light remote sensing images and infrared remote sensing images;

[0181] The first input module 1202 is used to input visible light remote sensing images and infrared remote sensing images into the target detection model, and to extract features from the visible light remote sensing images and infrared remote sensing images through a multi-branch convolutional neural network to obtain visible light remote sensing image feature maps and infrared remote sensing image feature maps.

[0182] The fusion module 1203 is used to perform multi-scale fusion operations on the feature maps of visible light remote sensing images and infrared remote sensing images through the feature pyramid network to obtain the enhanced feature maps of visible light remote sensing images and infrared remote sensing images.

[0183] The second input module 1204 is used to input the enhanced feature map of the visible light remote sensing image and the enhanced feature map of the infrared remote sensing image into the region proposal network respectively, generate the candidate regions of the visible light remote sensing image features and the candidate regions of the infrared remote sensing image features, and extract features from the candidate regions of the visible light remote sensing image features and the candidate regions of the infrared remote sensing image features respectively to obtain the visible light candidate region features and the infrared candidate region features.

[0184] The matching module 1205 is used to match the visible light candidate region features and the infrared candidate region features through a cross-modal similarity measurement network to obtain the association matching results, which include target location information and confidence information.

[0185] The extraction module 1206 is used to extract the RoI features of the target region from the visible light remote sensing image and the infrared remote sensing image respectively based on the association matching results, so as to obtain the visible light RoI features and the infrared RoI features;

[0186] The third input module 1207 is used to input visible light RoI features and infrared RoI features into the hierarchical convolutional encoder, and extract features through the hierarchical convolutional encoder to obtain visible light semantic feature maps and infrared semantic feature maps.

[0187] The fourth input module 1208 is used to input the visible light semantic feature map and the infrared semantic feature map into the detection head to obtain the target detection result, which includes the predicted target category and the predicted target bounding box.

[0188] The generation module 1209 is used to generate a target semantic description based on the visible light semantic feature map and the infrared semantic feature map.

[0189] Corresponding to the semantic understanding method based on cross-modal image target detection provided in the embodiments of this application, the embodiments of this application also provide an electronic device for performing the semantic understanding method based on cross-modal image target detection, such as... Figure 13As shown, the electronic device includes: a processor 1301; and a memory 1302 for storing a program of a semantic understanding method based on cross-modal image target detection. After the device is powered on and the program of the semantic understanding method based on cross-modal image target detection is run by the processor, the following steps are performed:

[0190] Acquire visible light and infrared remote sensing images;

[0191] Visible light remote sensing images and infrared remote sensing images are input into the target detection model. Feature extraction is performed on the visible light remote sensing images and infrared remote sensing images through a multi-branch convolutional neural network to obtain feature maps of visible light remote sensing images and infrared remote sensing images.

[0192] The feature pyramid network is used to perform multi-scale fusion operations on the feature maps of visible light remote sensing images and infrared remote sensing images respectively to obtain the enhanced feature maps of visible light remote sensing images and infrared remote sensing images.

[0193] The enhanced feature maps of visible light remote sensing images and infrared remote sensing images are respectively input into the region proposal network to generate candidate regions for visible light remote sensing image features and candidate regions for infrared remote sensing image features. Features are then extracted from the candidate regions for visible light remote sensing image features and candidate regions for infrared remote sensing image features, respectively, to obtain the visible light candidate region features and the infrared candidate region features.

[0194] The visible light candidate region features and infrared candidate region features are matched by a cross-modal similarity measurement network to obtain the association matching results, which include target localization information and confidence information.

[0195] Based on the association matching results, the RoI features of the target region are extracted from the visible light remote sensing image and the infrared remote sensing image respectively, thus obtaining the visible light RoI features and the infrared RoI features;

[0196] Visible light RoI features and infrared RoI features are input into a hierarchical convolutional encoder, and features are extracted by the hierarchical convolutional encoder to obtain visible light semantic feature maps and infrared semantic feature maps.

[0197] The visible light semantic feature map and the infrared semantic feature map are input into the detection head to obtain the target detection result, which includes the predicted target category and the predicted target bounding box.

[0198] Based on visible light semantic feature maps and infrared semantic feature maps, a semantic description of the target is generated.

[0199] Corresponding to the semantic understanding method based on cross-modal image target detection provided in the embodiments of this application, the embodiments of this application also provide a computer-readable storage medium storing a program for the semantic understanding method based on cross-modal image target detection. This program is executed by a processor to perform the following steps:

[0200] Acquire visible light and infrared remote sensing images;

[0201] Visible light remote sensing images and infrared remote sensing images are input into the target detection model. Feature extraction is performed on the visible light remote sensing images and infrared remote sensing images through a multi-branch convolutional neural network to obtain feature maps of visible light remote sensing images and infrared remote sensing images.

[0202] The feature pyramid network is used to perform multi-scale fusion operations on the feature maps of visible light remote sensing images and infrared remote sensing images respectively to obtain the enhanced feature maps of visible light remote sensing images and infrared remote sensing images.

[0203] The enhanced feature maps of visible light remote sensing images and infrared remote sensing images are respectively input into the region proposal network to generate candidate regions for visible light remote sensing image features and candidate regions for infrared remote sensing image features. Features are then extracted from the candidate regions for visible light remote sensing image features and candidate regions for infrared remote sensing image features, respectively, to obtain the visible light candidate region features and the infrared candidate region features.

[0204] The visible light candidate region features and infrared candidate region features are matched by a cross-modal similarity measurement network to obtain the association matching results, which include target localization information and confidence information.

[0205] Based on the association matching results, the RoI features of the target region are extracted from the visible light remote sensing image and the infrared remote sensing image respectively, thus obtaining the visible light RoI features and the infrared RoI features;

[0206] Visible light RoI features and infrared RoI features are input into a hierarchical convolutional encoder, and features are extracted by the hierarchical convolutional encoder to obtain visible light semantic feature maps and infrared semantic feature maps.

[0207] The visible light semantic feature map and the infrared semantic feature map are input into the detection head to obtain the target detection result, which includes the predicted target category and the predicted target bounding box.

[0208] Based on visible light semantic feature maps and infrared semantic feature maps, a semantic description of the target is generated.

[0209] Corresponding to the semantic understanding method based on cross-modal image target detection provided in the embodiments of this application, the embodiments of this application also provide a computer program containing instructions that, when executed by a computer, cause the computer to perform the following steps:

[0210] Acquire visible light and infrared remote sensing images;

[0211] Visible light remote sensing images and infrared remote sensing images are input into the target detection model. Feature extraction is performed on the visible light remote sensing images and infrared remote sensing images through a multi-branch convolutional neural network to obtain feature maps of visible light remote sensing images and infrared remote sensing images.

[0212] The feature pyramid network is used to perform multi-scale fusion operations on the feature maps of visible light remote sensing images and infrared remote sensing images respectively to obtain the enhanced feature maps of visible light remote sensing images and infrared remote sensing images.

[0213] The enhanced feature maps of visible light remote sensing images and infrared remote sensing images are respectively input into the region proposal network to generate candidate regions for visible light remote sensing image features and candidate regions for infrared remote sensing image features. Features are then extracted from the candidate regions for visible light remote sensing image features and candidate regions for infrared remote sensing image features, respectively, to obtain the visible light candidate region features and the infrared candidate region features.

[0214] The visible light candidate region features and infrared candidate region features are matched by a cross-modal similarity measurement network to obtain the association matching results, which include target localization information and confidence information.

[0215] Based on the association matching results, the RoI features of the target region are extracted from the visible light remote sensing image and the infrared remote sensing image respectively, thus obtaining the visible light RoI features and the infrared RoI features;

[0216] Visible light RoI features and infrared RoI features are input into a hierarchical convolutional encoder, and features are extracted by the hierarchical convolutional encoder to obtain visible light semantic feature maps and infrared semantic feature maps.

[0217] The visible light semantic feature map and the infrared semantic feature map are input into the detection head to obtain the target detection result, which includes the predicted target category and the predicted target bounding box.

[0218] Based on visible light semantic feature maps and infrared semantic feature maps, a semantic description of the target is generated.

[0219] It should be noted that for a detailed description of the semantic understanding device, electronic device, computer-readable storage medium and computer program product based on cross-modal image target detection provided in the embodiments of this application, please refer to the relevant description of the semantic understanding method embodiments based on cross-modal image target detection provided in the embodiments of this application, which will not be repeated here.

[0220] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0221] In a typical configuration, an electronic device includes one or more processors (Central Processing Units), input / output interfaces, network interfaces, and memory.

[0222] Memory may include non-persistent storage in computer-readable media, such as random access memory and / or non-volatile memory, like read-only memory or flash memory. Memory is an example of computer-readable media.

[0223] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable operations, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, compact disc read-only memory, digital video disc or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0224] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, compact disc read-only memory, optical storage, etc.) containing computer-usable program code.

[0225] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A semantic understanding method based on cross-modal image target detection, characterized in that, The target detection model for target detection includes a multi-branch convolutional neural network, a feature pyramid network, a region proposal network, a cross-modal similarity measurement network, a hierarchical convolutional encoder, a detection head, a two-layer LSTM decoder, fully connected layers, and a residual encoder. The multi-branch convolutional neural network includes shared convolutional layers, visible light-specific convolutional layers, and infrared-specific convolutional layers. The cross-modal similarity measurement network includes two weight-sharing branches, each branch comprising multiple convolutional layers and normalization layers, including: Acquire visible light and infrared remote sensing images; The visible light remote sensing image and the infrared remote sensing image are input into the target detection model, and the visible light remote sensing image and the infrared remote sensing image are extracted by the multi-branch convolutional neural network to obtain the visible light remote sensing image feature map and the infrared remote sensing image feature map. The feature pyramid network is used to perform multi-scale fusion operations on the visible light remote sensing image feature map and the infrared remote sensing image feature map to obtain the visible light remote sensing image enhancement feature map and the infrared remote sensing image enhancement feature map. The enhanced feature maps of the visible light remote sensing image and the enhanced feature maps of the infrared remote sensing image are respectively input into the region proposal network to generate candidate regions of visible light remote sensing image features and candidate regions of infrared remote sensing image features. Features are extracted from the candidate regions of visible light remote sensing image features and candidate regions of infrared remote sensing image features respectively to obtain visible light candidate region features and infrared candidate region features. The visible light candidate region features and infrared candidate region features are matched by the cross-modal similarity measurement network to obtain the association matching results, which include target location information and confidence information. Based on the association matching results, the RoI features of the target region are extracted from the visible light remote sensing image and the infrared remote sensing image respectively to obtain the visible light RoI features and the infrared RoI features; The visible light RoI features and infrared RoI features are residual encoded by the residual encoder to obtain the visible light RoI enhanced features and infrared RoI enhanced features. The visible light RoI enhancement features and infrared RoI enhancement features are input into the hierarchical convolutional encoder, and the features are extracted by the hierarchical convolutional encoder to obtain the visible light semantic feature map and the infrared semantic feature map. The visible light semantic feature map and the infrared semantic feature map are input into the detection head to obtain the target detection result, which includes the predicted target category and the predicted target bounding box. Based on the hidden state, visible light semantic feature map, and infrared semantic feature map of the dual-layer LSTM decoder at the previous time step, attention weights are calculated, and the visible light semantic feature map and infrared semantic feature map are weighted based on the attention weights to obtain a weighted feature vector. The word embedding vector and weighted feature vector predicted in the previous time step are used as the input of the two-layer LSTM decoder, and the current hidden state is output. The current hidden state and the weighted feature vector are input into the fully connected layer to generate a target semantic description.

2. The semantic understanding method based on cross-modal image target detection according to claim 1, characterized in that, The attention weights include visible light spatial attention weights and infrared spatial attention weights. The calculation of these attention weights based on the hidden state, visible light semantic feature map, and infrared semantic feature map of the dual-layer LSTM decoder at the previous time step includes: Perform global average pooling on the visible light semantic feature map to obtain the average feature of visible light spatial description; Perform global max pooling on the visible light semantic feature map to obtain the maximum feature describing the visible light space; Global average pooling is performed on the infrared semantic feature map to obtain the infrared spatial description average feature; Perform global max pooling on the infrared semantic feature map to obtain the maximum feature for infrared spatial description; The average feature and the maximum feature of visible light spatial description are convolved to obtain the average enhanced feature and the maximum enhanced feature of visible light spatial description, respectively. The average enhancement feature and the maximum enhancement feature of the visible light spatial description are fused to obtain the initial weights for visible light spatial attention. The average feature and the maximum feature of the infrared spatial description are convolved respectively to obtain the average enhanced feature and the maximum enhanced feature of the infrared spatial description. The average enhancement feature and the maximum enhancement feature of the infrared spatial description are fused to obtain the initial weights for infrared spatial attention. The visible light spatial attention initial weights and infrared spatial attention initial weights are adjusted based on the hidden state of the previous time step using a dual-layer LSTM decoder, to obtain the visible light spatial attention weights and infrared spatial attention weights.

3. The semantic understanding method based on cross-modal image target detection according to claim 1, characterized in that, The attention weights include visible light channel attention weights and infrared channel attention weights. The calculation of these attention weights based on the hidden state, visible light semantic feature map, and infrared semantic feature map of the dual-layer LSTM decoder at the previous time step includes: Perform global average pooling on the visible light semantic feature map to obtain the average features describing the visible light channels; Perform global max pooling on the visible light semantic feature map to obtain the maximum feature describing the visible light channel; Global average pooling is performed on the infrared semantic feature map to obtain the average feature describing the infrared channels; Perform global max pooling on the infrared semantic feature map to obtain the maximum feature describing the infrared channel; The average feature and the maximum feature of the visible light channel description are mapped to obtain the average enhancement feature and the maximum enhancement feature of the visible light channel description, respectively. The average enhancement feature and the maximum enhancement feature of the visible light channel description are fused to obtain the initial weights for the visible light channel attention. The average feature and the maximum feature of the infrared channel description are convolved respectively to obtain the average enhanced feature and the maximum enhanced feature of the infrared channel description. The average enhancement feature and the maximum enhancement feature of the infrared channel description are fused to obtain the initial weights for infrared channel attention. The initial attention weights of the visible light channel and the infrared channel are adjusted based on the hidden state of the dual-layer LSTM decoder at the previous time step to obtain the attention weights of the visible light channel and the infrared channel.

4. The semantic understanding method based on cross-modal image target detection according to claim 2 or 3, characterized in that, The attention weights include inter-modal attention weights for visible light and inter-modal attention weights for infrared light. The calculation of these attention weights based on the hidden state, visible light semantic feature map, and infrared semantic feature map of the dual-layer LSTM decoder at the previous time step includes: Perform global average pooling on the visible light semantic feature map to obtain the overall average feature of visible light description; Perform global average pooling on the infrared semantic feature map to obtain the average feature of the overall infrared description; The initial weights for attention between visible light modalities are determined based on the average features of the overall visible light description. The initial weights for intermodal attention in infrared are determined based on the average features of the overall infrared description. The initial attention weights between visible light modes and between infrared modes are adjusted based on the hidden state of the dual-layer LSTM decoder in the previous time step to obtain the attention weights between visible light modes and between infrared modes.

5. The semantic understanding method based on cross-modal image target detection according to claim 1, characterized in that, The training method for the target detection model includes: Obtain a training sample set, which includes sample visible light remote sensing images, sample infrared remote sensing images, sample target detection box labels, and sample target semantic description labels corresponding to each sample target; The sample visible light remote sensing image and sample infrared remote sensing image are input into the target detection model to be trained to obtain the sample target detection results and sample target semantic description information; The total loss is calculated based on the sample target detection results, sample target semantic description information, sample target detection box labels, and sample target semantic description labels corresponding to each sample target. The model parameters of the target detection model to be trained are updated based on the total loss.

6. The semantic understanding method based on cross-modal image target detection according to claim 5, characterized in that, The total loss includes: contrast loss, triplet loss, and regularization loss.

Images