Open-domain target recognition method and system based on multimodal large model

Abstract

This invention provides an open-domain target recognition method and system based on a multimodal large model, relating to the fields of computer vision and multimodal fusion technology. First, visible light and infrared sensor data streams and an open-domain small-sample training set are acquired to construct visible light and infrared scene context semantic graphs. Cross-modal scene graph collaborative discovery processing generates a cross-modal scene semantic association graph. A multi-detection algorithm collaborative ensemble network is constructed, using the cross-modal scene semantic association graph as contextual prior knowledge input to guide multi-branch collaborative detection. The pseudo-label generation network is trained with small-sample pseudo-labels using the open-domain small-sample training set, generating pseudo-labels and pseudo-boundary boxes for unlabeled images. These are combined with the original set to form an expanded training set. The multi-detection algorithm collaborative ensemble network is then jointly trained across modal multiple branches to generate an open-domain target detection model and output category and location information. This invention effectively improves the accuracy and generalization ability of open-domain target recognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision and multimodal fusion technology, and more specifically, to an open-domain target recognition method and system based on a large multimodal model. Background Technology

[0002] Open-domain target recognition is one of the core research directions in computer vision. Its goal is to classify and locate any target object appearing in a scene without being limited by a pre-defined set of categories. In practical applications, visible light imaging and infrared thermal imaging are two of the most commonly used sensing methods. Visible light images can provide rich texture and color information, while infrared images can capture the thermal radiation characteristics of targets under low light or occlusion conditions. The two have natural complementary advantages.

[0003] However, existing technologies face numerous challenges in open-domain target recognition scenarios. Regarding multimodal fusion, current solutions typically employ simple feature concatenation or attention-weighted methods to merge visible light and infrared features. These approaches only achieve shallow fusion at the feature level, failing to uncover the deep semantic relationships between the two modalities. This results in ineffective coordination of semantic information from different modalities in complex open scenes, severely limiting the accuracy and robustness of target recognition. In terms of few-sample training, most existing technologies rely on large-scale fully supervised labeled data for model training. When faced with a large number of unseen categories in the open domain, the lack of sufficient labeled samples drastically reduces the model's generalization ability. While some existing technologies attempt to use pseudo-labeling techniques for semi-supervised learning on unlabeled data, the pseudo-label generation process is usually independent of the detection network's training process. The quality of the generated pseudo-labels is difficult to guarantee, and the multimodal scene context information is not fully utilized to guide pseudo-label generation, leading to significant discrepancies between the pseudo-label results and the actual targets. Furthermore, existing detection networks typically employ a single-branch structure or a simple multi-branch parallel structure, lacking an effective coordination mechanism between branches. This prevents them from dynamically adjusting detection strategies based on contextual prior knowledge of different modal scenarios, further limiting the overall performance of open-domain target recognition. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide an open-domain target recognition method based on a multimodal large model, the method comprising:

[0005] Acquire visible light sensing data stream and infrared sensing data stream of the target scene, and acquire an open domain few-shot training set, which includes visible light few-shot training images, infrared few-shot training images, and real bounding box annotation data and real category label data corresponding to the visible light few-shot training images and the infrared few-shot training images.

[0006] A visible light scene context semantic graph is constructed using the visible light sensing data stream, and an infrared scene context semantic graph is constructed using the infrared sensing data stream. Cross-modal scene graph collaborative discovery processing is performed on the visible light scene context semantic graph and the infrared scene context semantic graph to generate a cross-modal scene semantic association graph.

[0007] A multi-detection algorithm collaborative integration network is constructed, which includes a first visible light detection branch, a second infrared detection branch, and a third cross-modal fusion detection branch. The cross-modal scene semantic association graph is used as the contextual prior knowledge input of the third cross-modal fusion detection branch. The scene context information in the cross-modal scene semantic association graph guides the multi-detection algorithm collaborative integration network to perform multi-branch collaborative target detection processing.

[0008] The pre-constructed pseudo-label generation network is trained on the open domain few-sample training set to generate a trained pseudo-label generation network. The trained pseudo-label generation network is then called to generate pseudo-labels and pseudo-boundary boxes on unlabeled open scene images, resulting in a pseudo-category label set and a pseudo-boundary box label set.

[0009] The open-domain small sample training set, the unlabeled open scene image, the pseudo-class label set, and the pseudo-boundary box label set are combined into an expanded training set. The expanded training set is used to perform cross-modal multi-branch joint training on the multi-detection algorithm collaborative ensemble network to generate a trained open-domain target detection model. The open-domain target detection model is then used to perform open-domain target recognition processing on the target scene, outputting the open-domain category information and open-domain bounding box localization information of the target object.

[0010] Furthermore, embodiments of the present invention also provide an open-domain target recognition system based on a multimodal large model, characterized in that it includes:

[0011] A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to execute the above-described open-domain target recognition method based on a multimodal large model by executing the machine-executable instructions.

[0012] Based on the above, by constructing visible light scene context semantic graphs and infrared scene context semantic graphs and performing cross-modal scene graph collaborative discovery processing to generate cross-modal scene semantic association graphs, a deep association topology between visible light and infrared modalities can be established at the scene semantic level. This ensures that the contextual information of the two modalities no longer exists in isolation but forms a complementary and enhanced unified semantic expression. Furthermore, by constructing a multi-detection algorithm collaborative integration network comprising a first visible light detection branch, a second infrared detection branch, and a third cross-modal fusion detection branch, and using the cross-modal scene semantic association graph as the contextual prior knowledge input for the third cross-modal fusion detection branch, collaborative work of the three detection branches under the guidance of unified scene semantics is achieved. This allows each branch to dynamically adjust its detection strategy based on cross-modal association information, significantly improving target detection accuracy in complex open scenes. Simultaneously, by utilizing training with small visible light samples… The open-domain small-sample training set of training images, infrared small-sample training images, and their real bounding box annotation data and real class label data is used to train the pseudo-label generation network to improve its pseudo-labeling ability. This enables the pseudo-label generation network to learn the essential feature distribution of cross-modal targets under conditions of very few labeled samples, thereby generating high-quality pseudo-class label sets and pseudo-bounding box annotation sets for unlabeled open scene images, effectively expanding the scale of training data. Finally, the open-domain small-sample training set, unlabeled open scene images, pseudo-class label sets, and pseudo-bounding box annotation sets are combined into an expanded training set for cross-modal multi-branch joint training of a multi-detection algorithm collaborative ensemble network. The resulting open-domain target detection model has both strong generalization ability under small-sample conditions and multi-modal collaborative detection ability, thus significantly improving the overall performance of open-domain target recognition in terms of category coverage breadth and localization accuracy. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the execution flow of the open domain target recognition method based on a multimodal large model provided in an embodiment of the present invention.

[0014] Figure 2 This is a logical schematic diagram of the open domain target recognition method based on a multimodal large model provided in an embodiment of the present invention.

[0015] Figure 3 This is a schematic diagram of exemplary hardware and software components of an open-domain target recognition system based on a multimodal large model provided in an embodiment of the present invention. Detailed Implementation

[0016] Figure 1 This is a flowchart illustrating an open-domain target recognition method based on a multimodal large model according to an embodiment of the present invention. The following is a detailed explanation. Figure 2 This paper provides a detailed introduction to the open-domain target recognition method based on a multimodal large model.

[0017] Step S110: Obtain the visible light sensing data stream and infrared sensing data stream of the target scene, and obtain the open domain few-shot training set. The open domain few-shot training set includes visible light few-shot training images, infrared few-shot training images, and the real bounding box annotation data and real category label data corresponding to the visible light few-shot training images and the infrared few-shot training images.

[0018] In this embodiment, in an unattended border monitoring scenario, an optoelectronic turntable deployed on an observation tower continuously acquires visible light and infrared sensor data streams of the target scene. The visible light sensor data stream is a sequence of color digital images with a resolution of 1920 x 1080 pixels and a frame rate of 30 frames per second, while the infrared sensor data stream is a sequence of thermal imaging grayscale digital images with the same resolution and frame rate. Simultaneously, an open-domain small-sample training set is obtained from a publicly available military target small-sample dataset. This open-domain small-sample training set contains 200 visible light small-sample training images and 200 infrared small-sample training images, with each visible light small-sample training image corresponding to one infrared small-sample training image to form an image pair. Each image is accompanied by corresponding ground truth bounding box annotation data and ground truth class label data. The ground truth bounding box annotation data is stored as a quadruple of the top-left x-coordinate, top-left y-coordinate, bounding box width, and bounding box height. The ground truth class label data is stored as integer identifiers, such as a value of 1 representing an armored vehicle and a value of 2 representing a fighter jet. All data is encrypted during collection and transmission, and its use has been authorized and complies with data security and privacy protection regulations.

[0019] Step S120: Construct a visible light scene context semantic graph using the visible light sensing data stream, construct an infrared scene context semantic graph using the infrared sensing data stream, perform cross-modal scene graph collaborative discovery processing on the visible light scene context semantic graph and the infrared scene context semantic graph, and generate a cross-modal scene semantic association graph.

[0020] Step S121: Input the visible light sensing data stream into the visible light target background separation network, perform response activation processing on the foreground target region in the visible light sensing data stream through the target region activation module to generate a visible light target region mask map, and perform region segmentation processing on the background region in the visible light sensing data stream through the background region extraction module to generate a visible light background region mask map.

[0021] In this embodiment, each frame of visible light image from the visible light sensing data stream obtained in step S110 is input into a pre-trained visible light target background separation network. This visible light target background separation network adopts a U-Net architecture. Its encoder consists of five convolutional layer groups, each containing two 3x3 convolutional kernels and one 2x2 max-pooling layer, used to progressively extract high-dimensional semantic features and reduce the spatial resolution of the feature map. The decoder consists of five upsampling layer groups, each containing a 2x2 deconvolutional layer and two 3x3 convolutional kernels, used to progressively restore the spatial resolution of the feature map and generate pixel-by-pixel classification results. The target region activation module of this visible light target background separation network is essentially the binary classification feature map output from the last layer of the decoder, where the value of each pixel, after activation by the Sigmoid function, represents the probability that the pixel belongs to the foreground target region. The foreground activation threshold is set to 0.5, pixels with a probability value greater than 0.5 are marked as 1, and the rest are marked as 0, generating a visible light target region mask map. Simultaneously, the background region extraction module of this visible light target-background separation network performs a logical inversion operation on the foreground target region mask to obtain a visible light background region mask. Pixels with a value of 1 in this visible light background region mask represent background regions.

[0022] Step S122: Extract the visible light target visual feature vector corresponding to each visible light target region from the visible light sensing data stream according to the visible light target region mask, and extract the visible light background visual feature vector corresponding to each visible light background region from the visible light sensing data stream according to the visible light background region mask.

[0023] In this embodiment, connected component analysis is performed on the visible light target region mask. A depth-first search algorithm is used to traverse all pixels in the mask, grouping adjacent pixels marked as 1 into the same connected component. Each connected component corresponds to a visible light target region. For each detected visible light target region, its minimum bounding rectangle is calculated, and the image patch corresponding to this bounding rectangle is cropped from the original visible light image. This image patch is scaled to a fixed size of 224 by 224 pixels and then input into a pre-trained ResNet50 convolutional neural network. This ResNet50 network contains 5 residual modules, each consisting of multiple convolutional layers and skip connections. The 2048-dimensional feature vector output from the global average pooling layer of this network is taken as the visible light target visual feature vector of the visible light target region. For the visible light background region mask, connected component analysis is also performed, grouping adjacent pixels marked as 1 into the same connected component. Each connected component corresponds to a visible light background region. For small background regions with an area less than 5% of the total frame area, they are merged with the adjacent largest background region to ensure the semantic integrity of the background region. For each visible light background region, its center point coordinates are extracted, and a 224x224 pixel image block is cropped with this center point as the center. This block is then input into the ResNet50 network described above, and a 2048-dimensional feature vector is extracted as the visible light background visual feature vector for that visible light background region.

[0024] Step S123: Define each visible light target region as a visible light target node and each visible light background region as a visible light background node. The visible light target node stores the corresponding visible light target visual feature vector, and the visible light background node stores the corresponding visible light background visual feature vector.

[0025] In this embodiment, an empty graph data structure is created to construct a visible light scene context semantic graph. For each visible light target region extracted in step S122, a visible light target node object is created in the graph structure. This visible light target node object contains the following attribute fields: node type identifier set to target type, unique node number, set of spatial coordinates of the target region, and the visible light target visual feature vector extracted in step S122. For each visible light background region extracted in step S122, a visible light background node object is created in the graph structure. This visible light background node object contains the following attribute fields: node type identifier set to background type, unique node number, set of spatial coordinates of the background region, and the visible light background visual feature vector extracted in step S122.

[0026] Step S124: Perform co-occurrence relationship statistical processing on any two visible light map nodes, and calculate the proportion of frames in which the two visible light map nodes appear simultaneously in the continuous frame sequence of the visible light sensing data stream as the co-occurrence frequency statistical parameter. Calculate the spatial distance offset and relative direction angle between the corresponding regions of the two visible light map nodes as the spatial position association parameter.

[0027] In this embodiment, for any two visible light graph nodes in the graph structure constructed in step S123, denoted as node A and node B, their co-occurrence relationship in the time dimension is statistically analyzed. A sequence of N consecutive frames of images from the visible light sensing data stream is taken, where N is 300. For each frame, it is detected whether the target region or background region corresponding to node A appears in that frame, and simultaneously, it is detected whether the region corresponding to node B appears in that frame. The number of frames in which node A and node B appear simultaneously is counted, denoted as M. The formula for calculating the co-occurrence frequency statistical parameter is Cooccur = M / N. Simultaneously, for each frame in which node A and node B appear simultaneously, the Euclidean distance between the center point coordinates of the region corresponding to node A and the center point coordinates of the region corresponding to node B is calculated to obtain the spatial distance value. The spatial distance values ​​of all co-occurring frames are averaged to obtain the average spatial distance offset Dist. The angle between the vector pointing from the center point of the region corresponding to node A to the center point of the region corresponding to node B and the positive horizontal direction is calculated as the relative direction angle. The relative direction angles of all co-occurring frames are then circularly averaged to obtain the average relative direction angle Angle.

[0028] Step S125: Normalize the co-occurrence frequency statistics, spatial distance offset, and relative direction angle to obtain normalized co-occurrence frequency statistics, spatial distance offset, and relative direction angle; connect visible light map nodes whose normalized co-occurrence frequency statistics satisfy the preset co-occurrence frequency threshold using co-occurrence relationship connection edges, and store the normalized co-occurrence frequency statistics, spatial distance offset, and relative direction angle in the corresponding co-occurrence relationship connection edges to generate the visible light scene context semantic graph.

[0029] In this embodiment, the set of co-occurrence frequency statistics parameters between all node pairs calculated in step S124 is subjected to Min-Max normalization, linearly mapping the parameter values ​​to the interval between 0 and 1. Similarly, the spatial distance offset between all node pairs is subjected to Min-Max normalization, and all relative direction angles are normalized by dividing by 360 degrees to obtain the angle ratio value between 0 and 1. The co-occurrence frequency threshold Th_co is set to 0.3. For any two visible light map nodes, if the normalized co-occurrence frequency statistics parameter between them is greater than Th_co, a co-occurrence relationship connection edge is established between these two nodes. This co-occurrence relationship connection edge is stored as an undirected edge, and the normalized co-occurrence frequency statistics parameter value, the normalized spatial distance offset, and the normalized relative direction angle value are recorded in the edge's attributes. All visible light target nodes, visible light background nodes, and the co-occurrence relationship connection edges between nodes are combined to form the visible light scene context semantic graph G_vis.

[0030] Step S126: Construct an infrared scene context semantic graph based on the infrared sensing data stream, and perform cross-modal semantic association processing on the visible light scene context semantic graph and the infrared scene context semantic graph.

[0031] In this embodiment, the operation process is completely symmetrical to steps S121 to S125 for the infrared sensing data stream. Specifically, the infrared sensing data stream is input into a pre-trained infrared target background separation network. This infrared target background separation network has the same U-Net architecture as the visible light target background separation network, but its weight parameters are independent. An infrared target region mask is generated by the target region activation module of this infrared target background separation network, and an infrared background region mask is generated by the background region extraction module. The infrared target thermal radiation feature vector of each infrared target region is extracted based on the infrared target region mask, and the infrared background thermal radiation feature vector of each infrared background region is extracted based on the infrared background region mask. Each infrared target region is defined as an infrared target node, and each infrared background region is defined as an infrared background node. The infrared target node stores the corresponding infrared target thermal radiation feature vector, and the infrared background node stores the corresponding infrared background thermal radiation feature vector. Temperature correlation processing is performed on any two infrared image nodes. The temperature difference parameter and temperature gradient direction parameter between the corresponding regions of the two infrared image nodes are calculated. After normalization, infrared image nodes that meet the temperature difference threshold condition are connected using temperature gradient relationship connection edges to generate an infrared scene context semantic graph G_ir. Then, cross-modal semantic correlation processing is performed on the visible light scene context semantic graph G_vis and the infrared scene context semantic graph G_ir.

[0032] Step S127: Input the infrared sensing data stream into the infrared target background separation network, perform response activation processing on the foreground target region in the infrared sensing data stream through the target region activation module to generate an infrared target region mask, and perform region segmentation processing on the background region in the infrared sensing data stream through the background region extraction module to generate an infrared background region mask.

[0033] In this embodiment, each frame of infrared image from the infrared sensing data stream acquired in step S110 is input into a pre-trained infrared target background separation network. This infrared target background separation network uses the same U-Net architecture as the visible light target background separation network, but because infrared images are single-channel grayscale images, the number of input layer channels is set to 1. During training, this infrared target background separation network uses a large number of infrared military target images, enabling it to effectively distinguish between high-temperature target foregrounds and low-temperature backgrounds in infrared images. The binary classification feature map output by the target region activation module of this infrared target background separation network is activated by Sigmoid, and the foreground activation threshold is set to 0.5. Pixels with a probability value greater than 0.5 are marked as 1, and the rest are marked as 0, generating an infrared target region mask. This infrared target region mask is then logically inverted to obtain the infrared background region mask.

[0034] Step S128: Extract the infrared target appearance feature vector corresponding to each infrared target region from the infrared sensing data stream according to the infrared target region mask, and extract the infrared background appearance feature vector corresponding to each infrared background region from the infrared sensing data stream according to the infrared background region mask.

[0035] In this embodiment, connected component analysis is performed on the infrared target region mask image. The same depth-first search algorithm as in step S122 is used to group adjacent pixels marked as 1 into the same connected component, with each connected component corresponding to an infrared target region. For each infrared target region, its minimum bounding rectangle is calculated, and the image patch corresponding to this bounding rectangle is cropped from the original infrared image. This image patch is scaled to a fixed size of 224×224 pixels and input into a pre-trained ResNet50 convolutional neural network. Unlike the visible light image processing in step S122, the number of input channels of the first convolutional layer of this ResNet50 network is modified to 1 to adapt to single-channel infrared images. The 2048-dimensional feature vector output from the global average pooling layer of this network is taken as the infrared target appearance feature vector of the infrared target region. This feature vector encodes high-level appearance information such as brightness distribution patterns and texture contours formed by differences in target and background temperatures and material emissivity in the infrared image, rather than physical absolute temperature or emissivity parameters. For the infrared background region, the same method is used to extract the apparent feature vector of the infrared background.

[0036] Step S129: Define each infrared target region as an infrared target node and each infrared background region as an infrared background node. The infrared target node stores the corresponding infrared target thermal radiation feature vector, and the infrared background node stores the corresponding infrared background thermal radiation feature vector. Calculate the temperature difference parameter and temperature gradient direction parameter between any two infrared map nodes. Normalize the temperature difference parameter and temperature gradient direction parameter to obtain normalized temperature difference parameters and temperature gradient direction parameters. Connect the infrared map nodes whose normalized temperature difference parameters meet the preset temperature difference threshold condition using temperature gradient relationship connection edges, and store the normalized temperature difference parameter and temperature gradient direction parameter in the corresponding temperature gradient relationship connection edges to generate the infrared scene context semantic graph.

[0037] In this embodiment, an empty graph data structure is created to construct the infrared scene context semantic graph. For each infrared target region extracted in step S128, an infrared target node object is created in the graph structure, which stores the corresponding infrared target thermal radiation feature vector. For each infrared background region, an infrared background node object is created, storing the corresponding infrared background thermal radiation feature vector. For any two infrared graph nodes, the absolute value of the difference between the average temperature values ​​of their corresponding regions is calculated as the temperature difference parameter DeltaT. The vector pointing from the center of one region to the center of another region is calculated, and the angle between this vector and the principal direction of the temperature gradient is used as the temperature gradient direction parameter GradAng. The DeltaT of all node pairs is normalized using Min-Max, and the GradAng of all node pairs is normalized by dividing by 360 degrees. The temperature difference threshold Th_temp is set to 0.2. For any two infrared map nodes, if the normalized DeltaT is less than Th_temp, meaning the temperatures of the two regions are similar, a temperature gradient relationship edge is established between them, and the normalized DeltaT and GradAng values ​​are stored in the edge's attributes. All infrared target nodes, infrared background nodes, and the temperature gradient relationship edges between nodes are combined to form the infrared scene context semantic graph G_ir.

[0038] Step S1210: For each visible light target node in the visible light scene context semantic map, retrieve the infrared map node with a matching spatial location in the infrared scene context semantic map, and calculate the cross-modal feature similarity metric between the visible light target visual feature vector of the visible light target node and the infrared target thermal radiation feature vector of the infrared map node with a matching spatial location through the cross-modal feature similarity calculation module.

[0039] In this embodiment, for each visible light target node in the visible light scene context semantic graph G_vis, its spatial coordinates are extracted, such as the coordinates of the center point of the target area corresponding to that node. In the infrared scene context semantic graph G_ir, all infrared target nodes are traversed, and the Euclidean distance between the center point coordinates of each infrared target node and the center point coordinates of the visible light target node is calculated. The spatial matching distance threshold Th_dist is set to 50 pixels. Infrared target nodes with an Euclidean distance less than Th_dist are considered as candidate infrared map nodes with matching spatial positions. If there are multiple candidates, the one with the smallest Euclidean distance is selected as the matched infrared map node. Then, the cross-modal feature similarity calculation module is called, which adopts a cosine similarity-based calculation method. The 2048-dimensional visible light target visual feature vector of the visible light target node is denoted as V_vis, and the 2048-dimensional infrared target thermal radiation feature vector of the matched infrared target node is denoted as V_ir. First, L2 norm normalization is performed on the two feature vectors respectively, that is, each feature vector is divided by its own modulus to obtain a unit vector. The formula for calculating cosine similarity is Similarity=(V_vis·V_ir) / (||V_vis||*||V_ir||). Since L2 normalization has been performed, the denominator is 1, so the similarity is directly equal to the dot product of the two normalized vectors. The value of this similarity ranges from -1 to 1, and the closer the value is to 1, the more similar the features of the two modalities are.

[0040] Step S1211: When the cross-modal feature similarity metric meets the preset cross-modal corresponding threshold condition, establish a cross-modal semantic correspondence connection edge between the visible light target node and the corresponding infrared map node, store the cross-modal feature similarity metric in the cross-modal semantic correspondence connection edge, and combine the visible light scene context semantic map, the infrared scene context semantic map, and all cross-modal semantic correspondence connection edges into the cross-modal scene semantic association map.

[0041] In this embodiment, the cross-modal correspondence threshold Th_cross is set to 0.6. For the cross-modal feature similarity metric Similarity calculated in step S1210, if Similarity is greater than Th_cross, it is determined that the visible light target node and the infrared target node correspond to the same physical entity. A cross-modal semantic correspondence connection edge is established between the visible light target node and the corresponding infrared target node. This cross-modal semantic correspondence connection edge is an undirected edge, and the calculated Similarity value is stored in the edge's attributes. Repeat the above steps. After processing all visible light target nodes, the visible light scene context semantic graph G_vis, the infrared scene context semantic graph G_ir, and all established cross-modal semantic correspondence connection edges are merged to generate a cross-modal scene semantic association graph G_cross. This G_cross is a heterogeneous graph containing two types of nodes, namely visible light nodes and infrared nodes, and three types of edges, namely co-occurrence relationship connection edges, temperature gradient relationship connection edges, and cross-modal semantic correspondence connection edges.

[0042] Step S130: Construct a multi-detection algorithm collaborative integration network that includes a first visible light detection branch, a second infrared detection branch, and a third cross-modal fusion detection branch. Use the cross-modal scene semantic association graph as the contextual prior knowledge input of the third cross-modal fusion detection branch. Guide the multi-detection algorithm collaborative integration network to perform multi-branch collaborative target detection processing through the scene context information in the cross-modal scene semantic association graph.

[0043] Step S131: Construct the first visible light detection branch of the multi-detection algorithm collaborative integration network. The first visible light detection branch includes a visible light feature pyramid extraction network and a visible light target detection head network. The visible light feature pyramid extraction network performs multi-layer feature mapping processing on the visible light sensing data stream to generate a set of visible light multi-scale feature maps. The visible light target detection head network performs candidate region generation and category prediction processing on the set of visible light multi-scale feature maps.

[0044] In this embodiment, a first visible light detection branch is constructed within a multi-detection algorithm collaborative integration network. The visible light feature pyramid extraction network of this first visible light detection branch adopts a ResNet50-FPN architecture. ResNet50 serves as the backbone network, comprising five stages that output feature maps with different downsampling ratios relative to the original image: 4, 8, 16, and 32, respectively. The FPN (Feature Pyramid Network) fuses these multi-scale features from top to bottom, generating feature maps of five scales, denoted as P3, P4, P5, P6, and P7, each with 256 channels, through upsampling and lateral connections. The visible light target detection head network adopts a RetinaNet detection head structure, comprising two sub-networks: one for classification and one for regression. The classification sub-network consists of four 3x3 convolutional layers, each followed by a ReLU activation function, and finally outputs the class probability distribution of each anchor box through a 3x3 convolutional layer. The regression sub-network has a similar structure, outputting the bounding box offset of each anchor box relative to a preset anchor box. The first visible light detection branch processes the input visible light image to generate a set of visible light multi-scale feature maps and multi-scale candidate target regions and their category predictions.

[0045] Step S132: Construct the second infrared detection branch of the multi-detection algorithm collaborative integration network. The second infrared detection branch includes an infrared feature pyramid extraction network and an infrared target detection head network. The infrared feature pyramid extraction network performs multi-layer feature mapping processing on the infrared sensing data stream to generate an infrared multi-scale feature map set. The infrared target detection head network performs candidate region generation and category prediction processing on the infrared multi-scale feature map set.

[0046] In this embodiment, a second infrared detection branch is constructed within the multi-detection algorithm collaborative ensemble network. The structure of this second infrared detection branch is completely symmetrical to the first visible light detection branch. The infrared feature pyramid extraction network also adopts the ResNet50-FPN architecture, but the number of input channels for ResNet50 is modified to 1, and the weight parameters use weights pre-trained on the infrared image dataset. This multi-detection algorithm collaborative ensemble network performs multi-layer feature mapping on the input infrared image, generating a set of infrared multi-scale feature maps, also containing five scales from P3 to P7, with 256 channels. The infrared target detection head network has the same structure as the visible light target detection head network, including a classification subnetwork and a regression subnetwork, outputting multi-scale candidate target regions and their category predictions in the infrared modality.

[0047] Step S133: Construct the third cross-modal fusion detection branch of the multi-detection algorithm collaborative integration network. The third cross-modal fusion detection branch includes a cross-modal feature interaction fusion module and a cross-modal target detection head network. The cross-modal feature interaction fusion module receives the visible light multi-scale feature map set and the infrared multi-scale feature map set as input, and simultaneously receives the cross-modal scene semantic association map as contextual prior knowledge input.

[0048] In this embodiment, a third cross-modal fusion detection branch is constructed within a multi-detection algorithm collaborative integration network. The core of the cross-modal feature interaction fusion module of this third cross-modal fusion detection branch is a graph convolutional encoder and a cross-attention fusion unit. The graph convolutional encoder receives the cross-modal scene semantic association graph G_cross generated in step S1211 as input. G_cross contains a node feature matrix X (of shape [N, D], where N is the total number of nodes and D is the feature dimension 2048) and an adjacency matrix A (of shape [N, N]). The graph convolutional encoder consists of two stacked graph convolutional networks. The propagation rule for each layer is defined as H^{(l+1)}=ReLU(A_hat*H^{(l)}*W^{(l)}), where A_hat=D^{-1 / 2}*(A+I)*D^{-1 / 2} is the normalized symmetric adjacency matrix, I is the identity matrix, D is the degree matrix, H^{(0)}=X is the initial node features, and W^{(l)} is the learnable weight matrix of the l-th layer. After passing through the graph convolutional encoder, the features of each node are updated to a hidden representation that incorporates neighborhood context information. It simultaneously receives the set of visible light multi-scale feature maps P3_vis to P7_vis generated by the first visible light detection branch and the set of infrared multi-scale feature maps P3_ir to P7_ir generated by the second infrared detection branch as input.

[0049] Step S134: Input the cross-modal scene semantic association graph into the graph convolutional encoder in the cross-modal feature interaction fusion module. The graph convolutional encoder performs multi-layer graph convolutional message passing processing on the visible light target nodes, infrared target nodes, co-occurrence relationship connection edges, temperature gradient relationship connection edges, and cross-modal semantic corresponding connection edges in the cross-modal scene semantic association graph to generate a context-enhanced visible light node feature vector for each visible light target node and a context-enhanced infrared node feature vector for each infrared target node.

[0050] In this embodiment, the cross-modal scene semantic association graph G_cross is input into the graph convolutional encoder in the cross-modal feature interaction fusion module. The graph convolutional encoder first initializes the node features and edge relationships in G_cross. For visible light target nodes, their initial features are the visible light target visual feature vector extracted in step S122. For infrared target nodes, their initial features are the infrared target thermal radiation feature vector extracted in step S128. For background nodes, their features are initialized as the feature vector of the corresponding region. The encoder performs two layers of graph convolutional message passing. In the first layer, each node aggregates the features of its neighboring nodes. Specifically, for node i, the features of its neighboring node j are weighted and summed using edge weights, which are determined by the edge type: the weight of co-occurrence relation connecting edges is the normalized co-occurrence frequency parameter; the weight of temperature gradient relation connecting edges is the reciprocal of the normalized temperature difference parameter; and the weight of cross-modal semantic correspondence connecting edges is the normalized feature similarity value. The aggregated neighbor features and the node's own features are fused through a fully connected layer and then through a ReLU activation function to obtain the first layer output. In the second layer, the above process is repeated, expanding the aggregation scope to second-order neighbor nodes. After two rounds of message passing, the hidden representation learned by each node incorporates scene context information from a wider range around it. Finally, the encoder outputs the context-enhanced visible light node feature vector for each visible light target node, and the context-enhanced infrared node feature vector for each infrared target node.

[0051] Step S135: Perform cross-modal feature fusion processing on the context-enhanced visible light multi-scale feature map set and the context-enhanced infrared multi-scale feature map set, and generate multi-branch collaborative detection integration results.

[0052] Step S136: Based on the correspondence between the visible light target node in the cross-modal scene semantic association graph and the spatial position in the visible light multi-scale feature map set, perform element-wise addition and fusion processing on the context-enhanced visible light node feature vector and the visible light multi-scale feature map at the corresponding spatial position to generate a context-enhanced visible light multi-scale feature map set.

[0053] In this embodiment, for each feature map layer in the visible light multi-scale feature map set, such as the P4_vis layer, its spatial resolution is 1 / 16 of the original image. For each visible light target node in the cross-modal scene semantic association map, the center coordinates of its corresponding target region in the original image are mapped to the coordinate position on the P4_vis feature map. The context-enhanced visible light node feature vector (dimensional 2048) of this node is reduced to 256 dimensions through a 1x1 convolutional layer, making it consistent with the number of channels in the P4_vis feature map. Then, the dimensionality-reduced node feature vector is added element-wise to the feature vector at the corresponding position on the P4_vis feature map. A node may correspond to a spatial region rather than a single pixel, so all feature points within that region are added to the node's feature vector. For spatial locations not covered by any visible light target nodes, the features remain unchanged. The same operation is performed on all scale feature maps P3_vis to P7_vis to generate a context-enhanced visible light multi-scale feature map set.

[0054] Step S137: Based on the correspondence between the infrared target nodes in the cross-modal scene semantic association map and the spatial positions in the infrared multi-scale feature map set, the context-enhanced infrared node feature vectors and the infrared multi-scale feature maps at the corresponding spatial positions are fused element-wise to generate a context-enhanced infrared multi-scale feature map set.

[0055] In this embodiment, the infrared multi-scale feature map set P3_ir to P7_ir is subjected to an operation that is completely symmetrical to step S136. The context-enhanced infrared node feature vector generated in step S134 is reduced to 256 dimensions through an independent 1x1 convolutional layer. Then, based on the spatial mapping position of the infrared target node on the infrared feature map, it is added element-wise with the feature map at the corresponding position to generate the context-enhanced infrared multi-scale feature map set.

[0056] Step S138: Input the context-enhanced visible light multi-scale feature map set and the context-enhanced infrared multi-scale feature map set into the cross-attention fusion unit in the cross-modal feature interaction fusion module. Calculate the cross-attention weight distribution of the visible light modal features and infrared modal features at each spatial scale level through the cross-attention fusion unit. Perform weighted fusion processing on the features of the two modalities according to the cross-attention weight distribution to generate a cross-modal fused feature map set.

[0057] In this embodiment, the set of context-enhanced visible light multi-scale feature maps generated in step S136 is denoted as V_layers, and the set of context-enhanced infrared multi-scale feature maps generated in step S137 is denoted as I_layers. For each scale level l (corresponding to P3 to P7), the cross-attention fusion unit performs the following operations: reshapes the feature maps of V_layers[l] into a sequence, where each spatial location is regarded as a query vector. Similarly, the feature maps of I_layers[l] are reshaped into a sequence, where each spatial location is regarded as a key vector and a value vector. The formula for calculating the cross-attention weight matrix is ​​Attention_weights=softmax((Q*K^T) / sqrt(d_k)), where Q comes from the visible light feature map, K comes from the infrared feature map, and d_k is the dimension of the key vector. Then, the calculated attention weights are applied to the value vector of the infrared feature map to obtain the weighted infrared features. This process can also be performed in reverse, using the infrared features as the query and the visible light features as the key and value, to calculate the attention in another direction. Finally, the original visible light features, the original infrared features, and the features weighted by attention in both directions are concatenated along the channel dimension, and then a 1x1 convolutional layer is used to fuse the number of channels back to 256 dimensions, generating a cross-modal fusion feature map for this scale. This operation is performed on all scale levels to obtain the cross-modal fusion feature map set F_fusion_layers.

[0058] Step S139: Input the cross-modal fusion feature map set into the cross-modal target detection head network, and perform multi-scale candidate target region extraction processing on the cross-modal fusion feature map set through the candidate region generation layer in the cross-modal target detection head network to generate a cross-modal candidate target region set.

[0059] In this embodiment, the cross-modal target detection head network adopts the same RetinaNet detection head structure as the visible light target detection head network. The cross-modal fusion feature map set F_fusion_layers generated in step S138 is input into this detection head network. This detection head network pre-defines multiple anchor boxes of different sizes and aspect ratios at each spatial location of each feature map scale. For each anchor box, the classification sub-network predicts its probability of belonging to each target category, and the regression sub-network predicts its offset relative to the ground truth bounding box. By setting a classification confidence threshold of 0.05, a portion of candidate anchor boxes are selected. Then, a non-maximum suppression algorithm is performed on the remaining candidate boxes, that is, first sorted in descending order of classification confidence, and then removing candidate boxes whose intersection-union ratio (IoU) with higher confidence bounding boxes is greater than 0.5. The final retained candidate boxes are the cross-modal candidate target region set.

[0060] Step S1310: Integrate and fuse the visible light detection results output by the visible light target detection head network of the first visible light detection branch, the infrared detection results output by the infrared target detection head network of the second infrared detection branch, and the cross-modal detection results output by the cross-modal target detection head network of the third cross-modal fusion detection branch to generate a multi-branch collaborative detection integration result.

[0061] In this embodiment, detection results are obtained from the outputs of three branches. Each detection result contains a series of detected targets, each consisting of bounding box coordinates, a class label, and a classification confidence score. During result integration, the bounding boxes output from all branches are first collected to form a detection box pool. For any two detection boxes in the pool, if their intersection-union ratio (IU) is greater than 0.5, they are determined to point to the same target. For multiple detection boxes identified as pointing to the same target, a weighted voting fusion method is used to determine the final class label. Specifically, the confidence vectors of all detection boxes pointing to the same target are summed element-wise, and the class with the largest sum is taken as the final class. For the bounding box positions, a weighted average fusion method is used, where the bounding box coordinates of each detection box are weighted by their confidence scores to obtain the final bounding box coordinates. After the above fusion process, the generated detection result is the multi-branch collaborative detection integration result.

[0062] Step S140: Use the open domain few-sample training set to train the pre-constructed pseudo-label generation network to generate a few-sample pseudo-labeling capability, and call the trained pseudo-label generation network to perform pseudo-label generation and pseudo-boundary box generation on unlabeled open scene images to obtain a pseudo-category label set and a pseudo-boundary box label set.

[0063] Step S141: Construct a pseudo-label generation network, which includes a shared feature encoder, a region proposal generation sub-network, a pseudo-class prediction sub-network, and a pseudo-boundary box regression sub-network. The shared feature encoder performs feature encoding processing on the input image to generate an input image feature map.

[0064] In this embodiment, a pseudo-annotation generation network is constructed. The shared feature encoder of this pseudo-annotation generation network adopts the backbone network structure of Faster R-CNN, specifically a combination of ResNet50 and a feature pyramid network. This shared feature encoder receives input images of arbitrary resolution and outputs a feature map with a spatial resolution of 1 / 16 of the input image and 256 channels. The region proposal generation sub-network consists of a 3x3 convolutional layer followed by two 1x1 branch convolutional layers. It generates a series of candidate proposal regions on this feature map using a sliding window. Each candidate proposal region is obtained by adjusting the position of an anchor box through position regression. The pseudo-class prediction sub-network and the pseudo-boundary box regression sub-network share a fully connected layer structure, receiving the features of each candidate proposal region output by the region proposal generation sub-network and outputting the class probability distribution and bounding box offset of the region, respectively.

[0065] Step S142: Input the visible light small sample training images and infrared small sample training images in the open domain small sample training set into the shared feature encoder respectively, and perform feature extraction processing on the visible light small sample training images and infrared small sample training images through the multi-level convolutional computation layer group in the shared feature encoder to generate visible light small sample feature mapping map and infrared small sample feature mapping map.

[0066] In this embodiment, a batch of training samples is taken from the open-domain few-sample training set, including visible light few-sample training images and their paired infrared few-sample training images. These two types of images are then input into the shared feature encoder of the pseudo-annotation generation network. Since the number of input channels in the ResNet50 network of the shared feature encoder is fixed at 3, for a single-channel infrared image, it is copied 3 times and stitched together to form a 3-channel pseudo-color image before input. The multi-level convolutional computation layers of this shared feature encoder progressively downsample and extract features from the input image, ultimately generating visible light few-sample feature maps and infrared few-sample feature maps.

[0067] Step S143: Input the visible light small sample feature map and the infrared small sample feature map into the region proposal generation sub-network respectively. Perform multi-scale window traversal scanning processing on the feature map through the sliding window scanning module in the region proposal generation sub-network to generate a set of visible light candidate proposal regions and a set of infrared candidate proposal regions.

[0068] In this embodiment, the visible light few-sample feature map generated in step S142 is input into the region proposal generation subnetwork. The sliding window scanning module in this subnetwork uses a 3x3 convolutional kernel, sliding across the feature map with a stride of 1. At the center of each sliding window, corresponding to a region in the original image, nine predefined anchor boxes of different scales and aspect ratios are used. The subnetwork outputs two predictions for each anchor box: a probability score indicating that the anchor box belongs to a foreground target, and the offset of the anchor box relative to the real target. By setting a foreground score threshold of 0.6, high-quality candidate anchor boxes are selected, and after non-maximum suppression processing, a set of visible light candidate proposal regions is finally obtained. Similarly, the same operation is performed on the infrared few-sample feature map to generate an infrared candidate proposal region set.

[0069] Step S144: Input the set of visible light candidate proposal regions into the pseudo-class prediction sub-network, perform semantic feature encoding and category classification processing on each visible light candidate proposal region through the multilayer perceptron in the pseudo-class prediction sub-network, generate a visible light pseudo-class prediction probability distribution, and calculate the visible light pseudo-class prediction loss value based on the visible light pseudo-class prediction probability distribution and the real category label data in the open domain small sample training set.

[0070] In this embodiment, for each visible light candidate proposal region, a fixed-size 7x7x256 feature map corresponding to that region is extracted from the feature map generated by the shared feature encoder using a region of interest alignment layer, and then flattened into a vector. This vector is then input into the multilayer perceptron of the pseudo-class prediction subnetwork. This multilayer perceptron contains two fully connected layers; the first layer outputs a 4096-dimensional value, and the second layer outputs a score equal to the number of target classes. This score is passed through a softmax function to obtain the visible light pseudo-class prediction probability distribution. The difference between this distribution and the true class label data is calculated using the cross-entropy loss function to obtain the visible light pseudo-class prediction loss value, Loss_cls_vis.

[0071] Step S145: Input the set of visible light candidate proposal regions into the pseudo bounding box regression sub-network, perform position offset prediction processing on each visible light candidate proposal region through the bounding box coordinate offset prediction layer in the pseudo bounding box regression sub-network, generate visible light pseudo bounding box offset prediction value, and calculate visible light pseudo bounding box regression loss value based on the visible light pseudo bounding box offset prediction value and the real bounding box annotation data in the open domain small sample training set.

[0072] In this embodiment, the feature vector corresponding to each visible light candidate proposal region is simultaneously input into the pseudo-boundary box regression sub-network. The structure of this pseudo-boundary box regression sub-network is similar to that of the pseudo-class prediction sub-network; its output layer is a 4-dimensional vector, representing the horizontal offset, vertical offset, width scaling, and height scaling of the center point of the candidate proposal region, respectively. The difference between the predicted offset and the true offset calculated based on the actual bounding box annotation data is calculated using the SmoothL1 loss function to obtain the visible light pseudo-boundary box regression loss value, Loss_reg_vis.

[0073] Step S146: Perform pseudo-label prediction and joint loss optimization on the infrared candidate proposal regions to obtain the trained pseudo-label generation network, and use it to generate a set of pseudo-class labels and a set of pseudo-boundary box labels for unlabeled open scene images.

[0074] Step S147: Input the infrared candidate proposal region set into the pseudo-class prediction sub-network, and perform semantic feature encoding and category classification processing on each infrared candidate proposal region through the multilayer perceptron in the pseudo-class prediction sub-network to generate an infrared pseudo-class prediction probability distribution. Calculate the infrared pseudo-class prediction loss value based on the infrared pseudo-class prediction probability distribution and the real category label data in the open domain small sample training set.

[0075] In this embodiment, the operation performed on the infrared candidate proposal region set is completely symmetrical to step S144. The feature vector of each infrared candidate proposal region is input into the pseudo-class prediction sub-network to obtain the infrared pseudo-class prediction probability distribution. Similarly, the cross-entropy loss function is used to calculate the difference between it and the true class label data to obtain the infrared pseudo-class prediction loss value Loss_cls_ir.

[0076] Step S148: Input the infrared candidate proposal region set into the pseudo bounding box regression sub-network, perform position offset prediction processing on each infrared candidate proposal region through the bounding box coordinate offset prediction layer in the pseudo bounding box regression sub-network, generate infrared pseudo bounding box offset prediction value, and calculate infrared pseudo bounding box regression loss value based on the infrared pseudo bounding box offset prediction value and the real bounding box annotation data in the open domain small sample training set.

[0077] In this embodiment, the infrared candidate proposed region set is subjected to an operation that is completely symmetrical to step S145, and the infrared pseudo bounding box regression loss value Loss_reg_ir is calculated.

[0078] Step S149: The visible light pseudo-class prediction loss value, the visible light pseudo-boundary box regression loss value, the infrared pseudo-class prediction loss value, and the infrared pseudo-boundary box regression loss value are weighted and summed to generate a joint pseudo-label training loss value. The pseudo-label generation network is updated according to the joint pseudo-label training loss value. Training is stopped when the joint pseudo-label training loss value tends to converge, and the trained pseudo-label generation network is obtained.

[0079] In this embodiment, weight coefficients are set for four loss terms, with the category prediction loss having a weight of 1 and the bounding box regression loss having a weight of 1. The joint pseudo-label training loss value Loss_total = Loss_cls_vis + Loss_reg_vis + Loss_cls_ir + Loss_reg_ir. A stochastic gradient descent optimizer is used, with an initial learning rate of 0.001, a momentum parameter of 0.9, and a weight decay parameter of 0.0005. Backpropagation and parameter updates are performed on all trainable parameters of the pseudo-label generation network. In each iteration, a batch of 8 images is randomly selected from the open-domain small sample training set for training. When the decrease in Loss_total is less than 0.001 for 10 consecutive iterations, training is stopped, and the trained pseudo-label generation network is obtained.

[0080] Step S1410: Obtain an unlabeled open scene image, which includes an unlabeled visible light image and an unlabeled infrared image. Input the unlabeled visible light image and the unlabeled infrared image into the trained pseudo-label generation network, and generate an unlabeled visible light candidate region set and an unlabeled infrared candidate region set through the shared feature encoder and the region proposal generation sub-network.

[0081] In this embodiment, a batch of new unlabeled open scene images are acquired. These images come from the same deployment scene as the training data but have not been manually labeled. For each pair of unlabeled visible light images and unlabeled infrared images, they are respectively input into the pseudo-labeling generation network trained in step S149. This pseudo-labeling generation network follows the process of steps S142 and S143, first extracting feature maps, and then generating unlabeled visible light candidate proposal regions and unlabeled infrared candidate proposal regions through a region proposal generation sub-network.

[0082] Step S1411: Input the unlabeled visible light candidate proposal region set into the pseudo-class prediction subnetwork and pseudo-boundary box regression subnetwork of the trained pseudo-label generation network to generate visible light pseudo-class labels and visible light pseudo-boundary box labels. Input the unlabeled infrared candidate proposal region set into the pseudo-class prediction subnetwork and pseudo-boundary box regression subnetwork of the trained pseudo-label generation network to generate infrared pseudo-class labels and infrared pseudo-boundary box labels. Combine the visible light pseudo-class labels and the infrared pseudo-class labels into a pseudo-class label set. Combine the visible light pseudo-boundary box labels and the infrared pseudo-boundary box labels into a pseudo-boundary box label set.

[0083] In this embodiment, the unlabeled visible light candidate suggestion region set generated in step S1410 is input into the pseudo-class prediction subnetwork and the pseudo-boundary box regression subnetwork to obtain the class probability distribution and bounding box offset of each candidate suggestion region. The class with the highest probability exceeding the threshold of 0.7 is selected as the pseudo-class label for that candidate suggestion region, and the coordinates of the pseudo-boundary box in the original image are calculated based on the predicted offset. This operation is performed on all candidate suggestion regions to generate a visible light pseudo-class label set and a visible light pseudo-boundary box label set. The same operation is performed on the unlabeled infrared candidate suggestion region set to generate an infrared pseudo-class label set and an infrared pseudo-boundary box label set. Finally, the pseudo-class labels of the visible light and infrared modes are merged into a pseudo-class label set, and the pseudo-boundary box labels of the two modes are merged into a pseudo-boundary box label set.

[0084] Step S150: Combine the open domain small sample training set, the unlabeled open scene image, the pseudo-class label set, and the pseudo-boundary box label set into an expanded training set. Use the expanded training set to perform cross-modal multi-branch joint training on the multi-detection algorithm collaborative ensemble network to generate a trained open domain target detection model. Use the open domain target detection model to perform open domain target recognition processing on the target scene and output the open domain category information and open domain bounding box localization information of the target object.

[0085] Step S151: Construct a subset of real labeled training samples by combining the visible light small sample training images and infrared small sample training images in the open domain small sample training set with the corresponding real bounding box annotation data and real class label data.

[0086] In this embodiment, all original ground truth labeled training samples are extracted from the open-domain small sample training set. Each training sample is an image pair (one visible light image and one infrared image) and its corresponding ground truth bounding box annotation data and ground truth class label data. These samples are organized into a ground truth labeled training sample subset D_real.

[0087] Step S152: Construct a subset of pseudo-labeled training samples by combining the unlabeled visible light image and the unlabeled infrared image in the unlabeled open scene image with the corresponding pseudo-class labels in the pseudo-class label set and the corresponding pseudo-boundary box labels in the pseudo-boundary box label set.

[0088] In this embodiment, for each pair of unlabeled open scene images obtained in step S1410, they are paired with the corresponding pseudo-class labels and pseudo-boundary box annotations generated in step S1411. The paired samples are then organized into a pseudo-labeled training sample subset D_pseudo.

[0089] Step S153: Merge the subset of real labeled training samples and the subset of pseudo labeled training samples to generate the expanded training set. Each training sample in the expanded training set contains input image data, bounding box annotation data and category label data. The input image data is a visible light image or an infrared image.

[0090] In this embodiment, the subset of real labeled training samples D_real and the subset of pseudo labeled training samples D_pseudo are merged to obtain a larger expanded training set D_aug. Each sample in this expanded training set contains image data (which may be a visible light image or an infrared image), bounding box annotation data (which may be real annotations or pseudo annotations), and category label data (which may be real labels or pseudo labels).

[0091] Step S154: Input the visible light training samples in the expanded training set into the first visible light detection branch of the multi-detection algorithm collaborative integration network, generate the category prediction result and bounding box prediction result of the visible light detection branch through the visible light feature pyramid extraction network and the visible light target detection head network of the first visible light detection branch, and calculate the category prediction loss value and bounding box regression loss value of the visible light detection branch.

[0092] In this embodiment, visible light training samples are extracted from the expanded training set D_aug. The visible light image of this sample is input into the first visible light detection branch of the multi-detection algorithm collaborative ensemble network. Following the process of step S131, the first visible light detection branch outputs a multi-scale feature map and generates class prediction and bounding box prediction through the detection head network. FocalLoss is used as the loss function for class prediction to solve the problem of imbalanced positive and negative samples. SmoothL1Loss is used as the loss function for bounding box regression. Based on the prediction results and the annotation data of the sample, the class prediction loss value Loss_vis_cls and the bounding box regression loss value Loss_vis_reg of the visible light detection branch are calculated.

[0093] Step S155: Input the infrared training samples in the expanded training set into the second infrared detection branch of the multi-detection algorithm collaborative integration network, generate the category prediction result and bounding box prediction result of the infrared detection branch through the infrared feature pyramid extraction network and infrared target detection head network of the second infrared detection branch, and calculate the category prediction loss value and bounding box regression loss value of the infrared detection branch.

[0094] In this embodiment, infrared training samples are taken from the expanded training set D_aug and input into the second infrared detection branch of the multi-detection algorithm collaborative ensemble network. Following the process of step S132, the category prediction loss value Loss_ir_cls and the bounding box regression loss value Loss_ir_reg of the infrared detection branch are calculated.

[0095] Step S156: Input the visible light training samples and infrared training samples in the expanded training set into the third cross-modal fusion detection branch of the multi-detection algorithm collaborative integration network in pairs. Input the cross-modal scene semantic association graph as contextual prior knowledge into the third cross-modal fusion detection branch. Generate the category prediction result and bounding box prediction result of the cross-modal detection branch through the cross-modal feature interaction fusion module and the cross-modal target detection head network of the third cross-modal fusion detection branch. Calculate the category prediction loss value and bounding box regression loss value of the cross-modal detection branch.

[0096] In this embodiment, pairs of visible light training samples and infrared training samples are extracted from the expanded training set D_aug. These are simultaneously input into the third cross-modal fusion detection branch. Simultaneously, the cross-modal scene semantic association graph G_cross generated in step S1211 is also input as prior knowledge into the cross-modal feature interaction fusion module of this branch. Following the process of steps S133 to S139, the third cross-modal fusion detection branch finally outputs the category prediction result and bounding box prediction result of the cross-modal detection branch. FocalLoss and SmoothL1Loss are used to calculate the category prediction loss value Loss_fusion_cls and the bounding box regression loss value Loss_fusion_reg, respectively.

[0097] Step S157: Perform joint loss optimization training on the multi-detection algorithm collaborative integration network to generate a trained open-domain target detection model, and use the model to infer and output the target scene.

[0098] Step S158: The category prediction loss and bounding box regression loss of the visible light detection branch, the category prediction loss and bounding box regression loss of the infrared detection branch, and the category prediction loss and bounding box regression loss of the cross-modal detection branch are weighted and fused to generate the total loss value for multi-branch joint training.

[0099] In this embodiment, weight coefficients are set for the loss of each branch. The visible light branch has a weight of 1, the infrared branch has a weight of 1, and the cross-modal fusion branch has a weight of 1.5 to emphasize the importance of the fusion branch. The total loss value for multi-branch joint training is Loss_total_train = Loss_vis_cls + Loss_vis_reg + Loss_ir_cls + Loss_ir_reg + 1.5 * (Loss_fusion_cls + Loss_fusion_reg).

[0100] Step S159: Perform backpropagation and gradient update processing on the network parameters of the multi-detection algorithm collaborative integration network according to the total loss value of the multi-branch joint training. In each training iteration, synchronously update the network weight parameters of the first visible light detection branch, the second infrared detection branch, and the third cross-modal fusion detection branch. When the total loss value of the multi-branch joint training meets the preset training convergence condition, stop training and generate the trained open domain target detection model.

[0101] In this embodiment, the Adam optimizer is used to optimize all parameters of the multi-detection algorithm collaborative ensemble network. The initial learning rate is set to 0.0001, and it decays to 0.9 times its original value every 5 training epochs. During training, each batch of samples contains 8 real-labeled samples and 8 pseudo-labeled samples. When the absolute value of the change in the total loss value over 5 consecutive batches is less than 0.01, the model is considered converged, and training stops. The network model saved at this point is the completed open-domain object detection model M_final.

[0102] Step S1510: Input the visible light sensing data stream and infrared sensing data stream of the target scene into the trained open domain target detection model. The open domain target detection model outputs the first visible light target category detection result and the first visible light target bounding box localization result through the first visible light detection branch, outputs the second infrared target category detection result and the second infrared target bounding box localization result through the second infrared detection branch, and outputs the third cross-modal target category detection result and the third cross-modal target bounding box localization result through the third cross-modal fusion detection branch.

[0103] In this embodiment, during the inference phase, visible light image frames and corresponding infrared image frames of the target scene acquired in real time are simultaneously input into the trained open-domain target detection model M_final. The model runs three detection branches in parallel, each outputting its own detection results. The output of the first visible light detection branch includes the probability vector of each detected target category and the bounding box coordinates. The output structure of the second infrared detection branch is the same. The third cross-modal fusion detection branch outputs the fused category probability vector and bounding box coordinates.

[0104] Step S1511: Perform weighted voting fusion processing on the first visible light target category detection result, the second infrared target category detection result, and the third cross-modal target category detection result to generate open domain category information for each target object in the target scene. Perform weighted average fusion processing on the first visible light target bounding box positioning result, the second infrared target bounding box positioning result, and the third cross-modal target bounding box positioning result to generate open domain bounding box positioning information for each target object in the target scene.

[0105] In this embodiment, the outputs of the three branches are integrated. For the category detection results, weighted voting fusion is used. The voting weights of the three branches are set to 0.3, 0.3, and 0.4, respectively. Each branch's category prediction for the same target candidate region can be regarded as a vote. The votes of all branches are added according to their weights, and the category with the highest total number of votes is selected as the open domain category information of the final category. For the bounding box localization results, weighted average fusion is used. The bounding box coordinates (top-left x-coordinate, top-left y-coordinate, width, and height) of the three branches pointing to the same target are weighted and averaged separately, with the weights consistent with the category voting weights. The calculated bounding box coordinates are used as the open domain bounding box localization information of the final result. Finally, the fused category information and bounding box information are output to complete the open domain target recognition of the target scene.

[0106] For example, the method may further include: step S210: extracting all visible light target nodes and infrared target nodes from the cross-modal scene semantic association graph to construct a complete set of cross-modal target nodes.

[0107] In this embodiment, based on the cross-modal scene semantic association graph G_cross generated in step S1211, the graph structure is traversed. Vertices of all types of visible light target nodes and all types of infrared target nodes are extracted from G_cross. These two types of vertices are merged into a new cross-modal target node set V_all. Each node in this cross-modal target node set retains its original attributes, including spatial location coordinates, visual feature vectors or thermal radiation feature vectors, and its connection relationships with other nodes in the graph.

[0108] Step S220: Perform feature concatenation processing on the visible light target visual feature vector and infrared target thermal radiation feature vector corresponding to each cross-modal target node in the full set of cross-modal target nodes to generate the joint cross-modal feature description vector of the cross-modal target node.

[0109] In this embodiment, for each node in the complete set of cross-modal target nodes V_all, its original modality is determined. If the node is a visible light target node, the visible light target visual feature vector F_vis (dimension 2048) is retrieved from the attributes stored in the node. Then, in G_cross, an infrared target node with a cross-modal semantic correspondence edge with the node is found, and the infrared target thermal radiation feature vector F_ir of the infrared target node is retrieved. F_vis and F_ir are concatenated along the feature dimension to generate a joint cross-modal feature description vector F_joint=concat(F_vis, F_ir) with a dimension of 4096. If the node is an infrared target node, a symmetrical operation is performed: first, its F_ir is retrieved, and then the corresponding visible light node is found and its F_vis is retrieved and concatenated. If no matching cross-modal corresponding node is found for a target node in the graph, its feature vector is concatenated with a vector of all zeros to maintain dimensional uniformity.

[0110] Step S230: Input the joint cross-modal feature description vector of all cross-modal target nodes in the full set of cross-modal target nodes into the pre-constructed cross-modal scene graph clustering analysis network, and calculate the local density parameter and the minimum distance parameter to higher local density nodes for each joint cross-modal feature description vector through the density peak clustering module in the cross-modal scene graph clustering analysis network.

[0111] In this embodiment, a cross-modal scene graph clustering analysis network is constructed. The core of this cross-modal scene graph clustering analysis network is a density peak clustering module. The set of joint cross-modal feature description vectors {F_joint_i} of all nodes in V_all generated in step S220 is input into this module. First, the Euclidean distance between any two feature vectors is calculated to form a distance matrix D, where D(i, j) = ||F_joint_i - F_joint_j||2. Then, for each node i, its local density parameter ρ_i is calculated. The calculation method of this local density parameter is as follows: count the number of nodes j among all other nodes that satisfy D(i, j) < cutoff_distance, where cutoff_distance is a preset cutoff distance parameter. Next, for each node i, its minimum distance parameter δ_i to nodes with higher local density is calculated. Specifically, find all nodes j with local density ρ_j greater than ρ_i, and take the minimum D(i, j) among them as the value of δ_i. For the node with the maximum local density, its δ_i is equal to the maximum value among all D(i, j).

[0112] Step S240: Screen out the clustering center nodes from the cross-modal target node全集 according to the local density parameter and the minimum distance parameter, and divide each clustering center node and its corresponding density-reachable neighborhood nodes into a scene semantic cluster, generating an initial cross-modal scene semantic cluster set.

[0113] In this embodiment, the local density ρ_i of each node i is multiplied by the minimum distance δ_i to obtain a decision value γ_i = ρ_i * δ_i. The decision values γ_i of all nodes are sorted in descending order. The nodes before the point where the decision value mutates are selected as the clustering center nodes. The judgment method for the decision value mutation is as follows: calculate the γ_i ratio of adjacent sorted nodes, and when the ratio is greater than the mutation threshold alpha, it is considered a mutation. Each clustering center node corresponds to a cluster. Then, for each non-clustering center node, it is assigned to the cluster where the node with higher density and the closest distance to it is located. According to this rule, all nodes are assigned to each cluster to form an initial cross-modal scene semantic cluster set C_init = {C1, C2,..., Ck}, where each cluster Ci contains a set of cross-modal target nodes aggregated in the feature space.

[0114] Step S250: Perform intra-cluster cross-modal target node structure pattern extraction processing on each initial cross-modal scene semantic cluster in the initial cross-modal scene semantic cluster set, and extract the connection topology subgraph of all cross-modal target nodes in the initial cross-modal scene semantic cluster in the cross-modal scene semantic association graph.

[0115] It should be noted that the "全集" in step S230 seems to be an incorrect or incomplete expression in Chinese. It might need to be further clarified in the original context for a more accurate translation. Also, the " ", "

[0112] ", etc. are likely specific identifiers that should be left unchanged as per the requirements.In this embodiment, for each cluster Ci in the initial cross-modal scene semantic cluster set C_init, all cross-modal target nodes contained within that cluster are extracted. Then, in the original cross-modal scene semantic association graph G_cross, a subgraph is extracted, consisting of these nodes and all existing edges between them (including co-occurrence relationship edges, temperature gradient relationship edges, and cross-modal semantic correspondence edges). This subgraph is denoted as SubG_i, and it reflects the spatiotemporal and modal association structure between targets within that cluster.

[0116] Step S260: Perform frequent subgraph pattern mining on the connected topology subgraph, and extract the subgraph structures in the connected topology subgraph whose frequency of occurrence meets the preset frequency threshold as the scene semantic pattern graph primitives of the initial cross-modal scene semantic cluster.

[0117] In this embodiment, for each cluster Ci, the frequent subgraph SubG_i corresponding to the connection topology is extracted using the gSpan frequent subgraph mining algorithm. First, all edges in SubG_i are normalized and sorted by label. Then, a depth-first search strategy is used to gradually expand the subgraph, starting from a single edge. During the expansion process, the number of times the current subgraph appears in all isomorphic regions of SubG_i is calculated as its support. A minimum support threshold min_sup is set. The subgraph structures with support greater than min_sup are recorded as the scene semantic pattern graph primitive Pattern_i for that cluster. The graph primitive represents the recurring structured configuration pattern in the scene semantic cluster, such as "target A and target B often appear simultaneously and are very close to each other".

[0118] Step S270: Perform graph structure similarity comparison processing between the scene semantic pattern graph primitive and the predefined common target scene semantic pattern graph primitive library, and calculate the graph edit distance parameter between the scene semantic pattern graph primitive and each common target scene semantic pattern graph primitive in the common target scene semantic pattern graph primitive library.

[0119] In this embodiment, a common target scene semantic pattern graph primitive library Lib_common is pre-constructed. This library stores typical semantic pattern graph primitives trained with a large amount of standard scene data, such as the "soldier-armored vehicle-cooperative combat" pattern. For each scene semantic pattern graph primitive Pattern_i extracted in step S260, its graph edit distance is calculated with each standard primitive Lib_j in Lib_common. The graph edit distance refers to the minimum number of editing operations required to transform one graph into another. Operations include inserting nodes, deleting nodes, inserting edges, deleting edges, modifying node labels, and modifying edge labels. The calculated graph edit distance is denoted as GED(Pattern_i, Lib_j). The smaller the distance, the more similar the structures of the two graphs.

[0120] Step S280: Mark the scene semantic pattern graph primitives whose graph editing distance parameter is greater than the preset graph editing distance threshold as unconventional target scene semantic pattern graph primitives, and combine all unconventional target scene semantic pattern graph primitives into an unconventional target scene semantic pattern set.

[0121] In this embodiment, a graph edit distance threshold GED_th is set. For Pattern_i, if for all Lib_j∈Lib_common, min(GED(Pattern_i, Lib_j))>GED_th, then it is determined that Pattern_i cannot match any common pattern in the library, and it is marked as an unconventional target scene semantic pattern graph primitive. All detected unconventional graph primitives are collected to form the unconventional target scene semantic pattern set Set_unusual.

[0122] Step S290: Perform graph node attribute statistical processing on each unconventional target scene semantic pattern graph primitive in the unconventional target scene semantic pattern set, and extract the statistical features of visible light target node types and infrared target node types contained in the unconventional target scene semantic pattern graph primitive.

[0123] In this embodiment, for each unconventional graph primitive U_pattern in Set_unusual, the type distribution of all visible light target nodes and infrared target nodes in that subgraph is statistically analyzed. First, the category label of each node is extracted (if the node comes from pseudo-labels or real labels). Then, the frequency of occurrence of visible light target nodes of each category is calculated to obtain the visible light target node type statistical feature Vec_vis_stat. Similarly, the frequency of occurrence of infrared target nodes of each category is calculated to obtain the infrared target node type statistical feature Vec_ir_stat. These two statistical feature vectors are concatenated as the structural feature description of this unconventional graph.

[0124] Step S2100: Generate a corresponding unconventional target category semantic prototype vector for each unconventional target scene semantic pattern graph primitive based on the statistical features of the visible light target node type and the statistical features of the infrared target node type, and add the unconventional target category semantic prototype vector to the open domain category semantic prototype space of the multi-detection algorithm collaborative integration network.

[0125] In this embodiment, based on the statistical features Vec_vis_stat and Vec_ir_stat extracted in step S290, a semantic prototype vector Proto_unusual is generated for each unconventional U_pattern. Specifically, the semantic embedding vectors of all node categories appearing in the pattern (which can be obtained from the pre-trained language model through category names) are weighted and summed according to the frequency ratio in the statistical features to obtain Proto_unusual. This prototype vector represents the new target category or scene category implied by the unconventional pattern. Finally, the generated Proto_unusual is added to the open-domain category semantic prototype space of the multi-detection algorithm collaborative ensemble network. This semantic prototype space is the basis for the classifier to make category discrimination in step S1511. By adding the new prototype vector, the network gains the ability to identify these unconventional target categories.

[0126] Step S310: Extract visible light small sample training images and corresponding infrared small sample training images from the open domain small sample training set to construct visible light and infrared small sample training image pairs.

[0127] In this embodiment, from the open-domain few-sample training set obtained in step S110, each visible light few-sample training image and its corresponding infrared few-sample training image are paired to construct an image pair. Each image pair (I_vis, I_ir) together constitutes a basic training unit.

[0128] Step S320: Perform local target region random occlusion processing on the visible light small sample training image in the visible light infrared small sample training image pair to generate a visible light occlusion enhanced training image after local target region random occlusion.

[0129] In this embodiment, a random occlusion enhancement strategy is adopted for the visible light training image I_vis in each image pair. First, one or more target regions to be occluded are randomly determined. These regions can be existing real target bounding boxes in the image or randomly generated rectangles on the image. For each selected rectangle, all pixel values ​​inside it are replaced with 0 or the pixel mean of the entire dataset. The size and position of the occluded region are randomly generated in each training iteration to simulate a scenario where the target is partially occluded. The processed image is denoted as I_vis_occl.

[0130] Step S330: Perform local target region random occlusion processing on the infrared small sample training image in the visible light infrared small sample training image pair to generate an infrared occlusion enhanced training image after local target region random occlusion.

[0131] In this embodiment, a random occlusion enhancement operation, symmetrical to step S320, is performed on the infrared training image I_ir in the image pair. The randomly occluded region can be the same spatial location corresponding to the occlusion region of the visible light image, or it can be an independently and randomly generated location. The processed image is denoted as I_ir_occl. Through independent random occlusion, the robustness of the model to the case of missing single-modal information is enhanced.

[0132] Step S340: Construct a visible light occlusion enhancement scene context semantic map and an infrared occlusion enhancement scene context semantic map for the visible light occlusion enhancement training image and the infrared occlusion enhancement training image, respectively.

[0133] In this embodiment, the processes of steps S121 to S129 are completely repeated for the I_vis_occl image generated in step S320 and the I_ir_occl image generated in step S330. That is, target background separation, node extraction, feature extraction, and edge relationship construction are performed on the occluded visible light image and infrared image, respectively, to generate the visible light occlusion enhancement scene context semantic map G_vis_occl and the infrared occlusion enhancement scene context semantic map G_ir_occl.

[0134] Step S350: Perform cross-modal scene graph collaborative discovery processing on the visible light occlusion enhancement scene context semantic graph and the infrared occlusion enhancement scene context semantic graph to generate an occlusion enhancement cross-modal scene semantic association graph.

[0135] In this embodiment, the process of steps S1210 to S1211 is repeated for the two semantic graphs G_vis_occl and G_ir_occl generated in step S340. That is, spatial location matching and cross-modal feature similarity calculation are performed, and cross-modal semantic correspondence connection edges are established between nodes that meet the conditions. Finally, an occlusion-enhanced cross-modal scene semantic association graph G_cross_occl is generated.

[0136] Step S360: Randomly swap the spatial positions of the visible light small sample training image and the infrared small sample training image in the visible light and infrared small sample training image pair to generate a visible light spatial swap enhanced training image and an infrared spatial swap enhanced training image.

[0137] In this embodiment, spatial position swapping enhancement is performed on the original image pair (I_vis, I_ir). First, a local image block P_vis is randomly cropped from I_vis, and a local image block P_ir of the same size is cropped from I_ir at the corresponding spatial position. Then, P_vis is pasted into the corresponding position of I_ir to form a mixed image I_ir_mix; simultaneously, P_ir is pasted into the corresponding position of I_vis to form a mixed image I_vis_mix. Through the above spatial position swapping, a new modal mixing scene is created.

[0138] Step S370: Construct a visible light interchange enhancement scene context semantic map and an infrared interchange enhancement scene context semantic map for the visible light spatial interchange enhancement training image and the infrared spatial interchange enhancement training image, respectively.

[0139] In this embodiment, the process of steps S121 to S129 is repeated for the mixed images I_vis_mix and I_ir_mix generated in step S360 to generate visible light interchangeable enhanced scene context semantic map G_vis_mix and infrared interchangeable enhanced scene context semantic map G_ir_mix, respectively.

[0140] Step S380: Perform cross-modal scene graph collaborative discovery processing on the visible light interchangeable enhanced scene context semantic graph and the infrared interchangeable enhanced scene context semantic graph to generate an interchangeable enhanced cross-modal scene semantic association graph.

[0141] In this embodiment, the process of steps S1210 to S1211 is repeated for the two semantic graphs G_vis_mix and G_ir_mix generated in step S370 to generate an interchangeable enhanced cross-modal scene semantic association graph G_cross_mix.

[0142] Step S390: Perform scene graph structure comparison and verification processing on the original cross-modal scene semantic association graph corresponding to the occlusion-enhanced cross-modal scene semantic association graph and the interchange-enhanced cross-modal scene semantic association graph and the visible light infrared small sample training image, and retain the enhanced cross-modal scene semantic association graph whose scene graph structure consistency with the original cross-modal scene semantic association graph meets the preset consistency conditions.

[0143] In this embodiment, for the original image pair, an original cross-modal scene semantic association graph G_cross_orig has been generated in step S1211. For G_cross_occl generated in step S350 and G_cross_mix generated in step S380, their structural consistency with G_cross_orig is calculated respectively. Structural consistency can be measured by calculating the graph edit distance between the two graphs; the smaller the distance, the more similar the structures. The consistency condition is set to a graph edit distance less than a preset threshold. Only those augmented graphs that are structurally sufficiently similar to the original graph (i.e., edit distance less than the threshold) are retained. For dissimilar augmented graphs, indicating that the augmentation operation violates the basic semantics of the scene, they are discarded. Through the above verification, it is ensured that the augmented samples used for training are semantically reasonable.

[0144] Step S3100: Combine the retained enhanced cross-modal scene semantic association graph and its corresponding enhanced training images and original bounding box annotation data into a scene graph enhancement training sample set.

[0145] In this embodiment, for each enhanced cross-modal scene semantic association graph that passes the structural consistency check, such as G_cross_occl, it is combined with the corresponding enhanced training image (I_vis_occl, I_ir_occl) and the ground truth bounding box annotation data carried by the original image pair to form a new enhanced training sample. All enhanced samples that pass the check constitute the scene graph enhanced training sample set D_aug_graph. This scene graph enhanced training sample set can be used to further expand the expanded training set in step S153 in subsequent training, enhancing the robustness of the model to complex scene structural changes.

[0146] Step S410: Input the unlabeled visible light image and the unlabeled infrared image from the unlabeled open scene image into the trained open domain target detection model.

[0147] In this embodiment, after obtaining the open-domain object detection model M_final trained in step S159, a new batch of unlabeled open scene image pairs (unlabeled visible light image I_unlab_vis and unlabeled infrared image I_unlab_ir) is acquired. These image pairs are then input into M_final to execute the inference process.

[0148] Step S420: Call the first visible light detection branch of the open domain target detection model to perform target detection processing on the unlabeled visible light image and generate a first visible light detection confidence score set.

[0149] In this embodiment, the first visible light detection branch of M_final processes the input I_unlab_vis and outputs all detected target candidate regions and their corresponding category confidence scores. The detection results are then collected into a first visible light detection confidence score set Conf_vis.

[0150] Step S430: Call the second infrared detection branch of the open domain target detection model to perform target detection processing on the unlabeled infrared image and generate a second infrared detection confidence score set.

[0151] In this embodiment, the second infrared detection branch of M_final processes the input I_unlab_ir and outputs all detected target candidate regions and their corresponding category confidence scores. The detection results are then collected into a second infrared detection confidence score set Conf_ir.

[0152] Step S440: Call the third cross-modal fusion detection branch of the open domain target detection model to perform joint target detection processing on the unlabeled visible light image and the unlabeled infrared image, and generate a third cross-modal detection confidence score set.

[0153] In this embodiment, the third cross-modal fusion detection branch of M_final performs joint processing on the image pair (I_unlab_vis, I_unlab_ir) and outputs the fused detection results, including the category confidence score for each detection region. These detection results are then collected into a third cross-modal detection confidence score set, Conf_fusion.

[0154] Step S450: Perform multi-branch confidence weighted fusion processing on the first visible light detection confidence score set, the second infrared detection confidence score set, and the third cross-modal detection confidence score set to generate a fused confidence score set.

[0155] In this embodiment, the detection results output from the three branches are matched. For any two detection boxes output from the three branches, if their intersection-union ratio (IU) is greater than 0.5, they are considered to detect the same target. For the same target, the confidence scores from the three branches are taken and weighted and summed according to the weights set in step S1511 to obtain the fusion confidence score of the target. Finally, the fusion confidence scores of all targets constitute the fusion confidence score set Conf_fused.

[0156] Step S460: Select the detection target regions whose fusion confidence scores are higher than the preset high confidence threshold from the fusion confidence score set, and use the category prediction results and bounding box prediction results corresponding to the selected detection target regions as high confidence pseudo-label data.

[0157] In this embodiment, the high confidence threshold Th_high is set to 0.9. All detected target regions with a fusion confidence score greater than 0.9 are selected from Conf_fused. For these regions, their corresponding category prediction results and bounding box prediction results are regarded as "credible" pseudo-labels, forming high-confidence pseudo-label data D_high_pseudo.

[0158] Step S470: Merge and deduplicate the high-confidence pseudo-label data with the pseudo-category label set and the pseudo-boundary box label set to generate an expanded pseudo-label data set.

[0159] In this embodiment, the D_high_pseudo generated in step S460 is merged with the pseudo-class label set and pseudo-boundary box annotation set generated in step S1411. For overlapping pseudo-label boxes appearing on the same image, if their intersection-union ratio is greater than 0.5, the one with the higher confidence is retained, and the other is removed. The merged and deduplicated result is denoted as D_pseudo_aug.

[0160] Step S480: Combine the expanded pseudo-labeled data set with the open domain small sample training set again to form an updated expanded training set, and use the updated expanded training set to retrain the open domain target detection model.

[0161] In this embodiment, the D_pseudo_aug generated in step S470 is combined again with the original open-domain small sample training set in step S110 to form a larger updated and expanded training set D_train_update. Using this new training set, the open-domain object detection model M_final is retrained again according to the process of steps S157 to S159, and its weight parameters are updated.

[0162] Step S490: After each round of retraining, the unlabeled open scene image is re-generated with pseudo-labels using the retrained open domain object detection model, and the updated and expanded training set is iteratively updated.

[0163] In this embodiment, steps S410 to S480 are repeated to form an iterative self-training loop. In each iteration, the more powerful model obtained from the previous retraining is used to regenerate higher-quality pseudo-labels on the unlabeled data. These newly generated pseudo-labels are then used to expand the training set and retrain the model. After each iteration, the model's performance improves, resulting in more accurate pseudo-labels and forming a positive feedback loop.

[0164] Step S4100: When the growth rate of the number of high-confidence pseudo-labeled data is lower than the preset growth threshold in multiple consecutive iterations, stop the iteration and combine the expanded pseudo-labeled dataset generated in the last round into a refined pseudo-labeled dataset.

[0165] In this embodiment, an iteration stopping condition is set. In each iteration, the number of high-confidence pseudo-labeled data D_high_pseudo generated in step S460 is counted. If the increase in this number is lower than the preset growth threshold Delta_min for three consecutive iterations, it indicates that the model performance has reached saturation, and further iteration yields little benefit. At this point, the iteration is stopped, and the expanded pseudo-labeled data set D_pseudo_final generated in the last round is used as the refined pseudo-labeled data set. This refined data set can be combined with the original small sample set to train the final, optimal open-domain object detection model.

[0166] Step S510: Obtain a pre-trained large-scale visual-language joint embedding model, which includes a visual encoding branch and a language encoding branch. The language encoding branch can map any text category name to a semantic embedding vector of fixed dimension.

[0167] In this embodiment, a joint visual-language embedding model, such as the CLIP model, pre-trained on a large-scale image-text dataset is obtained. This model contains two independent encoders: a visual encoder (ViT or ResNet) for extracting image features and a language encoder (Transformer) for extracting text features. The two encoders map images and text to the same high-dimensional semantic embedding space (e.g., 512-dimensional), ensuring that semantically similar image-text pairs have similar embedding vectors. Specifically, the language encoder can map arbitrary category name text (e.g., "armored vehicle") to a 512-dimensional semantic embedding vector.

[0168] Step S520: Extract the real category name text from the real category label data of the open domain small sample training set, input the real category name text into the language encoding branch of the large-scale visual-language joint embedding model, and generate the real category semantic embedding vector corresponding to each real category name text.

[0169] In this embodiment, the corresponding category name text, such as "tank" and "fighterjet," is extracted from the real category label data in the open-domain few-sample training set. This text is then sequentially input into the language encoding branch of a pre-trained visual-language joint embedding model. This language encoding branch encodes each category name and outputs a corresponding semantic embedding vector V_emb_real_i. All semantic embedding vectors of the real categories form a set.

[0170] Step S530: Extract pseudo-category name text from the pseudo-category label set, input the pseudo-category name text into the language encoding branch of the large-scale visual-language joint embedding model, and generate a pseudo-category semantic embedding vector corresponding to each pseudo-category name text.

[0171] In this embodiment, the name text corresponding to all pseudo-categories is extracted from the pseudo-category label set generated in step S1411. These names may contain some errors or noise. The name text is then input into the same language encoding branch to obtain the semantic embedding vector V_emb_pseudo_j for each pseudo-category.

[0172] Step S540: Use the real category semantic embedding vector and the pseudo category semantic embedding vector as semantic points to be organized, and construct a set of semantic points.

[0173] In this embodiment, all V_emb_real_i generated in step S520 and all V_emb_pseudo_j generated in step S530 are merged to form a semantic point set Point_set={V_emb_real_i}∪{V_emb_pseudo_j} to be organized.

[0174] Step S550: Perform semantic point density estimation processing on each semantic point in the semantic point set, and calculate the number of other semantic points contained in the semantic embedding space with the semantic point as the center and a preset radius parameter as the neighborhood range for each semantic point as the neighborhood density value of the semantic point.

[0175] In this embodiment, for each vector V_emb_k in the semantic point set Point_set, its Euclidean distance to all other points is calculated in the semantic embedding space. A radius parameter R_neighbor is set. The number of other points with a distance less than R_neighbor is counted and used as the neighborhood density value Density_k of that semantic point. This neighborhood density value reflects the density of the semantic space surrounding that semantic point.

[0176] Step S560: Sort all semantic points in descending order of neighborhood density value, and select semantic points as semantic prototype anchors starting from the semantic point with the largest neighborhood density value. Semantic points whose semantic similarity to the semantic prototype anchor is greater than a preset similarity threshold are assigned to the semantic prototype cluster corresponding to the semantic prototype anchor.

[0177] In this embodiment, all semantic points are sorted in descending order of their Density_k values. An empty list is initialized to store visited points. The sorted list is traversed, and for each unvisited semantic point, it is selected as a new semantic prototype anchor. The cosine similarity between this anchor and all other unvisited semantic points is calculated. A preset similarity threshold, Sim_th, is used. All points (including the anchor itself) with a cosine similarity greater than Sim_th are grouped into the same semantic prototype cluster, Cluster_anchor, and these points are marked as visited. This process is repeated until all points have been visited.

[0178] Step S570: Perform intra-cluster semantic averaging on all semantic points within each semantic prototype cluster, calculate the mean of the semantic embedding vector of all semantic points within the cluster in each dimension, and generate the semantic prototype vector corresponding to the semantic prototype cluster.

[0179] In this embodiment, for each semantic prototype cluster (Cluster) generated in step S560, the element-wise mean of all semantic point vectors within the cluster is calculated. Assuming there are M semantic points within the cluster, each a 512-dimensional vector, the d-th dimension of the new prototype vector Proto_dim is equal to (1 / M)*sum_{pointinCluster}V_emb_point[d]. The calculated 512-dimensional vector Proto_k is the semantic prototype vector corresponding to that cluster.

[0180] Step S580: Perform spatial distribution optimization processing on the semantic prototype vectors corresponding to all semantic prototype clusters, and adjust the distribution position of the semantic prototype vectors in the semantic embedding space by maximizing the cosine distance between different semantic prototype vectors.

[0181] In this embodiment, the initial semantic prototype vector set {Proto_k} generated in step S570 is obtained. To make the categories more discriminative, the prototype vectors are further optimized. The optimization goal is to maximize the average cosine distance between different prototypes, that is, to disperse the prototypes in the semantic space. This can be achieved through a small optimization process: define a loss function L_dist=-(1 / (K*(K-1)))*sum_{i≠j}cosine_similarity(Proto_i, Proto_j), and use gradient descent to fine-tune the prototype vectors to minimize L_dist, that is, maximize dissimilarity. The optimized prototype vector set is denoted as {Proto_opt_k}.

[0182] Step S590: Combine all semantic prototype vectors after spatial distribution optimization into an open domain category semantic prototype space, where each semantic prototype vector in the open domain category semantic prototype space corresponds to an open domain identifiable target category.

[0183] In this embodiment, the optimized semantic prototype vector set {Proto_opt_k} generated in step S580 is defined as an open and scalable category semantic prototype space Space_open. Each prototype vector Proto_opt_k in this category semantic prototype space represents a specific target category that can be identified from the image, and its semantics are jointly defined by the real category text and pseudo category text that generated the prototype.

[0184] Step S5100: Embed the open domain category semantic prototype space into the classification output layer of the open domain object detection model, replacing the original fixed category classification weight parameters in the classification output layer.

[0185] In this embodiment, the classification output layer of a traditional open-domain object detection model is typically a fully connected layer with a weight matrix W_cls of shape [N_class, D_feat], where N_class is a predefined fixed number of classes. Here, the open-domain category semantic prototype space Space_open generated in step S590 is used to replace this fully connected layer. Specifically, the model's classification output layer no longer contains learnable weights, but instead directly calculates the cosine similarity between the target feature vector output by the detection head and each semantic prototype vector in Space_open, and uses the Softmax function to convert the similarity into a probability distribution. Through this replacement, the model gains the ability to recognize any newly added class, simply by adding the semantic prototype vector of the new class to Space_open.

[0186] Step S610: Add multiple graph convolutional propagation layers to the graph convolutional encoder of the cross-modal feature interaction fusion module to construct a depth graph convolutional propagation layer stack. The depth graph convolutional propagation layer stack includes a first-layer graph convolutional propagation layer, multiple intermediate-layer graph convolutional propagation layers, and a last-layer graph convolutional propagation layer.

[0187] In this embodiment, the graph convolutional encoder described in step S133 is extended to construct a deeper graph convolutional propagation layer stack. This depth stack contains L layers of graph convolutions, where L is set to 8. The first layer is the initial layer, the second to seventh layers are intermediate layers, and the eighth layer is the final layer. Each layer has the same graph convolutional propagation rule: H^{(l+1)}=ReLU(A_hat*H^{(l)}*W^{(l)}), where A_hat is the normalized adjacency matrix, and W^{(l)} is the learnable weight matrix specific to the l-th layer.

[0188] Step S620: Input the cross-modal scene semantic association graph into the first layer of the depth graph convolutional propagation layer stack, and perform neighborhood feature aggregation processing on each graph node in the cross-modal scene semantic association graph through the first layer graph convolutional propagation layer, aggregating the feature information of the first-order neighboring nodes of each graph node to generate a set of hidden representation vectors for the first layer graph nodes.

[0189] In this embodiment, the node feature matrix X (N×D) of the cross-modal scene semantic association graph G_cross is input into the first layer of the depth graph convolutional propagation layer stack. The first layer graph convolutional network aggregates the first-order feature information of all its direct neighbors for each node according to the propagation rule H^{(1)}=ReLU(A_hat*X*W^{(0)}), generating a new hidden representation for each node. The first-layer hidden representations of all nodes constitute the set H1.

[0190] Step S630: Pass the first-layer graph node hidden representation vector set to the first intermediate graph convolutional propagation layer, and perform second-order neighborhood feature aggregation processing on the first-layer graph node hidden representation vector of each graph node through the first intermediate graph convolutional propagation layer to generate a second-order graph node hidden representation vector set.

[0191] In this embodiment, H1 generated in step S620 is passed as input to the second layer graph convolution. The second layer propagates according to H^{(2)}=ReLU(A_hat*H1*W^{(1)}). After this layer, each node aggregates the feature information of its second-order neighbors, because this information has been indirectly contained through the H1 of its first-order neighbors. A set of second-order hidden representations H2 for all nodes is generated.

[0192] Step S640: The second-order graph node hidden representation vector set is sequentially passed to each subsequent intermediate graph convolutional propagation layer. Each intermediate graph convolutional propagation layer performs higher-order neighborhood feature aggregation processing on the received graph node hidden representation vectors to generate a graph node hidden representation vector set of the corresponding order.

[0193] In this embodiment, the H2 generated in step S630 is passed to layers 3, 4, and up to layer 7. In layer 1, the input is H^{(l-1)}, and the output is H^{(l)}=ReLU(A_hat*H^{(l-1)}*W^{(l-1)}). With each layer, the receptive field of each node expands outward by one order. After layer 7, each node aggregates rich contextual information from its 7th-order neighbors.

[0194] Step S650: The last graph convolutional propagation layer in the depth graph convolutional propagation layer stack receives the graph node hidden representation vector set output by the second-to-last intermediate graph convolutional propagation layer, and performs final-order neighborhood feature aggregation processing through the last graph convolutional propagation layer to generate the last graph node hidden representation vector set.

[0195] In this embodiment, the output H7 from the 7th layer is input into the 8th layer (the last layer). The graph convolution in the last layer continues to propagate H8=ReLU(A_hat*H7*W^{(7)}), generating the final order (8th order) set of node hidden representation vectors H8.

[0196] Step S660: Perform cross-layer feature concatenation processing on the first-layer graph node hidden representation vector set, the graph node hidden representation vector sets of all intermediate graph convolutional propagation layers, and the last-layer graph node hidden representation vector set to generate a cross-layer multi-scale graph convolutional feature vector for each graph node.

[0197] In this embodiment, for each graph node, its output vectors H1, H2, H3, H4, H5, H6, H7, and H8 from different network layers are concatenated along the feature dimensions. Each H layer has a dimension of D_h (e.g., 256 dimensions), and after concatenation, the feature dimension of each node becomes 8*D_h. This concatenated vector is called the cross-layer multi-scale graph convolutional feature vector F_deep for that node. This cross-layer connection combines information from shallow (local neighborhood) and deep (global neighborhood) layers, providing a richer feature representation.

[0198] Step S670: Input the cross-layer multi-scale graph convolution feature vector of each graph node into the learnable feature fusion weight generation module, and assign adaptive fusion weight parameters to the graph node hidden representation vectors output by different graph convolution propagation layers through the feature fusion weight generation module.

[0199] In this embodiment, a lightweight feature fusion weight generation module is constructed. It receives the cross-layer multi-scale feature vector F_deep from each node as input, passes it through a sub-network containing two fully connected layers, and outputs a weight vector W_layer = softmax(FC2(ReLU(FC1(F_deep)))). Each element w_l in this weight vector corresponds to the importance weight of the feature in the l-th layer, and the sum of all weights is 1. This feature fusion weight generation module dynamically learns a weight for each node and each layer, rather than using a fixed fusion method.

[0200] Step S680: Perform weighted fusion processing on the cross-layer multi-scale graph convolution feature vector of each graph node according to the adaptive fusion weight parameters to generate the deep context-enhanced graph node feature vector of each graph node.

[0201] In this embodiment, for each node, the weight vector W_layer generated in step S670 is used to perform a weighted fusion of its hidden representations H1 to H8 in each layer. The fusion calculation formula is F_node_final=∑_{l=1}^{8}w_l*H_l. The resulting F_node_final is a fused feature vector with dimension D_h (e.g., 256 dimensions). This fused feature vector integrates information from each layer, and the weights are adaptive to the node.

[0202] Step S690: The deep context-enhanced graph node feature vector is used as the context-enhanced visible light node feature vector and the context-enhanced infrared node feature vector, and input into the cross-attention fusion unit of the cross-modal feature interaction fusion module for subsequent cross-modal feature fusion processing, replacing the original context-enhanced graph node feature vector output by the shallow graph convolutional encoder.

[0203] In this embodiment, the deep context-enhanced graph node feature vector F_node_final (applicable to both visible light and infrared nodes) generated in step S680 is used as a new context feature representation. In steps S136 and S137, the feature vectors output by the shallow graph convolutional encoder were originally used. In this step, the newly generated deep feature vector F_node_final replaces the original feature vector. This deeper, more context-aware node feature will be used to perform element-wise addition and fusion with the visible light / infrared multi-scale feature maps in steps S136 and S137, thereby providing a more powerful scene understanding guide for the entire model.

[0204] Step S6100: Combine the cross-modal feature interaction fusion module containing the depth map convolutional propagation layer stack with the cross-modal target detection head network to form a deep context-aware cross-modal fusion network.

[0205] In this embodiment, the depth map convolutional propagation layer stack implemented in steps S610 to S690, the cross-attention fusion unit in step S138, and the cross-modal target detection head network in step S139 are connected in series to form a complete end-to-end depth context-aware cross-modal fusion network. As the core component of the third cross-modal fusion detection branch in the multi-detection algorithm collaborative integration network, it can more effectively aggregate scene context information of multi-level neighborhoods and improve the performance of open domain target detection by adaptively fusing multi-scale features.

[0206] In one exemplary embodiment, an open-domain target recognition system based on a multimodal large model is provided. This system can be a terminal, server, etc., and its internal structure diagram can be as follows: Figure 3As shown, the system includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of this open-domain target recognition system based on a multimodal large model provides computational and control capabilities. The memory of this system includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of this system is used for exchanging information between the processor and external devices. The communication interface of this system is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, near-field communication, or other technologies. When the computer program is executed by the processor, it implements an open-domain target recognition method based on a multimodal large model. The display unit of this open-domain target recognition system based on a multimodal large model is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of this open-domain target recognition system based on a multimodal large model can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad set on the shell of the open-domain target recognition system based on a multimodal large model, or an external keyboard, touchpad, or mouse, etc.

[0207] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. An open-domain target recognition method based on a multimodal large model, characterized in that, The method includes: Acquire visible light sensing data stream and infrared sensing data stream of the target scene, and acquire an open domain few-shot training set, which includes visible light few-shot training images, infrared few-shot training images, and real bounding box annotation data and real category label data corresponding to the visible light few-shot training images and the infrared few-shot training images. A visible light scene context semantic graph is constructed using the visible light sensing data stream, and an infrared scene context semantic graph is constructed using the infrared sensing data stream. Cross-modal scene graph collaborative discovery processing is performed on the visible light scene context semantic graph and the infrared scene context semantic graph to generate a cross-modal scene semantic association graph. A multi-detection algorithm collaborative integration network is constructed, which includes a first visible light detection branch, a second infrared detection branch, and a third cross-modal fusion detection branch. The cross-modal scene semantic association graph is used as the contextual prior knowledge input of the third cross-modal fusion detection branch. The scene context information in the cross-modal scene semantic association graph guides the multi-detection algorithm collaborative integration network to perform multi-branch collaborative target detection processing. The pre-constructed pseudo-label generation network is trained on the open domain few-sample training set to generate a trained pseudo-label generation network. The trained pseudo-label generation network is then called to generate pseudo-labels and pseudo-boundary boxes on unlabeled open scene images, resulting in a pseudo-category label set and a pseudo-boundary box label set. The open-domain small sample training set, the unlabeled open scene image, the pseudo-class label set, and the pseudo-boundary box label set are combined into an expanded training set. The expanded training set is used to perform cross-modal multi-branch joint training on the multi-detection algorithm collaborative ensemble network to generate a trained open-domain target detection model. The open-domain target detection model is then used to perform open-domain target recognition processing on the target scene, outputting the open-domain category information and open-domain bounding box localization information of the target object.

2. The open-domain target recognition method based on a multimodal large model according to claim 1, characterized in that, A visible light scene context semantic graph is constructed using the visible light sensing data stream, and an infrared scene context semantic graph is constructed using the infrared sensing data stream. Cross-modal scene graph collaborative discovery processing is performed on the visible light scene context semantic graph and the infrared scene context semantic graph to generate a cross-modal scene semantic association graph, including: The visible light sensing data stream is input into the visible light target background separation network. The target region activation module performs response activation processing on the foreground target region in the visible light sensing data stream to generate a visible light target region mask. The background region extraction module performs region segmentation processing on the background region in the visible light sensing data stream to generate a visible light background region mask. Based on the visible light target region mask, the visible light target visual feature vector corresponding to each visible light target region is extracted from the visible light sensing data stream; based on the visible light background region mask, the visible light background visual feature vector corresponding to each visible light background region is extracted from the visible light sensing data stream. Each visible light target region is defined as a visible light target node, and each visible light background region is defined as a visible light background node. The visible light target node stores the corresponding visible light target visual feature vector, and the visible light background node stores the corresponding visible light background visual feature vector. For any two visible light map nodes, perform co-occurrence relationship statistical processing, and calculate the proportion of frames in which the two visible light map nodes appear simultaneously in a continuous frame sequence of the visible light sensing data stream as a co-occurrence frequency statistical parameter. Calculate the spatial distance offset and relative direction angle between the corresponding regions of the two visible light map nodes as spatial position association parameters. The co-occurrence frequency statistics, spatial distance offset, and relative direction angle are normalized to obtain normalized co-occurrence frequency statistics, spatial distance offset, and relative direction angle. Visible light map nodes whose normalized co-occurrence frequency statistics satisfy a preset co-occurrence frequency threshold are connected using co-occurrence relationship connection edges. The normalized co-occurrence frequency statistics, spatial distance offset, and relative direction angle are stored in the corresponding co-occurrence relationship connection edges to generate the visible light scene context semantic graph. An infrared scene context semantic graph is constructed based on the infrared sensing data stream, and cross-modal semantic association processing is performed between the visible light scene context semantic graph and the infrared scene context semantic graph.

3. The open-domain target recognition method based on a multimodal large model according to claim 2, characterized in that, The process of constructing an infrared scene context semantic map based on the infrared sensing data stream and performing cross-modal semantic association processing between the visible light scene context semantic map and the infrared scene context semantic map includes: The infrared sensing data stream is input into the infrared target background separation network. The target region activation module performs response activation processing on the foreground target region in the infrared sensing data stream to generate an infrared target region mask. The background region extraction module performs region segmentation processing on the background region in the infrared sensing data stream to generate an infrared background region mask. Based on the infrared target region mask, extract the infrared target thermal radiation feature vector corresponding to each infrared target region from the infrared sensing data stream; based on the infrared background region mask, extract the infrared background thermal radiation feature vector corresponding to each infrared background region from the infrared sensing data stream. Each infrared target region is defined as an infrared target node, and each infrared background region is defined as an infrared background node. The infrared target node stores the corresponding infrared target thermal radiation feature vector, and the infrared background node stores the corresponding infrared background thermal radiation feature vector. The temperature difference parameter and temperature gradient direction parameter between any two infrared map nodes are calculated. The temperature difference parameter and temperature gradient direction parameter are normalized to obtain normalized temperature difference parameters and temperature gradient direction parameters. Infrared map nodes whose normalized temperature difference parameters meet the preset temperature difference threshold condition are connected using temperature gradient relationship connection edges, and the normalized temperature difference parameter and temperature gradient direction parameter are stored in the corresponding temperature gradient relationship connection edges to generate the infrared scene context semantic graph. For each visible light target node in the visible light scene context semantic map, an infrared map node with a matching spatial location is retrieved in the infrared scene context semantic map. The cross-modal feature similarity calculation module is used to calculate the cross-modal feature similarity measure between the visible light target visual feature vector of the visible light target node and the infrared target thermal radiation feature vector of the infrared map node with a matching spatial location. When the cross-modal feature similarity metric meets the preset cross-modal correspondence threshold condition, a cross-modal semantic correspondence connection edge is established between the visible light target node and the corresponding infrared map node. The cross-modal feature similarity metric is stored in the cross-modal semantic correspondence connection edge. The visible light scene context semantic map, the infrared scene context semantic map, and all cross-modal semantic correspondence connection edges are combined into the cross-modal scene semantic association graph.

4. The open-domain target recognition method based on a multimodal large model according to claim 1, characterized in that, A multi-detection algorithm collaborative integration network is constructed, comprising a first visible light detection branch, a second infrared detection branch, and a third cross-modal fusion detection branch. The cross-modal scene semantic association graph is used as the contextual prior knowledge input for the third cross-modal fusion detection branch. The scene context information in the cross-modal scene semantic association graph guides the multi-detection algorithm collaborative integration network to perform multi-branch collaborative target detection processing, including: The first visible light detection branch of the multi-detection algorithm collaborative integration network is constructed. The first visible light detection branch includes a visible light feature pyramid extraction network and a visible light target detection head network. The visible light feature pyramid extraction network performs multi-layer feature mapping processing on the visible light sensing data stream to generate a set of visible light multi-scale feature maps. The visible light target detection head network performs candidate region generation and category prediction processing on the set of visible light multi-scale feature maps. A second infrared detection branch is constructed in the multi-detection algorithm collaborative integration network. The second infrared detection branch includes an infrared feature pyramid extraction network and an infrared target detection head network. The infrared feature pyramid extraction network performs multi-layer feature mapping processing on the infrared sensing data stream to generate an infrared multi-scale feature map set. The infrared target detection head network performs candidate region generation and category prediction processing on the infrared multi-scale feature map set. A third cross-modal fusion detection branch is constructed in the multi-detection algorithm collaborative integration network. The third cross-modal fusion detection branch includes a cross-modal feature interaction fusion module and a cross-modal target detection head network. The cross-modal feature interaction fusion module receives the visible light multi-scale feature map set and the infrared multi-scale feature map set as input, and simultaneously receives the cross-modal scene semantic association map as contextual prior knowledge input. The cross-modal scene semantic association graph is input into the graph convolutional encoder in the cross-modal feature interaction fusion module. The graph convolutional encoder performs multi-layer graph convolutional message passing processing on the visible light target node, infrared target node, co-occurrence relationship connection edge, temperature gradient relationship connection edge and cross-modal semantic corresponding connection edge in the cross-modal scene semantic association graph to generate a context-enhanced visible light node feature vector for each visible light target node and a context-enhanced infrared node feature vector for each infrared target node. Cross-modal feature fusion processing is performed on the context-enhanced visible light multi-scale feature map set and the context-enhanced infrared multi-scale feature map set to generate multi-branch collaborative detection integration results.

5. The open-domain target recognition method based on a multimodal large model according to claim 4, characterized in that, The process of performing cross-modal feature fusion processing on the context-enhanced visible light multi-scale feature map set and the context-enhanced infrared multi-scale feature map set, and generating a multi-branch collaborative detection ensemble result, includes: Based on the correspondence between the visible light target nodes in the cross-modal scene semantic association graph and the spatial positions in the visible light multi-scale feature map set, the context-enhanced visible light node feature vectors and the visible light multi-scale feature maps at the corresponding spatial positions are fused element-wise to generate a context-enhanced visible light multi-scale feature map set. Based on the correspondence between the infrared target nodes in the cross-modal scene semantic association map and the spatial positions in the infrared multi-scale feature map set, the context-enhanced infrared node feature vectors and the infrared multi-scale feature maps at the corresponding spatial positions are fused element-wise to generate a context-enhanced infrared multi-scale feature map set. The context-enhanced visible light multi-scale feature map set and the context-enhanced infrared multi-scale feature map set are input into the cross-attention fusion unit in the cross-modal feature interaction fusion module. The cross-attention fusion unit calculates the cross-attention weight distribution of the visible light modal features and the infrared modal features at each spatial scale level. The features of the two modalities are weighted and fused according to the cross-attention weight distribution to generate a cross-modal fused feature map set. The cross-modal fusion feature map set is input into the cross-modal target detection head network, and the candidate region generation layer in the cross-modal target detection head network performs multi-scale candidate target region extraction processing on the cross-modal fusion feature map set to generate a cross-modal candidate target region set; The visible light detection results output by the visible light target detection head network of the first visible light detection branch, the infrared detection results output by the infrared target detection head network of the second infrared detection branch, and the cross-modal detection results output by the cross-modal target detection head network of the third cross-modal fusion detection branch are integrated and fused to generate a multi-branch collaborative detection integration result.

6. The open-domain target recognition method based on a multimodal large model according to claim 1, characterized in that, The pre-constructed pseudo-label generation network is trained on the open-domain few-shot training set to improve its few-shot pseudo-labeling capability, generating a trained pseudo-label generation network. This trained network is then used to generate pseudo-labels and pseudo-boundary boxes on unlabeled open scene images, resulting in a pseudo-category label set and a pseudo-boundary box annotation set, including: A pseudo-label generation network is constructed, which includes a shared feature encoder, a region proposal generation sub-network, a pseudo-class prediction sub-network, and a pseudo-bounding box regression sub-network. The shared feature encoder performs feature encoding processing on the input image to generate an input image feature map. The visible light small sample training images and infrared small sample training images in the open domain small sample training set are respectively input into the shared feature encoder. The visible light small sample training images and infrared small sample training images are processed by the multi-level convolutional computation layer group in the shared feature encoder to generate visible light small sample feature map and infrared small sample feature map. The visible light small sample feature map and the infrared small sample feature map are respectively input into the region proposal generation subnetwork. The feature map is then subjected to multi-scale window traversal scanning processing by the sliding window scanning module in the region proposal generation subnetwork to generate a set of visible light candidate proposal regions and a set of infrared candidate proposal regions. The set of visible light candidate proposal regions is input into the pseudo-class prediction sub-network. The multilayer perceptron in the pseudo-class prediction sub-network performs semantic feature encoding and category classification on each visible light candidate proposal region to generate a visible light pseudo-class prediction probability distribution. The visible light pseudo-class prediction loss value is calculated based on the visible light pseudo-class prediction probability distribution and the real category label data in the open domain small sample training set. The set of visible light candidate proposal regions is input into the pseudo bounding box regression subnetwork. The bounding box coordinate offset prediction layer in the pseudo bounding box regression subnetwork performs position offset prediction processing on each visible light candidate proposal region to generate a visible light pseudo bounding box offset prediction value. The visible light pseudo bounding box regression loss value is calculated based on the visible light pseudo bounding box offset prediction value and the real bounding box annotation data in the open domain small sample training set. The infrared candidate proposal regions are subjected to pseudo-label prediction and joint loss optimization to obtain a trained pseudo-label generation network, which is then used to generate a set of pseudo-class labels and a set of pseudo-boundary box labels for unlabeled open scene images.

7. The open-domain target recognition method based on a multimodal large model according to claim 6, characterized in that, The step of performing pseudo-annotation prediction and joint loss optimization on the infrared candidate proposal regions to obtain a trained pseudo-annotation generation network, and using it to generate a set of pseudo-class labels and a set of pseudo-boundary box annotations for unlabeled open scene images, includes: The infrared candidate proposal region set is input into the pseudo-class prediction sub-network. The multilayer perceptron in the pseudo-class prediction sub-network performs semantic feature encoding and category classification on each infrared candidate proposal region to generate an infrared pseudo-class prediction probability distribution. The infrared pseudo-class prediction loss value is calculated based on the infrared pseudo-class prediction probability distribution and the real category label data in the open domain small sample training set. The infrared candidate proposal region set is input into the pseudo bounding box regression subnetwork. The bounding box coordinate offset prediction layer in the pseudo bounding box regression subnetwork performs position offset prediction processing on each infrared candidate proposal region to generate infrared pseudo bounding box offset prediction value. The infrared pseudo bounding box regression loss value is calculated based on the infrared pseudo bounding box offset prediction value and the real bounding box annotation data in the open domain small sample training set. The visible light pseudo-class prediction loss value, the visible light pseudo-boundary box regression loss value, the infrared pseudo-class prediction loss value, and the infrared pseudo-boundary box regression loss value are weighted and summed to generate a joint pseudo-label training loss value. The pseudo-label generation network is then updated with parameters based on the joint pseudo-label training loss value. Training is stopped when the joint pseudo-label training loss value tends to converge, resulting in the trained pseudo-label generation network. An unlabeled open scene image is acquired, which includes an unlabeled visible light image and an unlabeled infrared image. The unlabeled visible light image and the unlabeled infrared image are respectively input into the trained pseudo-label generation network. An unlabeled visible light candidate region set and an unlabeled infrared candidate region set are generated through the shared feature encoder and the region proposal generation sub-network. The unlabeled visible light candidate suggestion region set is input into the pseudo-class prediction subnetwork and pseudo-boundary box regression subnetwork of the trained pseudo-label generation network to generate visible light pseudo-class labels and visible light pseudo-boundary box labels. The unlabeled infrared candidate suggestion region set is input into the pseudo-class prediction subnetwork and pseudo-boundary box regression subnetwork of the trained pseudo-label generation network to generate infrared pseudo-class labels and infrared pseudo-boundary box labels. The visible light pseudo-class labels and the infrared pseudo-class labels are combined into a pseudo-class label set, and the visible light pseudo-boundary box labels and the infrared pseudo-boundary box labels are combined into a pseudo-boundary box label set.

8. The open-domain target recognition method based on a multimodal large model according to claim 1, characterized in that, The open-domain small sample training set, the unlabeled open scene images, the pseudo-class label set, and the pseudo-boundary box label set are combined into an expanded training set. This expanded training set is used to perform cross-modal, multi-branch joint training on the multi-detection algorithm collaborative ensemble network, generating a trained open-domain object detection model. This model is then used to perform open-domain object recognition processing on the target scene, outputting open-domain category information and open-domain bounding box localization information of the target object, including: The visible light small sample training images and infrared small sample training images in the open domain small sample training set are combined with the corresponding real bounding box annotation data and real class label data to construct a real labeled training sample subset; The unlabeled visible light image and unlabeled infrared image in the unlabeled open scene image are combined with the corresponding pseudo-class labels in the pseudo-class label set and the corresponding pseudo-boundary box labels in the pseudo-boundary box label set to construct a subset of pseudo-labeled training samples; The subset of real labeled training samples and the subset of pseudo-labeled training samples are merged to generate the expanded training set. Each training sample in the expanded training set contains input image data, bounding box annotation data, and category label data. The input image data is a visible light image or an infrared image. The visible light training samples in the expanded training set are input into the first visible light detection branch of the multi-detection algorithm collaborative integration network. The visible light feature pyramid extraction network and the visible light target detection head network of the first visible light detection branch generate the category prediction result and bounding box prediction result of the visible light detection branch. The category prediction loss value and bounding box regression loss value of the visible light detection branch are calculated. The infrared training samples in the expanded training set are input into the second infrared detection branch of the multi-detection algorithm collaborative integration network. The infrared feature pyramid extraction network and infrared target detection head network of the second infrared detection branch generate the category prediction result and bounding box prediction result of the infrared detection branch. The category prediction loss value and bounding box regression loss value of the infrared detection branch are calculated. The visible light training samples and infrared training samples in the expanded training set are input in pairs into the third cross-modal fusion detection branch of the multi-detection algorithm collaborative integration network. The cross-modal scene semantic association graph is used as contextual prior knowledge and simultaneously input into the third cross-modal fusion detection branch. The cross-modal feature interaction fusion module and cross-modal target detection head network of the third cross-modal fusion detection branch generate the category prediction result and bounding box prediction result of the cross-modal detection branch. The category prediction loss value and bounding box regression loss value of the cross-modal detection branch are calculated. The multi-detection algorithm collaborative ensemble network is trained with joint loss optimization to generate a trained open-domain target detection model, and the model is used to infer and output the target scene.

9. The open-domain target recognition method based on a multimodal large model according to claim 8, characterized in that, The step of performing joint loss optimization training on the multi-detection algorithm collaborative ensemble network to generate a trained open-domain target detection model, and using this model to infer and output the target scene, includes: The class prediction loss and bounding box regression loss of the visible light detection branch, the class prediction loss and bounding box regression loss of the infrared detection branch, and the class prediction loss and bounding box regression loss of the cross-modal detection branch are weighted and fused to generate the total loss value for multi-branch joint training. Based on the total loss value of the multi-branch joint training, the network parameters of the multi-detection algorithm collaborative integration network are backpropagated and gradient updated. In each training iteration, the network weight parameters of the first visible light detection branch, the second infrared detection branch, and the third cross-modal fusion detection branch are updated synchronously. When the total loss value of the multi-branch joint training meets the preset training convergence condition, the training is stopped, and the trained open domain target detection model is generated. The visible light sensing data stream and infrared sensing data stream of the target scene are input into the trained open domain target detection model. The open domain target detection model outputs the first visible light target category detection result and the first visible light target bounding box localization result through the first visible light detection branch, the second infrared target category detection result and the second infrared target bounding box localization result through the second infrared detection branch, and the third cross-modal fusion detection result and the third cross-modal target category detection result and the third cross-modal target bounding box localization result through the third cross-modal fusion detection branch. The detection results of the first visible light target category, the second infrared target category, and the third cross-modal target category are subjected to weighted voting fusion processing to generate open domain category information for each target object in the target scene. The location results of the first visible light target bounding box, the second infrared target bounding box, and the third cross-modal target bounding box are subjected to weighted average fusion processing to generate open domain bounding box location information for each target object in the target scene.

10. An open-domain target recognition system based on a multimodal large model, characterized in that, include: processor; A machine-readable storage medium for storing machine-executable instructions of the processor; The processor is configured to execute the open-domain target recognition method based on a multimodal large model as described in any one of claims 1 to 9 by executing the machine-executable instructions.

Images