A multi-modal multi-view target detection and matching method and system

Abstract

The application discloses a kind of multimodal multi-view target detection and matching method and system, the present application includes converting text description into text feature, image 1 and image 2 respectively extract image feature 1 and image feature 2, then fusion generates image perception text feature, text perception image feature 1 and text perception image feature 2;Text perception image feature 1, text perception image feature 2 are subjected to target feature extraction and boundary box detection to obtain target coding feature and its boundary box, image perception text feature is combined target coding feature 1 and target coding feature 2 to carry out region-text matching to determine target category, and region-region matching is carried out to target coding feature 1 and target coding feature 2 to identify the same target in image 1 and image 2.The present application aims to realize the target accurate detection and matching of multimodal multi-view.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a multimodal, multi-view target detection and matching method and system. Background Technology

[0002] In the current technological environment, the processing and analysis of multimodal and multi-view data is receiving increasing attention, especially in fields such as drone surveillance, disaster response, and environmental monitoring. These applications typically require accurate identification and matching of targets from images captured by different sensors (such as visible light, infrared, and hyperspectral sensors) and multiple viewpoints. However, existing target detection and matching technologies largely rely on image data from a single modality (mainly visible light) and a limited viewpoint, which restricts their effectiveness in more complex and diverse environments. Although zero-shot learning methods, such as open-world target detection algorithms YOLO-World, GroundingDINO, and GLIP, provide the ability to identify unseen targets, these algorithms are not optimized for target matching tasks in multimodal and multi-view images. Target matching—the ability to identify the same target in different images—is a critical task, but related work remains limited. Furthermore, most of these algorithms are limited to processing and identifying image data with relatively fixed viewpoints, and perform poorly when processing images with large variations in viewpoint due to shooting angles (such as the different pitch angles of a drone).

[0003] The YOLO-World network model is an existing open-vocabulary object detection algorithm that extends the functionality of traditional YOLO (You Only Look Once) detectors by combining vision-language modeling and pre-training on large-scale datasets, enabling these detectors to detect a wide range of object categories even with zero-shot detection. The core of the YOLO-World network model includes a re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN), which is used in conjunction with a region-text contrastive loss function to facilitate the interaction between visual and linguistic information. However, the YOLO-World network model suffers from the following technical challenges: 1. Modal adaptation limitations. The YOLO-World network model is primarily designed for processing visible light images and is not directly adapted to multimodal data such as infrared or hyperspectral images. This limits its effectiveness in applications requiring the fusion of data from multiple sensors for object recognition and matching. 2. Viewpoint dependence. While the YOLO-World network model can handle images from different viewpoints, it is not specifically optimized to adapt to images taken from extreme angles or different pitch angles. This can lead to a decrease in object detection performance when there are significant differences in viewing angles. 3. Lack of matching function. Although the YOLO-World network model performs well in object detection, it is not specifically optimized for object matching tasks in cross-modal and multi-view environments. This means that the algorithm may be inefficient or inaccurate when performing complex object matching tasks, such as ensuring the identification and association of the same object in different images. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal and multi-view target detection and matching method and system to address the above-mentioned problems in the prior art. The present invention aims to achieve accurate target detection and matching in multiple modes and multiple views.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A multimodal, multi-view target detection and matching method includes the following steps: S101, the input text description is converted into text features by a text encoder. For at least one different modality and viewpoint of the input image 1 and image 2, image 1 is extracted by image encoder 1 and image 2 is extracted by image encoder 2. S102, perform feature fusion interaction on text features, image feature 1 and image feature 2 to generate image-aware text features, text-aware image feature 1 and text-aware image feature 2 respectively; S103, target feature extraction and bounding box detection are performed on text-aware image feature 1 to obtain target encoding feature 1 and its bounding box. Target feature extraction and bounding box detection are performed on text-aware image feature 2 to obtain target encoding feature 2 and its bounding box. The image-aware text features are combined with target encoding feature 1 and target encoding feature 2 to perform region-text matching to determine the target category. Region-region matching is performed on target encoding feature 1 and target encoding feature 2 to identify the same target in image 1 and image 2.

[0006] Optionally, in step S101, images 1 and 2 are modally different and have the same or different viewing angles.

[0007] Optionally, the different modes specifically refer to Image 1 being a visible light image and Image 2 being an infrared image.

[0008] Optionally, in step S102, the feature fusion interaction of text features, image feature 1, and image feature 2 is implemented through a deep feature fusion interaction module, the function expression of which is: , , , , , , , In the above formula, This refers to image feature 1 that is perceived based on image feature 2; This represents image feature 2 that is perceived by image feature 1; , , They represent Queries, key-value matrices; Indicates image feature 1; Dimensions representing image features; , , They represent Queries, key-value matrices; Image feature 2; This represents the original image features and the features resulting from the fusion of mutually perceptive image features; This represents the weights used in feature fusion. , , These represent the text query, key, and value matrix, respectively. , , They represent Queries, key-value matrices; This indicates text features that possess image fusion perception capabilities; This indicates fused image features with text awareness; express and Features after weighted fusion; express and Features after weighted fusion; and These represent the weights during feature fusion; and the text features possessing image fusion feature awareness... Text features perceived for the output image. and Features after weighted fusion Image features 1 for outputting text-aware data. and Features after weighted fusion Image features for text perception 2.

[0009] Optionally, when performing target feature extraction and bounding box detection in step S103, the target feature extraction is to obtain target encoded feature 1 or target encoded feature 2 through a target feature extraction module composed of two stacked 3×3 convolutional layers, and the bounding box detection refers to using a fully connected layer to regress the input target encoded feature 1 or target encoded feature 2 to obtain the corresponding bounding box.

[0010] Optionally, the image-aware text features, target encoding feature 1, and target encoding feature 2 constitute a target set consisting of multiple targets with the same feature dimension. The targets in the image-aware text features, target encoding feature 1, and target encoding feature 2 have a one-to-one correspondence. Step S103, which combines the image-aware text features with target encoding feature 1 and target encoding feature 2 to perform region-text matching to determine the target category, includes: calculating the similarity between target pairs in the target sets of the image-aware text features and target encoding feature 1, respectively; and determining the similarity of each target in the target set of the image-aware text features within the target set of target encoding feature 1 based on the principle of maximum similarity. The targets in the target set are: the similarity between the target pairs in the target sets of image-aware text features and target coding features 2, respectively; and the target corresponding to each target in the target set of image-aware text features in the target set of target coding features 2, based on the principle of the highest similarity. The process of performing region-region matching on target coding features 1 and target coding features 2 to identify the same targets in images 1 and 2 includes calculating the similarity between the target pairs in the target sets of target coding features 1 and target coding features 2, respectively, and the target corresponding to each target in the target set of target coding features 1 in the target set of target coding features 2, based on the principle of the highest similarity.

[0011] Optionally, before step S101, a step of training a multimodal, multi-view object detection and matching network model consisting of image encoder 1, image encoder 2, deep feature fusion interaction module, target feature extraction module, and fully connected layer is included, and the loss function used in training the multimodal, multi-view object detection and matching network model is expressed as follows: , In the above formula, For loss function, , and These are the weights for each loss. For classifying losses, To detect and locate the loss, This is the cross-modal target matching loss.

[0012] Furthermore, the present invention also provides a multimodal, multi-view target detection and matching system, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the multimodal, multi-view target detection and matching method.

[0013] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program / instructions that are programmed or configured to execute the multimodal multi-view target detection and matching method by a processor.

[0014] In addition, the present invention also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the multimodal multi-view target detection and matching method by a processor.

[0015] Compared with existing technologies, the present invention has the following main advantages: The method of the present invention can achieve accurate target detection and matching in multiple modalities and multiple perspectives, can be used in a wider range of application scenarios, enhances its practicality and adaptability, solves the performance degradation problem of traditional target detection technology when processing images from extreme perspectives or different pitch angles, significantly improves the matching accuracy under multiple perspective conditions, and effectively reduces the dependence of target detection technology on perspective, improves the challenges encountered by existing technologies when dealing with diverse perspectives, ensures the stability and reliability of the system when facing images captured from various angles, and performs matching on different regions in the image so that it can identify and match the same or similar targets between different images, greatly expanding the application scope and effect of target matching. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the principle of the method in an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of visible light vehicle labeling in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of vehicle detection results in an embodiment of the present invention. Detailed Implementation

[0019] The multimodal and multi-view target detection and matching method of the present invention can realize multimodal and multi-view target detection and matching. The following will take the full fusion of text information, visible light image and infrared image to realize target detection in visible light and infrared image, and the matching of the same entity in different modalities through detection region matching algorithm as an example to further explain the multimodal and multi-view target detection and matching method of the present invention.

[0020] like Figure 1 As shown, the multimodal, multi-view target detection and matching method in this embodiment includes the following steps: S101, the input text description is converted into text features by a text encoder. For at least one different modality and viewpoint of the input image 1 and image 2, image 1 is extracted by image encoder 1 and image 2 is extracted by image encoder 2. S102, perform feature fusion interaction on text features, image feature 1 and image feature 2 to generate image-aware text features, text-aware image feature 1 and text-aware image feature 2 respectively; S103, target feature extraction and bounding box detection are performed on text-aware image feature 1 to obtain target encoding feature 1 and its bounding box. Target feature extraction and bounding box detection are performed on text-aware image feature 2 to obtain target encoding feature 2 and its bounding box. The image-aware text features are combined with target encoding feature 1 and target encoding feature 2 to perform region-text matching to determine the target category. Region-region matching is performed on target encoding feature 1 and target encoding feature 2 to identify the same target in image 1 and image 2.

[0021] As an optional implementation, images 1 and 2 in step S101 may have different modalities and either the same or different viewing angles. Specifically, in this embodiment, different modalities mean that image 1 is a visible light image and image 2 is an infrared image. During training, images 1 and 2 are randomly selected from a multimodal, multi-view dataset, and may be of different modalities, different viewing angles, or both. However, it should be noted that the method of this embodiment is applicable to images 1 and 2 with at least one different modality and viewing angle, regardless of the specific example provided.

[0022] In step S101 of this embodiment, three modal encoders are used: a text encoder (TextEncoder), a visible light visual encoder (RGB Encoder, image encoder 1), and an infrared visual encoder (InfraredEncoder, image encoder 2). These encoders are all known encoders, and pre-trained encoders or custom-trained encoders can be used as needed. Given a text input T, the text encoder (TextEncoder) produces a text feature vector W as the text feature. Given a visible light image (image 1), the RGB encoder (visible light visual encoder) produces a multi-level visible light feature vector X (image feature 1). Given an infrared image, the RGB encoder (Infrared Encoder) produces a multi-level infrared feature vector I (image feature 1).

[0023] In step S102 of this embodiment, the feature fusion interaction of text features, image feature 1, and image feature 2 is achieved through a deep feature fusion interaction module. In step S101 of this embodiment, a multimodal feature fusion module is used to perform feature fusion interaction of text features, image feature 1, and image feature 2, generating image-aware text features, text-aware image feature 1, and text-aware image feature 2, respectively. Specifically, the text feature vector W, the multi-level visible light feature vector X (image feature 1), and the multi-level infrared feature vector I (image feature 1) are input into the multimodal feature fusion module for full fusion between modalities. First, this embodiment designs a visible light to infrared cross-attention mechanism (X2I cross-attention) and an infrared to visible light cross-attention mechanism (I2X cross-attention). These two attention mechanisms realize the fusion of visual features from two different modalities. The specific operation details are as follows: in the X2I cross-attention module, combined with the transformer architecture, q=X, k=v=I, thereby enabling infrared information to be perceived in the visible light features. Similarly, in the I2X cross-attention module, let q=I and k=v=X, so that the infrared features contain visible light information, thereby realizing visual modal interaction. After obtaining the visual modal features with sufficient interaction, it is necessary to interact with the text features and visual features to construct a text-image cross-attention mechanism. After multi-level fusion of text information and visual information, we can obtain text features with image perception, visible light visual features with text perception, and infrared visual features with text perception. Through the above method, we can obtain features of various fully fused modalities. Specifically, the function expression of the deep feature fusion interaction module in this embodiment is: , , , , , , , In the above formula, This refers to image feature 1 that is perceived based on image feature 2; This represents image feature 2 that is perceived by image feature 1; , , They represent Queries, key-value matrices; Indicates image feature 1; Dimensions representing image features; , , They represent Queries, key-value matrices; Image feature 2; This represents the original image features and the features resulting from the fusion of mutually perceptive image features; This represents the weights used in feature fusion. , , These represent the text query, key, and value matrix, respectively. , , They represent Queries, key-value matrices; This indicates text features that possess image fusion perception capabilities; This indicates fused image features with text awareness; express and Features after weighted fusion; express and Features after weighted fusion; and These represent the weights during feature fusion; and the text features possessing image fusion feature awareness... Text features perceived for the output image. and Features after weighted fusion Image features 1 for outputting text-aware data. and Features after weighted fusion Image features for text perception 2. Given the input text description "a white car driving", the text feature vector can be obtained after encoding by the Text Encoder. Where C represents the number of nouns in the text, and D represents the dimension of the feature vector. Given a visible light image input and an infrared image input, after passing through the feature encoders of their respective modalities, multi-scale cascaded features {X1, X2, X3} of the visible light image and multi-scale cascaded features {I1, I2, I3} of the infrared image are obtained. The interaction of the two modal visual features is achieved through a cross-attention mechanism at different cascade levels, thereby realizing mutual perception between visible light and infrared, and a weighted attention mechanism is used to fuse the two modalities. The visual features after fusing the text features are obtained through a bidirectional cross-attention mechanism to obtain visually perceptive text features and text-perceptive visual features. The text-perceptive visual features are then fused with the original modal data through feature fusion.

[0024] In step S103 of this embodiment, when performing target feature extraction and bounding box detection, the target feature extraction is performed by obtaining target encoded feature 1 or target encoded feature 2 through a target feature extraction module composed of two stacked 3×3 convolutional layers. In this embodiment, target encoded feature 1 can be described as: , Target encoding feature 2 can be described as: , Where K represents the number of targets. The two visual features are processed using the Hungarian matching algorithm to obtain the maximum similarity feature matching of targets in the modal feature encoding sequence. In this embodiment, bounding box detection refers to using a fully connected layer to regress the input target encoding feature 1 or target encoding feature 2 to obtain the corresponding bounding box. After obtaining text-aware visual features, visible light and infrared features are processed by the target feature extraction module to obtain visible light target features and infrared target features, respectively. 1) To match target features, the Hungarian algorithm is used to match the features of the two modalities. After the matching algorithm, the correspondence between the feature vectors of the two modalities can be obtained. 2) The corresponding feature vectors in the two modalities are processed through a feature fusion method to obtain a fused visual and infrared feature vector. Then, the fused feature vector is compared with the text features using a similarity matching method similar to CLIP to classify the target feature. 3) The feature vectors of the two modalities are mapped through a fully connected layer to obtain the position of the target feature.

[0025] The fused visual feature code is formed by fusing the features that match the most between the two modal feature codes. The feature encoding S is compared with the text features using contrastive learning to obtain the type. The visible light target feature encoding M and the infrared target feature encoding N are used through a regression layer to obtain the position of the target in their respective modalities. In this embodiment, the image-perceived text features, target encoding feature 1, and target encoding feature 2 are a target set composed of multiple targets with the same feature dimension. The targets in the image-perceived text features, target encoding feature 1, and target encoding feature 2 have a one-to-one correspondence. In step S103, the image-perceived text features are combined with target encoding feature 1 and target encoding feature 2 to perform region-text matching to determine the target category. This includes calculating the similarity between target pairs in the target sets of the image-perceived text features and target encoding feature 1, respectively, and determining the target category of each target in the target set of the image-perceived text features based on the principle of the highest similarity. The method involves calculating the similarity between target pairs in both the image-perceived text features and target encoding features 2, and determining the target corresponding to each target in the image-perceived text features target set in the target encoding features 2 based on the principle of maximum similarity. The region-to-region matching of target encoding features 1 and target encoding features 2 to identify the same targets in images 1 and 2 includes calculating the similarity between target pairs in both target encoding features 1 and target encoding features 2, and determining the target corresponding to each target in the target encoding features 1 target set in the target encoding features 2 target set based on the principle of maximum similarity. Based on the text description, the model of this embodiment can obtain the position and classification of targets in visible light images, the position and classification of targets in infrared images, and the mapping relationship between targets in the two modalities, realizing object detection and matching under multimodal and multi-view conditions.

[0026] To achieve multimodal and multi-view target detection and matching, the loss function designed in this embodiment should include classification loss for two modalities, location detection loss for two modalities, and matching loss between modalities. The classification loss is achieved through contrastive learning loss, the location detection loss through mean squared error loss, the modal matching loss through [missing information], and the matching of targets between modalities through triplet loss. Thus, the multimodal and multi-view target detection and matching network model of this embodiment integrates information from multiple modalities, uses a cross-attention mechanism and text-visual alignment, and achieves cross-modal and cross-view target detection and recognition. Specifically, before step S101, a step of training the multimodal and multi-view target detection and matching network model composed of image encoder 1, image encoder 2, deep feature fusion interaction module, target feature extraction module, and fully connected layers is included. The loss function used when training the multimodal and multi-view target detection and matching network model has the following expression: , In the above formula, For loss function, , and These are the weights for each loss. For classifying losses, To detect and locate the loss, The cross-modal target matching loss is used. The classification loss, detection and localization loss, and cross-modal target matching loss are all well-known loss functions. For example, for the detection and localization loss, see: Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 779-788). The calculation of the cross-modal target matching loss can be found in: Schroff, F., Kalenichenko, D., & Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 815-823).

[0027] To verify the multimodal, multi-view target detection and matching method of this embodiment, it was validated on a server platform with Ubuntu 20.04, PyTorch 2.1, and CUDA 12.1 versions, using an Nvidia V100 32G graphics card and an i9-14900 CPU. To achieve multimodal and multi-view target detection and matching based on this embodiment, a multimodal, multi-view vehicle dataset was constructed, as detailed below. Figure 2 and Figure 3As shown in the figure, the dataset was primarily collected from parking lots and traffic roads using drones equipped with visible light and infrared sensors. The dataset contains images in both visible light and infrared modalities. To achieve multi-view and multi-modal capabilities, visible light and infrared images were acquired by drones at a flight altitude of 110 meters and a camera pitch angle of 90 degrees, and at a flight altitude of 80 meters and a camera pitch angle of 60 degrees, respectively. The dataset includes four categories: car, bus, truck, and van. Each target is labeled with its color and status, resulting in labels such as "a white car in motion." All targets in this dataset have a unique ID number, which enables unique identification of targets across multiple modalities and viewpoints, thus facilitating cross-modal and cross-viewpoint matching. The entity detection results (evaluation metric: mAP) of this embodiment in both visible light and infrared modalities are shown in Table 1.

[0028] Table 1. Detection results of the method in this embodiment on two modalities.

[0029] The matching results of entities in the two modalities of this embodiment are shown in Table 2.

[0030] Table 2 shows the entity matching results in the two modalities of the method in this embodiment.

[0031] In Table 2, Rank-1 represents the probability of a correct search among the top 1 targets with the highest confidence level retrieved for the current target; Rank-5 represents the probability of a correct search among the top 5 targets with the highest confidence level retrieved for the current target; and Rank-10 represents the probability of a correct search among the top 10 targets with the highest confidence level retrieved for the current target. The above results indicate that the model of the multimodal multi-view target detection and matching method in this embodiment possesses cross-modal detection and matching capabilities. The model of the multimodal multi-view target detection and matching method in this embodiment has the ability to detect corresponding targets in both modalities, and its detection performance in the visible light modality is better than its performance in the infrared modality. The model of the multimodal multi-view target detection and matching method in this embodiment has the ability to match the same target in both modalities. At Rank-1, the matching accuracy of the model in this embodiment is 83.1%, meaning that the correct matching rate for the retrieved target with the highest confidence level is 83.1%.

[0032] In summary, with the rapid development of automation technology and intelligent systems, especially in fields such as surveillance, autonomous driving, and remote sensing, the demand for accurately identifying and matching targets in images from different sensors and perspectives is increasing. Existing zero-shot target detection technologies, including state-of-the-art open-world target detection technologies, while making significant progress in identifying unseen object categories, primarily focus on processing single-modal data and are mainly concerned with target identification rather than matching them across different images. Traditional target matching techniques typically rely on hand-designed features or features extracted through deep learning, using these features to perform search and matching tasks in a database. However, the application of these methods in multimodal scenarios (such as combining visible light, infrared, and hyperspectral images) is severely limited. Furthermore, when targets are captured from multiple perspectives, traditional techniques often fail to effectively identify and match the same target because perspective changes significantly affect feature representation and similarity calculation. Against this backdrop, the motivation for the multimodal, multi-view target detection and matching method in this embodiment is to develop a technology capable of effectively matching targets across modalities and perspectives. This embodiment of the multimodal, multi-view target detection and matching method not only needs to process and fuse data from different sensors, but also must be able to process images captured from different angles and accurately match targets in these images. To achieve multimodal data integration, this embodiment focuses on developing an efficient data processing framework capable of integrating image data from different sensors such as visible light, infrared, and hyperspectral sensors. Through this integration, the system can be used in a wider range of application scenarios, enhancing its practicality and adaptability. To achieve multi-view matching optimization and address the performance degradation problem of traditional target detection techniques when processing images from extreme viewpoints or different pitch angles, this embodiment proposes an innovative loss function for multi-view matching. This loss function is specifically designed to optimize the extraction and matching process of image features, significantly improving matching accuracy under multi-view conditions and effectively reducing the dependence of target detection technology on viewpoint. In this way, this patent not only enhances the performance of target matching but also indirectly improves the challenges encountered by existing technologies in handling diverse viewpoints, ensuring the stability and reliability of the system when facing images captured from various angles. Furthermore, inspired by the CLIP and GLIP models, this embodiment of the multimodal, multi-view target detection and matching method proposes an innovative region-to-region matching approach. Unlike traditional text-to-image matching, this method specifically targets different regions within an image, enabling it to identify and match identical or similar targets across different images, significantly expanding the application scope and effectiveness of target matching.

[0033] Furthermore, this embodiment also provides a multimodal, multi-view target detection and matching system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the multimodal, multi-view target detection and matching method.

[0034] In addition, this embodiment also provides a computer-readable storage medium storing a computer program / instructions that are programmed or configured to execute the multimodal multi-view target detection and matching method by a processor.

[0035] In addition, this embodiment also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the multimodal multi-view target detection and matching method by a processor.

[0036] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0037] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal, multi-view target detection and matching method, characterized in that, Includes the following steps: S101, the input text description is converted into text features by a text encoder. For at least one different modality and viewpoint of the input image 1 and image 2, image 1 is extracted by image encoder 1 and image 2 is extracted by image encoder 2. S102, perform feature fusion interaction on text features, image feature 1 and image feature 2 to generate image-aware text features, text-aware image feature 1 and text-aware image feature 2 respectively; S103, target feature extraction and bounding box detection are performed on text-aware image feature 1 to obtain target encoding feature 1 and its bounding box. Target feature extraction and bounding box detection are performed on text-aware image feature 2 to obtain target encoding feature 2 and its bounding box. The image-aware text features are combined with target encoding feature 1 and target encoding feature 2 to perform region-text matching to determine the target category. Region-region matching is performed on target encoding feature 1 and target encoding feature 2 to identify the same target in image 1 and image 2. The image-perceived text features, target encoding feature 1, and target encoding feature 2 constitute a target set consisting of multiple targets with the same feature dimension. There is a one-to-one correspondence between the targets in the image-perceived text features, target encoding feature 1, and target encoding feature 2. Step S103, which combines the image-perceived text features with target encoding feature 1 and target encoding feature 2 to perform region-text matching to determine the target category, includes: calculating the similarity between target pairs in the target sets of both the image-perceived text features and target encoding feature 1; and determining the target category of each target in the target set of target encoding feature 1 based on the principle of maximum similarity. The corresponding targets are: calculating the similarity of target pairs in the target sets of image-perceived text features and target encoding features 2, respectively, and determining the target corresponding to each target in the target set of image-perceived text features in the target set of target encoding features 2 based on the principle of maximum similarity; the region-region matching for target encoding features 1 and target encoding features 2 to identify the same targets in image 1 and image 2 includes calculating the similarity of target pairs in the target sets of target encoding features 1 and target encoding features 2, respectively, and determining the target corresponding to each target in the target set of target encoding features 1 in the target set of target encoding features 2 based on the principle of maximum similarity.

2. The multimodal, multi-view target detection and matching method according to claim 1, characterized in that, In step S101, images 1 and 2 are modally different and have the same or different viewing angles.

3. The multimodal, multi-view target detection and matching method according to claim 2, characterized in that, The different modes specifically refer to Image 1 being a visible light image and Image 2 being an infrared image.

4. The multimodal, multi-view target detection and matching method according to claim 1, characterized in that, In step S102, the feature fusion interaction of text features, image feature 1, and image feature 2 is implemented through a deep feature fusion interaction module. The function expression of the deep feature fusion interaction module is as follows: ， ， ， ， ， ， ， In the above formula, This refers to image feature 1 that is perceived based on image feature 2; This represents image feature 2 that is perceived by image feature 1; , , They represent Queries, key-value matrices; Indicates image feature 1; Dimensions representing image features; , , They represent Queries, key-value matrices; Indicates image feature 2; This represents the original image features and the features resulting from the fusion of mutually perceptive image features; This represents the weights used in feature fusion. , , These represent the text query, key, and value matrix, respectively. , , They represent Queries, key-value matrices; This indicates text features that possess image fusion perception capabilities; This indicates fused image features with text awareness; express and Features after weighted fusion; express and Features after weighted fusion; and These represent the weights during feature fusion; and the text features possessing image fusion feature awareness... Text features perceived for the output image. and Features after weighted fusion Image features 1 for outputting text-aware data. and Features after weighted fusion Image features for text perception 2.

5. The multimodal, multi-view target detection and matching method according to claim 4, characterized in that, In step S103, when performing target feature extraction and bounding box detection, the target feature extraction is to obtain target encoded feature 1 or target encoded feature 2 through a target feature extraction module composed of two stacked 3×3 convolutional layers, and the bounding box detection refers to using a fully connected layer to regress the input target encoded feature 1 or target encoded feature 2 to obtain the corresponding bounding box.

6. The multimodal, multi-view target detection and matching method according to claim 1, characterized in that, Before step S101, there is also a step of training a multimodal, multi-view object detection and matching network model consisting of image encoder 1, image encoder 2, deep feature fusion interaction module, target feature extraction module, and fully connected layer. The loss function used in training the multimodal, multi-view object detection and matching network model has the following expression: ， In the above formula, For loss function, , and These are the weights for each loss. For classifying losses, To detect and locate the loss, This is the cross-modal target matching loss.

7. A multimodal, multi-view target detection and matching system, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the multimodal, multi-view target detection and matching method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program / instructions, characterized in that, The computer program / instructions are programmed or configured to execute the multimodal, multi-view target detection and matching method according to any one of claims 1 to 6 via a processor.

9. A computer program product comprising a computer program / instructions, characterized in that, The computer program / instructions are programmed or configured to execute the multimodal, multi-view target detection and matching method according to any one of claims 1 to 6 via a processor.

Images