A multimodal target detection method, device, terminal and storage medium

Abstract

The present invention discloses a multimodal target detection method, device, terminal and storage medium in the field of target detection technology, aiming to solve the problems of existing technologies, such as reliance on labeled data, insufficient diversity of visual backbones, and high training costs. The method comprises the following steps: acquiring a target image; inputting the target image into a pre-trained multimodal target detection model, outputting positioning and classification prediction results, and completing the detection; the present invention introduces the UNet network as a visual backbone network in the target detection task for the first time, breaking through the limitations of traditional supervised and unsupervised methods, and providing a new feature representation method for the target detection task; the present invention utilizes the self-supervised learning ability of the multimodal target detection model, reduces the reliance on a large amount of labeled data, reduces training costs, and improves the applicability of the model in data-scarce scenarios.

Description

Technical Field

[0001] The present invention relates to a multimodal target detection method, device, terminal and storage medium, and belongs to the technical field of target detection. Background Art

[0002] Existing disaster relief equipment is becoming increasingly intelligent and unmanned, replacing humans with equipment. Unmanned equipment requires the replacement of the human eye for target identification and positioning. Current intelligent rescue equipment must address the following challenges: how to quickly and accurately locate targets in complex post-disaster environments (where targets are irregularly shaped and obstructed), as well as how to identify unknown targets, both of which must be addressed by target detection models. The task of target detection is to predict a bounding box and a set of classification labels for a given image. Modern detectors extract image features using a visual backbone trained in either a supervised or unsupervised manner. Based on this, they design a detection head using either a first-stage anchor-free approach, a second-stage anchor-based approach, or an end-to-end approach based on transformers (a deep learning model architecture used for natural language processing and other sequence-to-sequence tasks) to identify objects of interest.

[0003] Existing technologies for object detection primarily rely on supervised and unsupervised visual backbone networks, such as ResNet101 (a 101-layer deep convolutional neural network) and MAE (a self-supervised pre-training framework for visual representation learning). While these methods have achieved remarkable results in object detection, they still have some technical limitations.

[0004] Supervised learning methods, such as ResNet101, require large amounts of precisely labeled training data to optimize the model. However, obtaining high-quality labeled data is costly and time-consuming. Especially in object detection tasks, labeling bounding boxes and category information requires extensive manual intervention. This reliance on labeled data limits the model's application in data-scarce scenarios and increases training costs.

[0005] While unsupervised learning methods, such as MAE, reduce reliance on labeled data, their representational capabilities are often limited by the design of the pre-training task. For example, MAE learns image features through a mask reconstruction task, but this task may not fully capture the fine-grained semantic information relevant to object detection. Consequently, unsupervised learning methods often perform worse than supervised learning methods in object detection tasks.

[0006] Existing methods for object detection primarily focus on discriminative and reconstructive models, with limited exploration of generative models. Generative models, such as diffusion models, possess powerful data generation and implicit knowledge learning capabilities, enabling them to learn rich visual features from large amounts of unlabeled data. However, existing techniques fail to fully exploit the potential of generative models, limiting the diversity of visual backbone networks and preventing them from providing more generalizable feature representations for object detection.

[0007] In summary, existing target detection methods usually have problems such as reliance on labeled data, insufficient diversity of visual backbones, and high training costs, which affect the performance and practicality of target detection. Summary of the Invention

[0008] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a multimodal target detection method, device, terminal and storage medium. For the first time, the UNet network is introduced as a visual backbone network in the target detection task, breaking through the limitations of traditional supervised and unsupervised methods, and providing a new feature representation method for the target detection task; the present invention combines the semantic information in the text-to-image task with the target detection task through the implicit knowledge learning mechanism of the UNet network, significantly improving the performance of the model in complex scenarios; the present invention utilizes the self-supervised learning ability of the multimodal target detection model, reduces the dependence on a large amount of labeled data, reduces the training cost, and at the same time improves the applicability of the model in data-scarce scenarios.

[0009] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0010] In a first aspect, the present invention provides a multimodal target detection method, comprising the following steps:

[0011] Acquire the target image;

[0012] Input the target image into a pre-trained multimodal target detection model, output the positioning and classification prediction results, and complete the detection; the multimodal target detection model includes a CLIP encoder, a VAE encoder, a UNet network, and a detection head;

[0013] The detection process of the target image by the multimodal target detection model specifically includes:

[0014] Input the target image into the CLIP encoder to generate textual prompt embeddings;

[0015] Input the target image into the VAE encoder to generate a latent feature map;

[0016] Embed the latent feature map and text prompt into the UNet network to obtain a multi-scale diffusion feature map;

[0017] Calculate the cross attention calculation results of the text prompt embedding and the latent feature map in the UNet network;

[0018] According to the cross-attention calculation result map and the multi-scale diffusion feature map, the channel dimension is spliced ​​to form an enhanced feature map;

[0019] The enhanced feature map is input into the detection head, and the positioning and classification prediction results are output.

[0020] Furthermore, the training of the multimodal target detection model specifically includes:

[0021] Obtain an image collection and a multimodal object detection model;

[0022] Randomly sample a set of random images from the image collection;

[0023] Feed a random image into the CLIP encoder to generate a textual prompt embedding;

[0024] Input a random image into the VAE encoder to generate a latent feature map;

[0025] Embed the latent feature map and text prompt into the UNet network to obtain a multi-scale diffusion feature map;

[0026] Calculate the cross attention calculation results of the text prompt embedding and the latent feature map in the UNet network;

[0027] According to the cross-attention calculation result map and the multi-scale diffusion feature map, the channel dimension is spliced ​​to form an enhanced feature map;

[0028] The enhanced feature map is input into the detection head, and the positioning and classification prediction results are output;

[0029] According to the output positioning and classification prediction results, the total loss function is calculated, and the parameters of the multimodal target detection model are updated according to the gradient backpropagation to complete the training of the multimodal target detection model.

[0030] Furthermore, the target image is input into the CLIP encoder to generate a text prompt embedding, specifically including:

[0031] Input the target image into the CLIP encoder to obtain the corresponding image features;

[0032] The corresponding image features are passed through two linear layers to obtain the mapped text hint embedding.

[0033] Furthermore, the channel dimension splicing of the cross-attention calculation result map and the multi-scale diffusion feature map to form an enhanced feature map specifically includes:

[0034] The second and third layers of the cross-attention calculation result map are spliced ​​with the third and fourth layers of the multi-scale diffusion feature map along the channel dimension to form an enhanced feature map.

[0035] Furthermore, the enhanced feature map is input into the detection head to output the positioning and classification prediction results, specifically including:

[0036] The enhanced feature map is input into the FPN module of the detection head to perform multi-scale feature fusion to obtain the fused feature map;

[0037] The fused feature map is input into the classification module and positioning module of the detection head, and the positioning and classification prediction results are output.

[0038] Furthermore, the total loss function is calculated by the following formula:

[0039] ;

[0040] Where: is the total loss function; Generate network submodules for the detection head region; It is the region of interest submodule of the detection head; is the classification loss of RPN; is the positioning loss of rpn; is the classification loss of roi; is the positioning loss of roi;

[0041] Furthermore, the classification loss of the roi is calculated by the following formula:

[0042] ;

[0043] Where: N is the total number of images in a group of images; n is the image sequence number in a group of images; C is the total number of categories in the target detection dataset; c is the category sequence number; is the classification prediction result of the image with image number n and category number c in a group of images; for The annotation information;

[0044] The classification loss of the RPN is calculated by the following formula:

[0045] ;

[0046] Where: is the number of foreground and background categories of the target detection dataset;

[0047] The positioning loss of the RPN is calculated by the following formula:

[0048] ;

[0049] ;

[0050] Where: M is The total number of regions processed, m is The total number of regions in the total number of regions processed; for The target positioning prediction result of the image with the total number of regions being processed being m; for The annotation information; is the smoothing error loss;

[0051] The positioning loss of the roi is calculated by the following formula:

[0052] ;

[0053] ;

[0054] Where: is the total number of regions processed by roi, It is the total number of regions in the total number of regions processed by roi; The total number of regions processed by roi is Target positioning prediction results of the image; for The annotation information.

[0055] In a second aspect, the present invention provides a multimodal target detection device, comprising:

[0056] Input module: used to obtain target image;

[0057] Detection module: used to input the target image into the pre-trained multimodal target detection model, output positioning and classification prediction results, and complete the detection;

[0058] The multimodal target detection model includes a CLIP encoder, a VAE encoder, a UNet network, and a detection head. The detection process of the target image by the detection module specifically includes:

[0059] Scene text prompt generation module: used to input the target image into the CLIP encoder and generate text prompt embedding;

[0060] Image encoding module: used to input the target image into the VAE encoder to generate a potential feature map;

[0061] Diffusion feature extraction module: used to embed the latent feature map and text prompt into the UNet network to obtain multi-scale diffusion feature maps;

[0062] Cross-attention map fusion module: used to calculate the cross-attention calculation result map of the text prompt embedding and the potential feature map in the UNet network; according to the channel dimension splicing of the cross-attention calculation result map and the multi-scale diffusion feature map, an enhanced feature map is formed;

[0063] Object detection module: used to input the enhanced feature map into the detection head and output positioning and classification prediction results.

[0064] In a third aspect, the present invention provides a terminal including a processor and a storage medium;

[0065] The storage medium is used to store instructions;

[0066] The processor is configured to operate according to the instructions to execute the steps of the method according to any one of the first aspects.

[0067] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] This multimodal target detection method introduces the UNet network as a visual backbone network in the target detection task for the first time, breaking through the limitations of traditional supervised and unsupervised methods; traditional methods are limited by the dependence on labeled data or pre-training task design, while the present invention uses the rich implicit knowledge learned by the UNet network to provide a new, more diverse and generalized feature representation method for target detection tasks, opening up a new technical route in the field of target detection; the present invention uses the self-supervised learning ability of the multimodal target detection model to reduce the dependence on large amounts of labeled data, reduce training costs, and at the same time improve the applicability of the model in data-scarce scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flowchart of a multimodal target detection method provided according to an embodiment of the present invention;

[0071] Figure 2 1 is a schematic diagram of a specific process of a multimodal target detection method provided according to an embodiment of the present invention;

[0072] Figure 3 3 is a structural diagram of a multimodal target detection device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0074] The term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects.

[0075] Example 1:

[0076] like Figures 1 to 2 As shown, the present invention provides a multimodal target detection method, comprising the following steps:

[0077] Acquire the target image;

[0078] Input the target image into a pre-trained multimodal target detection model, output the positioning and classification prediction results, and complete the detection; the multimodal target detection model includes a CLIP encoder, a VAE encoder, a UNet network, and a detection head;

[0079] The detection process of the target image by the multimodal target detection model specifically includes:

[0080] The target image is fed into the CLIP encoder (Contrastive Language Image Pretrained Encoder) to generate a textual cue embedding, which consists of:

[0081] Input the target image into the CLIP encoder to obtain the corresponding image features;

[0082] Pass the corresponding image features through two linear layers to obtain the mapped text hint embedding;

[0083] Input the target image into the VAE encoder (Variational Autoencoder) to generate a latent feature map;

[0084] Embedding the latent feature map and textual cues into a UNet (a convolutional neural network architecture designed for image segmentation) network to obtain a multi-scale diffusion feature map; wherein the UNet network is based on a text-to-image diffusion model UNet;

[0085] Calculate the cross attention calculation results of the text prompt embedding and the latent feature map in the UNet network;

[0086] The enhanced feature map is formed by splicing the cross-attention calculation result map and the multi-scale diffusion feature map in the channel dimension, including:

[0087] The second and third layers of the cross attention calculation result map are spliced ​​with the third and fourth layers of the multi-scale diffusion feature map along the channel dimension to form an enhanced feature map;

[0088] The enhanced feature map is input into the detection head, and the positioning and classification prediction results are output, including:

[0089] The enhanced feature map is input into the FPN (Feature Pyramid Network) module of the detection head to perform multi-scale feature fusion and obtain the fused feature map;

[0090] The fused feature map is input into the classification module and positioning module of the detection head, and the positioning and classification prediction results are output.

[0091] This paper introduces a generative paradigm visual backbone network, combines semantic text prompts with a position attention enhancement mechanism, and solves the problems of existing target detection methods relying on labeled data and lacking visual backbone diversity, thereby improving the generalization ability and performance of target detection. Specifically, Figure 2 The VQGAN encoder (discrete encoder) in is a form of VAE encoder.

[0092] Specifically, such as Figure 2As shown, the regional perception feature is SAM (a pre-trained model for image segmentation) which extracts the segmentation mask features of the image through the visual Transformer architecture. It can accurately capture the outline and spatial position of any object and is suitable for zero-sample segmentation tasks. The fully connected layer is the basic component of the neural network. The multi-layer perceptron performs high-dimensional mapping of the input features by stacking linear layers and nonlinear activation functions. It is often used in classification heads or feature fusion. It has a large number of parameters but high flexibility and can learn complex nonlinear relationships. The latent feature is in the potential diffusion model. The image is compressed into a low-dimensional latent variable through the VAE or VQGAN encoder. The diffusion process (noise addition and denoising) is carried out in the latent space, which significantly reduces the computational cost while retaining the generation quality. After the latent feature is decoded, a high-fidelity image is output. The cross-attention layer is in the text-to-image generation task. The UNet network introduces a cross-attention mechanism, uses text embedding as the key value, and the image feature as the query, dynamically generates spatial attention weights, and realizes fine-grained control of text semantics on local areas of the image. The region proposal network is Faster The core component of R-CNN generates anchor points on the feature map through a sliding window, predicts the target probability and bounding box offset of each anchor point, screens out high-quality candidate regions, and provides preliminary target positioning for the two-stage detector; the region of interest pooling is for the candidate boxes of different sizes output by the region proposal network of Faster R-CNN, and normalizes them into feature blocks of fixed size (such as 7×7) through bilinear interpolation or maximum pooling, retaining spatial information while adapting to the input requirements of the subsequent fully connected layer. It is a key operation for feature alignment in target detection.

[0093] Based on the text-to-image diffusion model framework, this paper innovatively proposes a method for applying a generative paradigm visual backbone network to target detection tasks. This method fully utilizes the rich implicit knowledge learned by the diffusion model during the text-to-image training process, possesses excellent visual perception capabilities, and can provide a new generative paradigm-based visual backbone network for target detection tasks.

[0094] Specifically, the present invention combines the semantic information in the text-to-image task with the target detection task through the implicit knowledge learning mechanism of the diffusion model, significantly improving the performance of the model in complex scenarios; the present invention utilizes the self-supervised learning ability of the multimodal target detection model (generative model) to reduce the dependence on a large amount of labeled data, reduce the training cost, and improve the applicability of the model in data-scarce scenarios; through the above innovations, the present invention provides a more diverse, generalizable and cost-effective solution for target detection tasks.

[0095] This paper introduces the UNet network (based on the text-to-image diffusion model UNet) as the visual backbone network for object detection for the first time, overcoming the limitations of traditional supervised (such as ResNet) and unsupervised (such as MAE) methods. Traditional methods are limited by their reliance on labeled data or pre-training task design. This paper, however, leverages the rich implicit knowledge learned by the UNet network (its deep understanding of object structure, texture, and scene relationships) to provide a new, more diverse, and more generalizable feature representation for object detection. This opens up a new technical path for the field of object detection.

[0096] In one embodiment, the training of the multimodal object detection model specifically includes:

[0097] Obtain an image collection and a multimodal object detection model;

[0098] Randomly sample a set of random images from the image collection;

[0099] Feed a random image into the CLIP encoder to generate a textual prompt embedding;

[0100] Input a random image into the VAE encoder to generate a latent feature map;

[0101] Embed the latent feature map and text prompt into the UNet network to obtain a multi-scale diffusion feature map;

[0102] Calculate the cross attention calculation results of the text prompt embedding and the latent feature map in the UNet network;

[0103] According to the cross-attention calculation result map and the multi-scale diffusion feature map, the channel dimension is spliced ​​to form an enhanced feature map;

[0104] The enhanced feature map is input into the detection head, and the positioning and classification prediction results are output;

[0105] According to the output positioning and classification prediction results, the total loss function is calculated, and the parameters of the multimodal target detection model are updated according to the gradient backpropagation to complete the training of the multimodal target detection model.

[0106] Specifically, the training of the multimodal target detection model includes:

[0107] Randomly sample a set of random images from the image collection ;

[0108] Random images Input into the CLIP image encoder to obtain the corresponding image features , whose size is ; Then, Through two linear layers, we get the mapped scene text prompts ;

[0109] The specific expression is as follows:

[0110] ; ;

[0111] in, Indicates the total number of images in the current group; ch indicates the number of channels; H indicates height; W indicates width; represents a multilayer perceptron;

[0112] The image Input into the VAE encoder to get the potential feature head ;

[0113] Then, and At the same time, it is input into the denoising UNet network of the text-to-image diffusion model to obtain a five-layer multi-scale diffusion feature map ; The size of the feature map of each layer is 、 、 、 ;

[0114] The specific expression is as follows:

[0115] ; ;

[0116] in, is the number of layers of the multi-scale diffusion feature map in the UNet network; is a multi-scale diffusion feature map;

[0117] Calculation results of cross attention between text hint embedding and latent feature map in UNet network ; The size of the feature map of each layer is 、 、 、 ; Select i=2, 3 layers and =3, 4 layers, and Perform channel-level merging operations and replace the results in the original multi-scale diffusion feature map. =3, 4 features, and the enhanced feature map is obtained , whose sizes are 、 、 、 、 ;

[0118] The specific expression is as follows:

[0119] ;

[0120] , ;

[0121] in, is the number of layers of the cross attention calculation result map; k is the number of layers of the enhanced feature map.

[0122] Diffusion feature Input into the FPN module, perform multi-scale feature fusion, and obtain the fused feature map ; Specifically, 1×1 convolution is used to adjust the number of channels of the fused feature maps of each layer so that their number of channels is consistent with 256; from the smallest size feature map Initially, the feature maps of each layer are fused with the feature maps of the previous layer by upsampling (using bilinear interpolation), and the high-level features of the top layer are retained to better utilize the semantic information; where p is the number of layers of the fused feature map;

[0123] The fused feature map Input into the classification module and positioning module of the detection head to output positioning and classification prediction results;

[0124] According to the output positioning and classification prediction results, the total loss function is calculated, and the parameters of the multimodal target detection model are updated according to the gradient backpropagation. After one round of training is completed, the training work is restarted or the training of the multimodal target detection model is ended.

[0125] In one embodiment, the total loss function is calculated by the following formula:

[0126] ;

[0127] Where: is the total loss function; Generate network submodules for the detection head region; It is the region of interest submodule of the detection head; is the classification loss of RPN; is the positioning loss of rpn; is the classification loss of roi; is the positioning loss of roi;

[0128] The classification loss of the roi is calculated by the following formula:

[0129] ;

[0130] Where: N is the total number of images in a group of images; n is the image sequence number in a group of images; C is the total number of categories in the target detection dataset; c is the category sequence number; is the classification prediction result of the image with image number n and category number c in a group of images; for The annotation information;

[0131] The classification loss of the RPN is calculated by the following formula:

[0132] ;

[0133] Where: is the number of foreground and background categories of the target detection dataset;

[0134] The positioning loss of the RPN is calculated by the following formula:

[0135] ;

[0136] ;

[0137] Where: M is The total number of regions processed, m is The total number of regions in the total number of regions processed; for The target positioning prediction result of the image with the total number of regions being processed being m; for The annotation information; is the smoothing error loss;

[0138] The positioning loss of the roi is calculated by the following formula:

[0139] ;

[0140] ;

[0141] Where: is the total number of regions processed by roi, It is the total number of regions in the total number of regions processed by roi; The total number of regions processed by roi is Target positioning prediction results of the image; for The annotation information.

[0142] In one embodiment, the network proposed in this application was trained and tested on the Day Sunny sub-dataset of the Diverse-dataset autonomous driving dataset. The AP metric was used to evaluate each class, and the mAP metric was used for overall evaluation. The comparison results with mainstream visual backbone network methods are shown in the following table:

[0143] Table 1. Evaluation results of Diverse-dataset dataset;

[0144]

[0145] In the table, Ours refers to the present invention;

[0146] The text-to-image diffusion model proposed in this invention, as a new visual backbone network method, significantly outperforms classic visual backbone networks such as Resnet50 (a 50-layer version of a deep convolutional neural network), Resnet101 and CLIP, and also achieves performance comparable to the current most advanced method Convnext (a modern convolutional neural network architecture), proving that the visual backbone network based on the generative paradigm (i.e., the text-to-image diffusion model UNet) can be used for target detection tasks.

[0147] This paper creatively combines the semantic information in the text-to-image generation task (embedded text prompts generated by CLIP) with the target detection task (localization and classification) through the implicit knowledge learning mechanism of the UNet, a text-to-image diffusion model. The core of this paper is to use the cross-attention mechanism within the UNet, a text-to-image diffusion model, to calculate the spatial correlation map between text semantics and image latent feature maps. This attention map accurately indicates the spatial position of concepts in the text description in the image. By splicing and fusing this attention map with the multi-scale visual features extracted by the diffusion model itself in the channel dimension, the model's ability to understand complex scenes (such as irregular target shapes, occlusions, background interference, etc.) is significantly enhanced, thereby improving the model's performance in complex scenes.

[0148] Example 2:

[0149] like Figure 3 As shown, the present invention provides a multimodal target detection device, comprising:

[0150] Input module: used to obtain target image;

[0151] Detection module: used to input the target image into the pre-trained multimodal target detection model, output positioning and classification prediction results, and complete the detection;

[0152] The multimodal target detection model includes a CLIP encoder, a VAE encoder, a UNet network, and a detection head. The detection process of the target image by the detection module specifically includes:

[0153] Scene text prompt generation module: used to input the target image into the CLIP encoder and generate text prompt embedding;

[0154] Image encoding module: used to input the target image into the VAE encoder to generate a potential feature map;

[0155] Diffusion feature extraction module: used to embed the latent feature map and text prompt into the UNet network to obtain multi-scale diffusion feature maps;

[0156] Cross-attention map fusion module: used to calculate the cross-attention calculation result map of the text prompt embedding and the potential feature map in the UNet network; according to the channel dimension splicing of the cross-attention calculation result map and the multi-scale diffusion feature map, an enhanced feature map is formed;

[0157] Object detection module: used to input the enhanced feature map into the detection head and output positioning and classification prediction results.

[0158] Specifically, the image encoding module compresses the original image into low-dimensional latent features, which are then fed into the diffusion feature extraction module. After the input image (size 448×800 pixels) is spatially compressed through the VAE encoder, a latent feature map (size 64×112×4) is generated. The VAE encoder pre-trains weights based on the Stable Diffusion model and freezes the parameters to avoid updating during training.

[0159] Scene Text Hint Generation Module: Generates text embeddings semantically aligned with the input image to enhance the feature expression capabilities of the diffusion model. After inputting an image (size 448×800 pixels), the CLIP image encoder extracts a global feature vector (dimension 768). The feature vector is then reduced to the same dimension as the diffusion model text embedding (dimension 768) through a two-layer MLP (multi-layer perceptron) to obtain the text hint embedding. Utilizing the image-text alignment feature of CLIP, visual features are converted into semantic text conditions, enhancing the high-level semantic perception capabilities of the diffusion model.

[0160] Diffusion Feature Extraction Module: Extracts multi-scale visual features from the UNet network of the diffusion model; is used to embed the latent feature map and text prompts into the UNet network to obtain a multi-scale diffusion feature map; the UNet network is pre-trained based on the LAION-4B dataset (a large-scale public dataset containing approximately 4 billion image-text pairs), and the parameters are frozen to preserve the generation capability.

[0161] Cross-attention graph fusion module: Captures the positional association between text and image through the cross-modal attention mechanism to enhance the position sensitivity of features; used to calculate the cross-attention calculation result graph (size 32×56×768) of text prompt embedding and potential feature map in the UNet network; based on the channel dimension splicing of the cross-attention calculation result graph and the multi-scale diffusion feature map, it forms an enhanced feature map (size 32×56×1024); introduces low-level position information to improve the positioning accuracy of target detection.

[0162] Object detection module: This module performs object classification and bounding box regression based on the enhanced feature maps. The enhanced feature maps are input into the Faster R-CNN (a deep learning model for object detection) detection head to generate the final detection results. During training, only the detection head parameters are optimized, and other modules are frozen to reduce computational costs.

[0163] Example 3:

[0164] An embodiment of the present invention further provides a terminal, including a processor and a storage medium;

[0165] The storage medium is used to store instructions;

[0166] The processor is configured to operate according to the instructions to execute the steps of the method described in embodiment 1.

[0167] Example 4:

[0168] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the method described in the first embodiment are implemented.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0173] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal target detection method, characterized in that: The following steps are involved: Acquire the target image; Input the target image into a pre-trained multimodal target detection model, output the positioning and classification prediction results, and complete the detection; the multimodal target detection model includes a CLIP encoder, a VAE encoder, a UNet network, and a detection head; The detection process of the target image by the multimodal target detection model specifically includes: Input the target image into the CLIP encoder to generate textual prompt embeddings; Input the target image into the VAE encoder to generate a latent feature map; Embed the latent feature map and text prompt into the UNet network to obtain a multi-scale diffusion feature map; Calculate the cross attention calculation results of the text prompt embedding and the latent feature map in the UNet network; According to the cross-attention calculation result map and the multi-scale diffusion feature map, the channel dimension is spliced ​​to form an enhanced feature map; The enhanced feature map is input into the detection head, and the positioning and classification prediction results are output; The target image is input into the CLIP encoder to generate text prompt embedding, specifically including: Input the target image into the CLIP encoder to obtain the corresponding image features; Pass the corresponding image features through two linear layers to obtain the mapped text hint embedding; The channel-dimensional splicing of the cross-attention calculation result map and the multi-scale diffusion feature map to form an enhanced feature map specifically includes: The second and third layers of the cross attention calculation result map are spliced ​​with the third and fourth layers of the multi-scale diffusion feature map along the channel dimension to form an enhanced feature map; The enhanced feature map is input into the detection head, and the positioning and classification prediction results are output, specifically including: The enhanced feature map is input into the FPN module of the detection head to perform multi-scale feature fusion to obtain the fused feature map; The fused feature map is input into the classification module and positioning module of the detection head, and the positioning and classification prediction results are output.

2. The multimodal target detection method according to claim 1, wherein: The training of the multimodal target detection model specifically includes: Obtain an image collection and a multimodal object detection model; Randomly sample a set of random images from the image collection; Feed a random image into the CLIP encoder to generate a textual prompt embedding; Input a random image into the VAE encoder to generate a latent feature map; Embed the latent feature map and text prompt into the UNet network to obtain a multi-scale diffusion feature map; Calculate the cross attention calculation results of the text prompt embedding and the latent feature map in the UNet network; According to the cross-attention calculation result map and the multi-scale diffusion feature map, the channel dimension is spliced ​​to form an enhanced feature map; The enhanced feature map is input into the detection head, and the positioning and classification prediction results are output; According to the output positioning and classification prediction results, the total loss function is calculated, and the parameters of the multimodal target detection model are updated according to the gradient backpropagation to complete the training of the multimodal target detection model.

3. The multimodal target detection method according to claim 2, wherein: The total loss function is calculated by the following formula: ； Where: is the total loss function; Generate network submodules for the detection head region; It is the region of interest submodule of the detection head; is the classification loss of RPN; is the positioning loss of rpn; is the classification loss of roi; is the positioning loss of roi.

4. The multimodal target detection method according to claim 3, wherein: The classification loss of the roi is calculated by the following formula: ； Where: N is the total number of images in a group of images; n is the image serial number in a group of images; C is the total number of categories in the target detection dataset; c is the category serial number; is the classification prediction result of the image with image number n and category number c in a group of images; for The annotation information; The classification loss of the RPN is calculated by the following formula: ； Where: is the number of foreground and background categories of the target detection dataset; The positioning loss of the RPN is calculated by the following formula: ； ； Where: M is The total number of regions processed, m is The total number of regions in the total number of regions processed; for The target positioning prediction result of the image with the total number of regions being processed being m; for The annotation information; is the smoothing error loss; The positioning loss of the roi is calculated by the following formula: ； ； Where: is the total number of regions processed by roi, It is the total number of regions in the total number of regions processed by roi; The total number of regions processed by roi is Target positioning prediction results of the image; for The annotation information.

5. A multimodal target detection device, based on the multimodal target detection method according to any one of claims 1 to 4, characterized in that: include: Input module: used to obtain target image; Detection module: used to input the target image into the pre-trained multimodal target detection model, output positioning and classification prediction results, and complete the detection; The multimodal target detection model includes a CLIP encoder, a VAE encoder, a UNet network, and a detection head. The detection module performs the following steps on the target image: Scene text prompt generation module: used to input the target image into the CLIP encoder and generate text prompt embedding; Image encoding module: used to input the target image into the VAE encoder to generate a potential feature map; Diffusion feature extraction module: used to embed the latent feature map and text prompt into the UNet network to obtain multi-scale diffusion feature maps; Cross-attention map fusion module: used to calculate the cross-attention calculation result map of the text prompt embedding and the potential feature map in the UNet network; according to the channel dimension splicing of the cross-attention calculation result map and the multi-scale diffusion feature map, an enhanced feature map is formed; Object detection module: used to input the enhanced feature map into the detection head and output positioning and classification prediction results.

6. A terminal, characterized in that: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images