Open vocabulary object detection method and device based on cross-modal feature fusion

Abstract

An open vocabulary object detection method and device based on cross-modal feature fusion add a deep feature fusion method in multiple stages of the model, realize the interaction function between picture and text feature information, learn high-quality language perception visual representation and realize excellent transfer learning performance, so as to enhance the generalization ability of the open vocabulary object detection model, thereby improving the detection ability of the model for unknown categories. In the method, the feature fusion module enhances the connection between the modes to realize the picture and text feature information interaction function, so as to generate more recognizable feature expression, thereby improving the accuracy of recognizing unknown categories.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to an open vocabulary object detection method, apparatus, device, and computer-readable storage medium based on cross-modal feature fusion. Background Technology

[0002] Open-vocabulary object detection is a challenging task. This method mimics the human ability to connect visual and linguistic information, forming a rich visual and semantic vocabulary for object detection. It can learn not only image information from the dataset but also text title information, extracting hidden information about unknown categories from the text and using this information for unknown category detection. This type of method generally involves three computational steps: extracting image and text features, fusing and aligning image and text features using a visual-language model, and finally outputting the aligned result. These three steps enable the model to learn multimodal information from images and text and utilize this information in the task of detecting unknown categories.

[0003] Most current open-vocabulary object detection methods utilize pre-trained visual language models to generalize from detecting known categories to detecting unknown categories. Visual language models are obtained through pre-training on large-scale image-text datasets, thus exhibiting significant generalization performance across various visual tasks. They align images and language vocabularies to the same feature space, bridging the gap between visual data and language vision, and effectively eliminating the differences between closed and open sets. However, a key issue arises when using pre-trained visual language models in open-vocabulary object detection tasks: visual language models learn the association between images and their image-level text descriptions, failing to accurately link textual concepts with image regions to achieve text-image alignment.

[0004] Therefore, how to adapt visual language models to region-level object detection tasks in open-vocabulary object detection tasks, and how to apply this knowledge to generalize from known categories to unknown categories, aligning textual conceptual information with image region information, so that models trained on known category training samples can better generalize to unknown category object detection, is a pressing problem. In open-vocabulary object detection tasks, reducing the distributional differences between visual and textual features and improving the model's understanding of both visual and linguistic aspects is a challenging problem. Therefore, researching methods to improve cross-modal feature fusion capabilities and enhance the model's understanding of textual information in open-vocabulary object detection is of great significance for this task. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application proposes an open vocabulary object detection method, apparatus, device, and computer-readable storage medium based on cross-modal feature fusion. The aim is to solve the problem in open vocabulary object detection methods of accurately aligning text concepts with image region features, thereby enhancing the ability to learn high-quality language-aware visual representation information, improving the model's cross-modal feature fusion performance, and enhancing the model's ability to detect unknown categories.

[0006] In a first aspect, embodiments of this application provide an open vocabulary target detection method based on cross-modal feature fusion, the open vocabulary target detection method based on cross-modal feature fusion comprising:

[0007] Add images containing a specific target to the image collection. In this context, the location of a specific target within its image and its category label are known information.

[0008] With image collection The region classifier module M1 is trained using the training set to obtain region cues containing the trained regions. M1';

[0009] For image collection any one of the pictures Extract global image features , the picture Input M1' to obtain the region classification result output by M1', then input the region classification result into the CLIP text encoder, and finally pass it through a fully connected layer for dimensionality upscaling to obtain the image. Textual feature information ,Will and Input the feature fusion module and obtain the fused image information output by the feature fusion module. ,Will The encoder of DAB-DETR is fed into the DAB-DETR encoder, and the decoder, which uses the region classification result as a query, is fed into DAB-DETR to obtain the object detection result. Based on the object detection result and known information, the image is calculated. The corresponding second type of loss; comprehensive image set The second type loss corresponding to all images in M2 is used to train the feature fusion module, the encoder of DAB-DETR, and the decoder of DAB-DETR until the stopping condition is met, and the trained M2' is obtained.

[0010] Target detection task using M1' and M2'.

[0011] In conjunction with the first aspect, in one implementation, the feature fusion module includes a self-attention component, a text-to-image cross-attention component, and an image-to-text cross-attention component.

[0012] In conjunction with the first aspect, in one implementation, global image features Extracted using the first three layers of the CLIP image encoder.

[0013] In conjunction with the first aspect, in one implementation, a picture set is used. The region classifier module M1 is trained using the training set to obtain region cues containing the trained regions. M1' includes:

[0014] For image collection any one of the pictures Based on specific targets in the image Regional features are extracted from the location in the image to obtain the regional image features. ,exist Add a region hint The region embedding results are obtained. ,calculate With regional text feature set The similarity of text features in each region is used to determine the predicted category label of a specific target based on the text feature corresponding to the maximum similarity. The image is then calculated by combining the category label of the specific target. The corresponding first type of loss involves processing each category label in the image dataset using a region description language, then feeding it into the CLIP text encoder to obtain the region text features corresponding to each category label, and finally adding them to the region text feature set. Comprehensive Image Collection Type I loss pairs for all images in the dataset provide region hints. Training continues until the stopping condition is met, at which point a prompt containing the trained region is obtained. M1'.

[0015] In conjunction with the first aspect, in one implementation, based on a specific target in the image Regional features are extracted from the location in the image to obtain the regional image features. include:

[0016] Based on specific objectives in images The location within the map is combined with RoI Pooling technology to obtain the region of interest map;

[0017] Feature extraction of the region of interest map is performed using the fourth layer of the CLIP image encoder to obtain the region image features. .

[0018] In conjunction with the first aspect, in one implementation, performing a target detection task using M1' and M2' includes:

[0019] Target category detection is performed using M1';

[0020] Target location detection is completed using M2'.

[0021] Secondly, embodiments of this application provide an open vocabulary target detection device based on cross-modal feature fusion, the open vocabulary target detection device based on cross-modal feature fusion comprising:

[0022] Builder modules are used to add images containing a specific target to an image set. In this context, the location of a specific target within its image and its category label are known information.

[0023] The first training module is used to train the image set. The region classifier module M1 is trained using the training set to obtain region cues containing the trained regions. M1';

[0024] The second training module is used for training the image set. any one of the pictures Extract global image features , the picture Input M1' to obtain the region classification result output by M1', then input the region classification result into the CLIP text encoder, and finally pass it through a fully connected layer for dimensionality upscaling to obtain the image. Textual feature information ,Will and Input the feature fusion module and obtain the fused image information output by the feature fusion module. ,Will The encoder of DAB-DETR is fed into the DAB-DETR encoder, and the decoder, which uses the region classification result as a query, is fed into DAB-DETR to obtain the object detection result. Based on the object detection result and known information, the image is calculated. The corresponding second type of loss; comprehensive image set The second type loss corresponding to all images in M2 is used to train the feature fusion module, the encoder of DAB-DETR, and the decoder of DAB-DETR until the stopping condition is met, and the trained M2' is obtained.

[0025] The detection module is used to perform target detection tasks using M1' and M2'.

[0026] In conjunction with the second aspect, in one implementation, the feature fusion module includes a self-attention component, a text-to-image cross-attention component, and an image-to-text cross-attention component.

[0027] Thirdly, embodiments of this application provide an open vocabulary target detection device based on cross-modal feature fusion. The open vocabulary target detection device based on cross-modal feature fusion includes a processor, a memory, and an open vocabulary target detection program based on cross-modal feature fusion stored in the memory and executable by the processor. When the open vocabulary target detection program based on cross-modal feature fusion is executed by the processor, it implements the steps of the open vocabulary target detection method based on cross-modal feature fusion as described in the first aspect.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium storing an open vocabulary target detection program based on cross-modal feature fusion, wherein when the open vocabulary target detection program based on cross-modal feature fusion is executed by a processor, it implements the steps of the open vocabulary target detection method based on cross-modal feature fusion as described in the first aspect.

[0029] The beneficial effects of the technical solutions provided in this application include:

[0030] This application addresses the problem of inaccurate alignment between text features and image region features in open vocabulary target detection tasks. It proposes an open vocabulary target detection method based on cross-modal feature fusion to improve the ability to fuse cross-modal feature information, enhance the performance of the detection model in learning high-quality language perception and visual representation information, and thus improve the model's generalization ability. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating an embodiment of the open vocabulary target detection method based on cross-modal feature fusion according to this application;

[0032] Figure 2 This is a schematic diagram of the model framework of an embodiment of the open vocabulary target detection method based on cross-modal feature fusion in this application;

[0033] Figure 3 This is a structural diagram of the feature fusion module in one embodiment of the open vocabulary target detection method based on cross-modal feature fusion in this application;

[0034] Figure 4 This is a schematic diagram of the cross-attention calculation process in one embodiment of the open vocabulary target detection method based on cross-modal feature fusion in this application;

[0035] Figure 5This is a schematic diagram of the functional modules of an embodiment of the open vocabulary target detection device based on cross-modal feature fusion according to this application;

[0036] Figure 6 This is a schematic diagram of the hardware structure of the open vocabulary target detection device based on cross-modal feature fusion involved in the embodiments of this application. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0039] In a first aspect, embodiments of this application provide an open vocabulary target detection method based on cross-modal feature fusion.

[0040] In one embodiment, reference is made to Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an embodiment of the open vocabulary target detection method based on cross-modal feature fusion proposed in this application. Figure 2 This is a schematic diagram of the model framework of an embodiment of the open vocabulary object detection method based on cross-modal feature fusion proposed in this application. Combined with... Figure 1 and Figure 2 This clarifies the processing logic of the embodiments in this application. For example... Figure 1 As shown, open vocabulary target detection methods based on cross-modal feature fusion include:

[0041] S10, Add images containing specific targets to the image set. In this context, the location of a specific target within its image and its category label are known information.

[0042] The image dataset is divided into a first number of known category labels and a second number of unknown category labels. Targets with known category labels are considered as specific targets, and images containing specific targets are added to the image set. Among them, the first quantity is greater than the second quantity, and the category label of a specific target and its position in the corresponding image are known information;

[0043] In this embodiment, a specific target refers to a target whose category label is known. Images containing specific targets are selected from the image dataset and added to the image set. The image dataset uses the COCO dataset as an example, dividing its 80 category labels into 48 known category labels and 17 unknown category labels. Based on this, targets with known category labels are considered as specific targets, and images containing these specific targets are added to the image set. Furthermore, the category label of a specific target and its location in the corresponding image are known information.

[0044] S20, with photo album The region classifier module M1 is trained using the training set to obtain region cues containing the trained regions. M1';

[0045] S20 includes:

[0046] For image collection any one of the pictures Based on specific targets in the image Regional features are extracted from the location in the image to obtain the regional image features. ,exist Add a region hint The region embedding results are obtained. ,calculate With regional text feature set The similarity of text features in each region is used to determine the predicted category label of a specific target based on the text feature corresponding to the maximum similarity. The image is then calculated by combining the category label of the specific target. The corresponding first type of loss involves processing each category label in the image dataset using a region description language, then feeding it into the CLIP text encoder to obtain the region text features corresponding to each category label, and finally adding them to the region text feature set. Comprehensive Image Collection Type I loss pairs for all images in the dataset provide region hints. Training continues until the stopping condition is met, at which point a prompt containing the trained region is obtained. M1';

[0047] Furthermore, in one embodiment, based on a specific target in the image Regional features are extracted from the location in the image to obtain the regional image features. include:

[0048] Based on specific objectives in images The location within the map is combined with RoI Pooling technology to obtain the region of interest map;

[0049] Feature extraction of the region of interest map is performed using the fourth layer of the CLIP image encoder to obtain the region image features. .

[0050] In this embodiment, the specific target is in the image. Since the location of the target is known, RoI Pooling technology can be used to obtain the region of interest map, i.e., the region map containing the specific target. Then, the fourth layer of the CLIP image encoder is used to extract features from the region of interest map, thereby obtaining the region image features. .

[0051] All category labels in the COCO dataset are collectively referred to as a category concept pool, such as "person," "dog," and "cat." This pool contains both known and unknown category labels. These labels undergo region description language processing. For example, the category label "person" is transformed into the phrase "A photo of a person," used to describe a target area. This processed data is then fed into the CLIP text encoder to obtain the region text features corresponding to the "person" category label. This process is repeated for each category label to obtain its corresponding region text features, which are then added to a region text feature set. .

[0052] Add a region hint The result obtained With regional text feature set Calculate the similarity of text features in each region, for example If the text features of the region corresponding to the category label "person" have the highest similarity, then it can be determined that... The predicted category label for the specific target is "person", and the true category label for the specific target is known. Therefore, the first type of loss can be calculated based on the predicted category label and the true label.

[0053] Each image can be processed in the above manner to obtain the Type I loss for each image. The combined Type I losses of all images can then be used to provide region suggestions. Training continues until the stopping condition is met, at which point a prompt containing the trained region is obtained. M1'.

[0054] By using learnable area prompts To enhance regional features and correct misalignment information, regional hints Region hints are trained on the locations of targets with known class labels in the training set and then trained on a detection dataset with known class annotations. The region features obtained by fine-tuning the CLIP image encoder are demonstrated experimentally to be effective on known image sets. This also works for unknown categories. During training, the weights of other models are frozen, and hints are only given to the regions that need to be learned. Partial training is performed, and once training is complete, the trained areas are highlighted. As a pre-trained model, parameters are frozen for use in training the weights of other models. This region suggestion method enhances region features, corrects misalignments, and reduces the distribution gap between the overall image feature mapping and region features through learnable region suggestions. Specifically, it relates to region image features. The final region embedding result is obtained by performing an addition operation. :

[0055]

[0056] in This represents element addition. It is the last attention pooling module of the CLIP visual encoder. The sums are added first and then passed through the attention pooling module.

[0057] Obtain the region embedding result Then, it is combined with the regional text feature set. Cosine similarity matching is performed on the text features of each region in the training. To make the matched category labels more accurate, training region hints are used. Make the area prompt The region classifier module M1 can accurately identify the category label of the target in the region, and after training, a module M1' containing trained region cues is obtained.

[0058] S30, with photo album The target detection model M2 is trained using the training set to obtain the trained M2';

[0059] S30 includes:

[0060] For image collection any one of the pictures Extract global image features , the image Input M1' to obtain the region classification result output by M1', then input the region classification result into the CLIP text encoder, and finally pass it through a fully connected layer for dimensionality upscaling to obtain the image. Textual feature information ,Will and Input the feature fusion module and obtain the fused image information output by the feature fusion module. ,Will The encoder of DAB-DETR is fed into the DAB-DETR encoder, and the decoder, which uses the region classification result as a query, is fed into DAB-DETR to obtain the object detection result. Based on the object detection result and known information, the image is calculated. The corresponding second type of loss; comprehensive image set The second type loss corresponding to all images in M2 is used to train the feature fusion module, the encoder of DAB-DETR, and the decoder of DAB-DETR until the stopping condition is met, and the trained M2' is obtained.

[0061] In this embodiment, global image features Extracted using the first three layers of the CLIP image encoder.

[0062] Image Input M1' and get the region classification result output by M1'. The region classification result is the image recognized by M1'. The category label of the target in each region is determined. The region classification results are input into the CLIP text encoder, and then the image is obtained after dimensionality increase through a fully connected layer. Textual feature information ,Will and Input the feature fusion module and obtain the fused image information output by the feature fusion module. .in:

[0063] To enable the model to not only obtain target category embedding information through the region classifier, but also align image and text features into the same semantic space during the target localization stage of the class-aware locator, and semantically enhance image features using text features to better understand unknown categories and improve the detection capability of unknown categories, a feature fusion module is adopted. This module mainly consists of three parts: self-attention, text-to-image cross-attention, and image-to-text cross-attention. Figure 3 As shown;

[0064] The calculation formula for the self-attention module is shown below, and the calculation logic diagram is as follows. Figure 4 As shown:

[0065]

[0066] , , These represent queries, keys, and values ​​of the input sequence, respectively. for , The dimension size is used to scale the attention size.

[0067] Cross-attention modules compute attention across two different sequences to handle the semantic relationships between them. For example, in translation tasks, aligning source and target language sentences requires cross-attention to calculate the attention weights between the two sentences. It splits the input tensor into... and It takes two parts, then uses one part as the query set and the other as the key-value set. Its output is a dataset of size [size missing]. The tensor, for each row vector, gives its attention weights over all row vectors. Specifically, let... and The calculation of cross attention is as follows:

[0068]

[0069] Will and The images are fed together into the feature fusion module for feature fusion, thereby obtaining fused image information. .

[0070] In this model, the DAB-DETR model is used as the location detector. The region classification result obtained by M1' is processed through a fully connected MLP operation, and the result is used as the content query of DAB-DETR. The location query remains unchanged and is a learnable query that will be continuously learned in subsequent training processes.

[0071] Image collection The DAB-DETR model was trained using the pre-trained M1' module. This module was then frozen, and only the Transformer encoder, decoder, and position query parts of the DAB-DETR model were trained. The DAB-DETR model was then used as the object detection model M2, and M2' was obtained after training.

[0072] In order for M2' to detect unknown categories, the detector needs to learn the target localization ability of unknown categories. However, during the training process, there are only a limited number of annotations for known categories. In order to better solve the problem of detecting unknown categories, this embodiment adopts the anchor pre-matching method, which encourages the generalization to the localization of perceived objects of unknown categories during the inference process.

[0073] In this embodiment, the object locator is implemented using a DETR-style Transformer encoder-decoder architecture, where the encoder refines the feature maps and the decoder decodes a set of object queries into bounding box predictions. DAB-DETR is used as the object locator, where each object query is associated with an anchor box, and each ground truth box is pre-matched to a set of query results with the same label. By associating anchor frames The allocation is based on classification:

[0074]

[0075] In the formula, For anchor frame Regional characteristics, Here, we embed the class name of class c, and cosine represents the cosine similarity; the higher the value, the higher the similarity. After pre-matching, we predict the class embedding for each object's query conditions to allow for category-aware bounding box regression. The conditional object query is calculated as follows:

[0076]

[0077] The MLP computation process is the computation of the multilayer perceptron in the Transformer, which yields each object query and its associated anchor box. , The matching cost calculation is defined as follows:

[0078]

[0079] The binary classification loss is expressed as follows: , This represents the localization loss between the predicted bounding box and the actual bounding box. In this embodiment, the focal loss is used for calculation. , Through This is achieved by weighting the loss with the GIoU loss. The model is optimized using the following loss to obtain the final loss calculation result:

[0080]

[0081] S40, using M1' and M2' for target detection task;

[0082] In this embodiment, after training is completed, target category detection is performed by M1' and target location detection is performed by M2'. These two parts together realize the target detection task, so that the detection of unknown categories can be achieved even with a dataset containing only known categories.

[0083] To verify the limitations of the method of this invention, experiments were conducted on COCO, a mainstream open-source dataset for target detection:

[0084] Following the convention of the COCO OVD benchmark for open vocabulary object detection, the 80 categories in the COCO dataset are divided into 48 known categories and 17 unknown categories. The model proposed in this study is trained on the 48 known categories, which contain 107,761 images and 665,387 object instances. The model is then evaluated on a validation set for the unknown categories, which contains 4,836 images and 33,152 object instances from the 48 known categories and 17 unknown categories.

[0085] Evaluation Metrics: This method uses AP50 as the evaluation metric, which calculates the average precision of each class on the intersection as 50%, and then averages it across all classes. In experiments, three metrics were calculated separately for unknown classes, known classes, and all classes to evaluate the model's generalization performance. The metrics list includes the additional dataset requirements for each method and whether they require information about unknown classes during retraining.

[0086] Experimental indicators:

[0087] Table 1: Evaluation of experimental performance on COCO.

[0088]

[0089] Table 1

[0090] The experimental results in Table 1 show that this method is superior to existing methods, improving the detection performance for known categories and demonstrating its superiority.

[0091] The beneficial effects of the embodiments of this application are:

[0092] 1. This method proposes an open-vocabulary target detection approach based on cross-modal feature fusion for cross-modal open-vocabulary target detection. It addresses the challenge of current pre-trained visual language models failing to capture more refined information across modalities, enabling deep interaction of image and text feature information, learning high-quality language-aware visual representations, and improving the model's generalization ability.

[0093] 2. This paper proposes an anchor box pre-matching method for open vocabulary object detection. The method adaptively assigns anchor boxes to different classes based on the image content, and all classes can be decoded sequentially, thereby eliminating the need to repeatedly decode each class.

[0094] Secondly, embodiments of this application also provide an open vocabulary target detection device based on cross-modal feature fusion.

[0095] In one embodiment, reference is made to Figure 5 , Figure 5 This is a functional module diagram of an embodiment of the open vocabulary target detection device based on cross-modal feature fusion according to this application. Figure 5 As shown, the open vocabulary target detection device based on cross-modal feature fusion includes:

[0096] Module 10 is used to add images containing a specific target into an image set. In this context, the location of a specific target within its image and its category label are known information.

[0097] The first training module 20 is used to train the image set. The region classifier module M1 is trained using the training set to obtain region cues containing the trained regions. M1';

[0098] The second training module 30 is used for training the image set. any one of the pictures Extract global image features , the picture Input M1' to obtain the region classification result output by M1', then input the region classification result into the CLIP text encoder, and finally pass it through a fully connected layer for dimensionality upscaling to obtain the image. Textual feature information ,Will and Input the feature fusion module and obtain the fused image information output by the feature fusion module. ,Will The encoder of DAB-DETR is fed into the DAB-DETR encoder, and the decoder, which uses the region classification result as a query, is fed into DAB-DETR to obtain the object detection result. Based on the object detection result and known information, the image is calculated. The corresponding second type of loss; comprehensive image set The second type loss corresponding to all images in M2 is used to train the feature fusion module, the encoder of DAB-DETR, and the decoder of DAB-DETR until the stopping condition is met, and the trained M2' is obtained.

[0099] The detection module 40 is used to perform target detection tasks using M1' and M2'.

[0100] Furthermore, in one embodiment, the feature fusion module includes a self-attention component, a text-to-image cross-attention component, and an image-to-text cross-attention component.

[0101] Furthermore, in one embodiment, global image features Extracted using the first three layers of the CLIP image encoder.

[0102] Furthermore, in one embodiment, the first training module 20 is used for:

[0103] For image collection any one of the pictures Based on specific targets in the image Regional features are extracted from the location in the image to obtain the regional image features. ,exist Add a region hint The region embedding results are obtained. ,calculate With regional text feature set The similarity of text features in each region is used to determine the predicted category label of a specific target based on the text feature corresponding to the maximum similarity. The image is then calculated by combining the category label of the specific target. The corresponding first type of loss involves processing each category label in the image dataset using a region description language, then feeding it into the CLIP text encoder to obtain the region text features corresponding to each category label, and finally adding them to the region text feature set. Comprehensive Image Collection Type I loss pairs for all images in the dataset provide region hints. Training continues until the stopping condition is met, at which point a prompt containing the trained region is obtained. M1'.

[0104] Furthermore, in one embodiment, the first training module 20 is specifically used for:

[0105] Based on specific objectives in images The location within the map is combined with RoI Pooling technology to obtain the region of interest map;

[0106] Feature extraction of the region of interest map is performed using the fourth layer of the CLIP image encoder to obtain the region image features. .

[0107] Furthermore, in one embodiment, the detection module 40 is specifically used for:

[0108] Target category detection is performed using M1';

[0109] Target location detection is completed using M2'.

[0110] The functions of each module in the above-mentioned open vocabulary target detection device based on cross-modal feature fusion correspond to the steps in the above-mentioned open vocabulary target detection method embodiment based on cross-modal feature fusion, and their functions and implementation processes will not be described in detail here.

[0111] Thirdly, this application provides an open vocabulary target detection device based on cross-modal feature fusion. The open vocabulary target detection device based on cross-modal feature fusion can be a personal computer (PC), laptop computer, server or other device with data processing capabilities.

[0112] Reference Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of an open vocabulary target detection device based on cross-modal feature fusion involved in the embodiments of this application. In the embodiments of this application, the open vocabulary target detection device based on cross-modal feature fusion may include a processor, a memory, a communication interface, and a communication bus.

[0113] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0114] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the open vocabulary target detection device based on cross-modal feature fusion, as well as interfaces used for interconnecting the open vocabulary target detection device based on cross-modal feature fusion with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0115] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0116] The processor can be a general-purpose processor, which can call the open vocabulary target detection program based on cross-modal feature fusion stored in memory and execute the open vocabulary target detection method based on cross-modal feature fusion provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the open vocabulary target detection program based on cross-modal feature fusion is called can be referred to the various embodiments of the open vocabulary target detection method based on cross-modal feature fusion in this application, and will not be repeated here.

[0117] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0118] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0119] This application stores an open vocabulary target detection program based on cross-modal feature fusion on a computer-readable storage medium, wherein when the open vocabulary target detection program based on cross-modal feature fusion is executed by a processor, it implements the steps of the open vocabulary target detection method based on cross-modal feature fusion as described above.

[0120] The method implemented when the open vocabulary target detection program based on cross-modal feature fusion is executed can be referred to in the various embodiments of the open vocabulary target detection method based on cross-modal feature fusion in this application, and will not be repeated here.

[0121] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0122] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0123] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0124] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0125] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0127] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An open vocabulary target detection method based on cross-modal feature fusion, characterized in that, The open vocabulary target detection method based on cross-modal feature fusion includes: Add images containing a specific target to the image collection. In this context, the location of a specific target within its image and its category label are known information. With image collection The region classifier module M1 is trained using the training set to obtain region cues containing the trained regions. M1' includes: for image sets any one of the pictures Based on specific targets in the image Regional features are extracted from the location in the image to obtain the regional image features. ,exist Add a region hint The region embedding results are obtained. ,calculate With regional text feature set The similarity of text features in each region is used to determine the predicted category label of a specific target based on the text feature corresponding to the maximum similarity. The image is then calculated by combining the category label of the specific target. The corresponding first type of loss, based on the comprehensive image set. Type I loss pairs for all images in the dataset provide region hints. Training continues until the stopping condition is met, at which point a prompt containing the trained region is obtained. M1'; where, by using learnable region cues To enhance regional features and correct misalignment information, regional hints The region features are trained using the locations of targets with known class labels from the training set, and fine-tuned using the CLIP image encoder. During training, the weights of other models are frozen, and hints are only given to the regions that need to be learned. Partial training was conducted; With image collection The object detection model M2 is trained using the image set to obtain the trained M2', which includes: for the image set any one of the pictures Extract global image features , the image Input M1' to obtain the region classification result output by M1', then input the region classification result into the CLIP text encoder, and finally pass it through a fully connected layer for dimensionality upscaling to obtain the image. Textual feature information ,Will and Input the feature fusion module and obtain the fused image information output by the feature fusion module. ,Will The encoder of DAB-DETR is fed into the DAB-DETR encoder, and the decoder, which uses the region classification result as a query, is fed into DAB-DETR to obtain the object detection result. Based on the object detection result and known information, the image is calculated. The corresponding second type of loss; comprehensive image set The second type loss corresponding to all images in M2 is used to train the feature fusion module, the encoder of DAB-DETR, and the decoder of DAB-DETR until the stopping condition is met, and the trained M2' is obtained. Target detection task using M1' and M2'.

2. The open vocabulary target detection method based on cross-modal feature fusion as described in claim 1, characterized in that, The feature fusion module includes a self-attention component, a text-to-image cross-attention component, and an image-to-text cross-attention component.

3. The open vocabulary target detection method based on cross-modal feature fusion as described in claim 1, characterized in that, Global image features Extracted using the first three layers of the CLIP image encoder.

4. The open vocabulary target detection method based on cross-modal feature fusion as described in claim 1, characterized in that, The open vocabulary target detection method based on cross-modal feature fusion also includes: For each category label in the image dataset, after processing with a region description language, the data is fed into the CLIP text encoder to obtain the region text features corresponding to each category label, and then added to the region text feature set. .

5. The open vocabulary target detection method based on cross-modal feature fusion as described in claim 1, characterized in that, Based on specific objectives in images Regional features are extracted from the location in the image to obtain the regional image features. include: Based on specific objectives in images The location within the map is combined with RoI Pooling technology to obtain the region of interest map; Feature extraction of the region of interest map is performed using the fourth layer of the CLIP image encoder to obtain the region image features. .

6. The open vocabulary target detection method based on cross-modal feature fusion as described in claim 1, characterized in that, Target detection tasks using M1' and M2' include: Target category detection is performed using M1'; Target location detection is completed using M2'.

7. An open vocabulary target detection device based on cross-modal feature fusion, characterized in that, The open vocabulary target detection device based on cross-modal feature fusion includes: Builder modules are used to add images containing a specific target to an image set. In this context, the location of a specific target within its image and its category label are known information. The first training module is used to train the image set. The region classifier module M1 is trained using the training set to obtain region cues containing the trained regions. M1' includes: for image sets any one of the pictures Based on specific targets in the image Regional features are extracted from the location in the image to obtain the regional image features. ,exist Add a region hint The region embedding results are obtained. ,calculate With regional text feature set The similarity of text features in each region is used to determine the predicted category label of a specific target based on the text feature corresponding to the maximum similarity. The image is then calculated by combining the category label of the specific target. The corresponding first type of loss, based on the comprehensive image set. Type I loss pairs for all images in the dataset provide region hints. Training continues until the stopping condition is met, at which point a prompt containing the trained region is obtained. M1'; where, by using learnable region cues To enhance regional features and correct misalignment information, regional hints The region features are trained using the locations of targets with known class labels from the training set, and fine-tuned using the CLIP image encoder. During training, the weights of other models are frozen, and hints are only given to the regions that need to be learned. Partial training was conducted; The second training module is used to train the image set. The object detection model M2 is trained using the image set to obtain the trained M2', which includes: for the image set any one of the pictures Extract global image features , the image Input M1' to obtain the region classification result output by M1', then input the region classification result into the CLIP text encoder, and finally pass it through a fully connected layer for dimensionality upscaling to obtain the image. Textual feature information ,Will and Input the feature fusion module and obtain the fused image information output by the feature fusion module. ,Will The encoder of DAB-DETR is fed into the DAB-DETR encoder, and the decoder, which uses the region classification result as a query, is fed into DAB-DETR to obtain the object detection result. Based on the object detection result and known information, the image is calculated. The corresponding second type of loss; comprehensive image set The second type loss corresponding to all images in M2 is used to train the feature fusion module, the encoder of DAB-DETR, and the decoder of DAB-DETR until the stopping condition is met, and the trained M2' is obtained. The detection module is used to perform target detection tasks using M1' and M2'.

8. The open vocabulary target detection device based on cross-modal feature fusion as described in claim 7, characterized in that, The feature fusion module includes a self-attention component, a text-to-image cross-attention component, and an image-to-text cross-attention component.

9. An open vocabulary target detection device based on cross-modal feature fusion, characterized in that, The open vocabulary target detection device based on cross-modal feature fusion includes a processor, a memory, and an open vocabulary target detection program based on cross-modal feature fusion stored in the memory and executable by the processor, wherein when the open vocabulary target detection program based on cross-modal feature fusion is executed by the processor, it implements the steps of the open vocabulary target detection method based on cross-modal feature fusion as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an open vocabulary target detection program based on cross-modal feature fusion, wherein when the open vocabulary target detection program based on cross-modal feature fusion is executed by a processor, it implements the steps of the open vocabulary target detection method based on cross-modal feature fusion as described in any one of claims 1 to 6.

Images