Target detection method and device, equipment, storage medium and computer program product

Abstract

The invention provides a target detection method and device, equipment, a storage medium and a computer program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a to-be-detected text and a to-be-detected image; inputting the to-be-detected text and the to-be-detected image into the open set target detection model to obtain a detection result output by the open set target detection model; wherein the open set target detection model comprises a mixing module of image information and text information, the mixing module comprises a multilayer multi-head self-attention mechanism and a multilayer multi-head cross attention mechanism based on category guidance, and the mixing module is used for enhancing a first image category feature vector and a text feature of a to-be-detected text, and obtaining a first enhanced text feature. According to the method, the mixing module is introduced into the open set target detection model, the image category feature vector is combined with the text feature, and the text feature carries the image semantic information, so that the influence caused by semantic ambiguity is reduced, and the zero-sample and few-sample reasoning capability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target detection method, device, equipment, storage medium and computer program product. Background Art

[0002] Driven by large models and high computing power, the traditional closed-set target detection model has been gradually replaced by the open-set target detection paradigm. However, for the current open-set target detection paradigm, the zero-shot and few-shot reasoning capabilities of the open-set target detection model are easily affected by the type and scale of pre-training data. In actual business needs, open-set target detection has the disadvantages of difficult to design text prompts and irregular attempts. Therefore, how to improve the zero-shot and few-shot reasoning capabilities of the open-set target detection model has become an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a target detection method, apparatus, device, storage medium and computer program product, which are used to solve the defects in the prior art of how to improve the zero-sample and few-sample reasoning capabilities of an open-set target detection model, and achieve the goal of introducing a hybrid module into the open-set target detection model, combining image category feature vectors with text features, so that the text features carry image semantic information, thereby reducing the impact of semantic ambiguity and improving the zero-sample and few-sample reasoning capabilities of the model.

[0004] The present invention provides a target detection method, comprising the following steps: Acquire a text to be detected and an image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; Inputting the to-be-detected text and the to-be-detected image into an open-set object detection model to obtain a detection result output by the open-set object detection model; Among them, the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer category-guided multi-head cross-attention mechanism LG_MCA, and the hybrid module is used to enhance the first image category feature vector and the text features of the text to be detected to obtain a first enhanced text feature.

[0005] According to a target detection method provided by the present invention, the open set target detection model further includes a text encoding module, an image encoding module, a neck processing module and a head processing module; The text encoding module is used to extract text features of the text to be detected and input the text features into the mixing module; The image encoding module is used to extract image features of the image to be detected and input the image features into the neck processing module; The mixing module is used to perform an MSA operation on the first image category feature vector, and perform an LG_MCA operation on the MSA operation result and the text feature to obtain the first enhanced text feature, and input the first enhanced text feature into the neck processing module; The neck processing module is used to perform feature fusion on the image feature and the first enhanced text feature, and input the fused feature into the head processing module; The head processing module is used to classify and locate the fused features and output the detection results.

[0006] According to a target detection method provided by the present invention, obtaining the first image category feature vector includes: Retrieving the image category of the image to be detected in the image category feature library; If the image category is retrieved in the image category feature library, a first image category feature vector corresponding to the image category is extracted from the image category feature library.

[0007] According to a target detection method provided by the present invention, obtaining the first image category feature vector also includes: Retrieving the image category of the image to be detected in the image category feature library; If the image category is not retrieved in the image category feature library, extracting image category features of a plurality of prompt images and storing them in the image category feature library; the prompt image has the same image category as the image to be detected; A first image category feature vector corresponding to the image category is extracted from the image category feature library.

[0008] According to a target detection method provided by the present invention, the open set target detection model is trained based on the following method: Get the training dataset; Constructing an image category feature library according to the training data set; Extracting a second image category feature vector from the image category feature library according to the category label of the sample image; According to the hybrid module, the text features of the sample text and the second image category feature vector are enhanced to obtain a second enhanced text feature; Determining a loss function according to the second enhanced text feature and the image feature of the sample image; According to the loss function and gradient back propagation, the structural parameters of the hybrid module are updated to obtain the open set target detection model.

[0009] According to a target detection method provided by the present invention, constructing an image category feature library based on the training data set includes: Load the training dataset and initialize an empty dictionary as the initial image category feature library; Traversing the coordinate information of all annotated bounding box regions belonging to label category i in the training data set to generate a coordinate list; Randomly select K bounding box region coordinate information from the coordinate list, extract image category features of the region where the K bounding box region coordinate information is located, so as to generate an image category feature matrix of the label category i; Adding the image category feature matrix of the label category i to the initial image category feature library; The traversal operation of the category labels in the training data set is repeatedly performed until all the category labels in the training data set are traversed to generate a complete image category feature library.

[0010] The present invention also provides a target detection device, comprising the following modules: An acquisition module, used to acquire the text to be detected and the image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; A detection module is used to input the text to be detected and the image to be detected into an open set target detection model to obtain a detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer category-guided multi-head cross-attention mechanism LG_MCA, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain a first enhanced text feature.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the target detection methods described above is implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the target detection method as described above is implemented.

[0013] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the target detection methods described above.

[0014] The target detection method, device, equipment, storage medium and computer program product provided by the present invention obtain the text to be detected and the image to be detected; input the text to be detected and the image to be detected into an open set target detection model to obtain the detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism and a multi-layer multi-head cross-attention mechanism based on category guidance, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain the first enhanced text feature. The present invention introduces a hybrid module into the open set target detection model to combine the image category feature vector with the text feature, so that the text feature carries the image semantic information, thereby reducing the impact of semantic ambiguity and improving the zero-sample and few-sample reasoning capabilities of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flow chart of the target detection method provided by the present invention.

[0017] Figure 2 It is a schematic diagram of the ITFB structure provided by the present invention.

[0018] Figure 3 It is a schematic diagram of the forward propagation process provided by the present invention.

[0019] Figure 4 It is a flowchart of the reasoning process 1 provided by the present invention.

[0020] Figure 5 It is a flowchart of the reasoning process 2 provided by the present invention.

[0021] Figure 6 It is a flowchart of the reasoning process 3 provided by the present invention.

[0022] Figure 7 It is a schematic diagram of the process of training an open set target detection model provided by the present invention.

[0023] Figure 8 It is a schematic diagram of the process of offline construction of an image category feature library provided by the present invention.

[0024] Fig. 9 It is a structural schematic diagram of the target detection device provided by the present invention.

[0025] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In related technologies, most of the mainstream open-set target detection paradigms such as Grounding Dino and YOLO-World adopt the following detection paradigm process: 1. The image is input to the image encoder part, where the image encoder can be Swin Transformer, ResNet50, etc. The image is subjected to multi-scale image feature extraction through the image encoder; 2. The text is input into the text encoder part to obtain text features, where the text encoder can be EVA CLIP, etc. 3. Input multi-scale image features and text features into the neck together, take FPN (Feature Pyramid Networks) as the typical architecture, and integrate various attention operations, such as MSA (Multi-Scale Attention), MCA (Multi-Channel Attention), and MSDA (Multi-Scale Dilation Attention), to refine the multi-scale image features while integrating the semantic information of the text features into the image features; 4. The text features and the refined image features that incorporate text semantic information are sent to the head. The head is generally composed of a classification head and a positioning head. The former determines the category of the detected target, and the latter regresses the target's position information and outputs the positioning result.

[0028] The emergence of Grounding Dino transforms the task of open-set target detection into a grounding task, realizing the generalization of open-field detection; the emergence of YOLO-World, with its extremely fast reasoning speed, further increases the possibility of the deployment and application of open-set target detection models. For the visual small model structure based on YOLO, the open-set target detection model has very impressive zero-sample and few-sample reasoning capabilities. For detection tasks where the prompt text vocabulary is unambiguous and the detection target description is clear, the open-set target detection model can meet the downstream detection tasks without training and fine-tuning.

[0029] However, for the current open-set object detection paradigm, the model's zero-shot and few-shot reasoning capabilities are severely constrained by the following issues: 1) Type and scale of pre-training data: The zero-sample and few-sample capabilities of the open-set target detection model largely depend on the type and scale of the pre-training data. At present, the pre-training data of the open-set target detection model can be roughly divided into: detection data, grounding data, rec data, and image-text-pair data. However, unlike the field of natural language processing, text information comes from the label tags or caption descriptions of the image annotations, but the text semantic information is still not rich enough. The effect of full fine-tuning of the text encoder module will be reduced, and due to the limitations of the graphics card computing power, a large part of the open-set target detection models will not unfreeze the parameters for training during the pre-training stage, resulting in differences in the alignment of the image features and text features output by the image encoder module and the text encoder module; 2) Text prompt word input: For the trained open-set target detection model, in order to give full play to its zero-shot and few-shot reasoning capabilities, it is necessary to try and design text prompt words for the input text encoding module. However, there is no experience in the design of text prompt words, which is completely a random trial and error without any rules. In addition, text encoders are mostly trained in the English context, and the degree of adaptation to Chinese is low. Most of them translate Chinese into English with the help of tools for prompt word input. The Chinese-English conversion process can easily introduce ambiguous information, causing semantic understanding deviations in subsequent zero-shot or few-shot reasoning scenario tasks, affecting their detection effects.

[0030] Therefore, the present invention proposes a target detection method, specifically a method for enhancing the zero-sample and few-sample reasoning capabilities of an open-set target detection model, which mainly solves the following technical problems: To alleviate the vulnerability of the open-set target detection model's zero-shot and few-shot reasoning capabilities to the type and size of pre-training data, enhance its zero-shot and few-shot reasoning capabilities based on transfer learning without full fine-tuning as much as possible.

[0031] It alleviates the drawbacks of the current open-set target detection in actual business needs, such as the difficulty in designing text prompts and irregular attempts, and reduces the impact of some ambiguous text inputs on the reasoning results.

[0032] Combine the following Figure 1-Figure 10 The object detection method, apparatus, device, storage medium and computer program product of the present invention are described.

[0033] Figure 1 It is a flow chart of the target detection method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101, obtaining text to be detected and image to be detected.

[0034] It can be understood that open set object detection is a computer vision task whose goal is to identify and detect known category objects in images (i.e. objects seen in training), while also being able to handle new categories of objects that have not been seen. This detection method is suitable for dynamic environments, such as surveillance, autonomous driving, and other scenarios.

[0035] Obtain the text to be detected and the image to be detected, wherein the text to be detected is at least one of the annotation information, category information and description information of the image to be detected. For example, the text information usually comes from the annotation of the image data, which may contain the category, location and other features of each object, and the annotation information provides the model with a clear description of which targets are present in the image. Category information refers to the specific type of the object to be detected in the image, such as "dog", "car" or "person", and these category labels help the model identify and classify the targets in the image. The phrase description is a more detailed explanation of the target, which may include the state, action or other features of the target. For example, "a fast-running dog" not only specifies the category, but also provides additional information about the target's behavior. This information can improve the model's ability to understand complex scenes.

[0036] Step 102, input the text to be detected and the image to be detected into an open set target detection model to obtain the detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer category-guided multi-head cross-attention mechanism LG_MCA, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain a first enhanced text feature.

[0037] The text to be detected and the image to be detected are input into the open set target detection model to obtain the detection result output by the open set target detection model. Among them, the open set target detection model introduces a plug-and-play image information-text information hybrid module (Image Text Fusion Block, ITFB), that is, a hybrid module of image information and text information, which is used to enhance the first image category feature vector and the text features of the text to be detected, that is, to integrate the image semantic information of the target to be detected into the text feature information, so that the text feature information can obtain more image detail information, so that the subsequent neck processing module and head processing module can strengthen the fusion process of text feature information and image features, and reduce the impact of a part of pure text feature input.

[0038] like Figure 2 As shown in the figure, the ITFB structure mainly adopts two attention structures: multi-head self-attention mechanism (MSA) and a new attention mechanism - multi-head cross-attention mechanism based on category guidance (LG_MCA). In the forward process of ITFB, the image category feature vector is enhanced by MSA, and then input into LG_MCA together with the text feature to enhance the text feature. Compared with the standard MCA, LG_MCA adds an additional category guidance mechanism. The unique category mask formed by the category guidance mechanism makes the vector in the text feature Token only perform cross-attention operation with the features of the corresponding category in the image category feature vector, avoiding cross-attention operation with vectors of other categories. It can be understood that it is equivalent to matching a single category feature in the text feature Token with multiple different categories of image category feature vectors one by one. If the match is successful, the corresponding category mask is 1, indicating that there is an association between the text feature and the image category feature vector, and then the category token in the text feature is used to perform the MCA operation with the corresponding category vector in the image category feature vector. It is equivalent to adding a matching process to the standard MCA.

[0039] LG_MCA can enhance the text features of the corresponding category more accurately, instead of randomly cross-enhancing the text features and visual features like the simple MCA operation. This operation can make the text features of this category better carry the image semantic information of the corresponding category and be more corresponding. The ITFB structure enhances the text features by stacking multiple layers of MSA and LG_MCA blocks, and finally outputs the enhanced text features.

[0040] The target detection method provided by the embodiment of the present invention obtains the text to be detected and the image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; the text to be detected and the image to be detected are input into the open set target detection model to obtain the detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer multi-head cross-attention mechanism LG_MCA based on category guidance, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain the first enhanced text feature. The present invention introduces a hybrid module in the open set target detection model to combine the image category feature vector with the text feature, so that the text feature carries the image semantic information, thereby reducing the impact of semantic ambiguity and improving the zero-sample and few-sample reasoning capabilities of the model.

[0041] Based on the above embodiment, the open set object detection model further includes a text encoding module, an image encoding module, a neck processing module and a head processing module; A text encoding module is used to extract text features of the text to be detected and input the text features into the mixing module; An image encoding module is used to extract image features of the image to be detected and input the image features into the neck processing module; A mixing module, used for performing an MSA operation on the first image category feature vector, and performing an LG_MCA operation on the MSA operation result and the text feature to obtain a first enhanced text feature, and inputting the first enhanced text feature into the neck processing module; A neck processing module, used for fusing the image feature and the first enhanced text feature, and inputting the fused feature into the head processing module; The head processing module is used to classify and locate the fused features and output the detection results.

[0042] For example, refer to Figure 3 , the target detection process involves the following calculation formula: (1); (2); (3); (4); in, Image is the input image, i.e. the image to be detected; ImageEncoder is the image encoding module; F_I Image features output by the image encoding module; Text is the input text, i.e. the text to be detected; TextEncoder It is a text encoding module; F_T Text features output by the text encoding module; Extra is an additional image vector, i.e., the image category feature vector, which generally comes from images of the same type as the target to be detected; F_T_Intensified for ITFB The enhanced text features of the structure output, such as the first enhanced text features; Formula (4) is the enhanced text features and image features after passing through the neck processing module ( Neck ) and the head processing module ( Head ) to get the test results Result .

[0043] Due to the different reasoning processes of target detection, the first image category feature vector is obtained in different ways. The embodiment of the present invention provides two methods for obtaining the first image category feature vector, which are: Method 1: In the image category feature library, the image category of the image to be detected is retrieved; if the image category is retrieved in the image category feature library, the first image category feature vector corresponding to the image category is extracted from the image category feature library. The image category of the image to be detected can be understood as the category of the detection target. For example, assuming that the image to be detected includes apples, the image category is apple.

[0044] Method 2: In the image category feature library, the image category of the image to be detected is retrieved; if the image category is not retrieved in the image category feature library, the image category features of multiple prompt images are extracted, and the image category features of the multiple prompt images are stored in the image category feature library; wherein the prompt image has the same image category as the image to be detected; and a first image category feature vector corresponding to the image category is extracted from the image category feature library.

[0045] In order to reflect the plug-and-play nature of the ITFB structure, the reasoning process of target detection is determined based on whether the ITFB structure is used and whether the pre-extracted image category feature library M contains the category of the target to be detected. The present invention mainly provides three reasoning processes, each of which is as follows: Reasoning process 1: Do not use the ITFB structure for enhanced reasoning, use the original open set target detection model reasoning process, and only input the prompt text and the detection target image. This reasoning process is suitable for when the input prompt text is unambiguous and the target is clearly specified. Only a single prompt text can have good zero-sample or few-sample reasoning capabilities. In addition, retaining this reasoning process can be used as a backup measure. The training of the ITFB structure may result in differences in category detection due to the category distribution of the training data set, weakening the detection generalization ability of some categories. In order to ensure the uniform and standardized usability of the use process, this reasoning process is retained, which truly reflects the plug-and-play nature of the ITFB structure.

[0046] refer to Figure 4, the implementation steps of the reasoning process 1 include: the image is subjected to multi-scale image feature extraction through the image encoder (ImageEncoder), and the multi-scale image features are output; the text is subjected to text feature extraction through the text encoder (TextEncoder), and the text features are output; the multi-scale image features and the text features are input into the neck processing module (Neck), and the multi-scale image features are refined by the neck processing module, and the semantic information of the text features is integrated into the image features; the text features and the image features refined and integrated with the text semantic information are sent to the head processing module (Head), and the head processing module determines the category of the detection target, and regresses the position information of the target to output the detection result Result.

[0047] Reasoning process 2: Use the ITFB structure for enhanced reasoning, and the category of the detected target is already in the image category feature library M. Figure 5 ,The implementation steps of reasoning process 2 include: searching in the image category feature library M through the incoming prompt text content (such as the image category of the image to be detected), if the category vector of the category detection target exists in the image category feature library M, directly take out the corresponding image category feature vector, and input it into the ITFB structure with the text features extracted by the text encoder to enhance the text features, and then the output enhanced text features are sent to the neck processing module and the head processing module together with the image features extracted by the image encoder to execute the detector process, and finally output the detection results.

[0048] Reasoning process 3: Use the ITFB structure for enhanced reasoning to detect that the target category is not in the image category feature library M. This process supports the function of inferring images from images. Figure 6 , the implementation steps of reasoning process 3 include: searching in the image category feature library M through the incoming prompt text content (such as the image category of the image to be detected). If the category vector of the detection target of this category does not exist in the image category feature library M, the user needs to upload at least 5 (the number of uploaded images can be determined based on the detection requirements) prompt images of the same category as the detection target. The process will automatically extract the corresponding image category features offline based on the incoming prompt images and update them in the image category feature library M; the subsequent steps are consistent with reasoning process 2. The image category feature vector and the text features extracted by the text encoder are input into the ITFB structure to enhance the text features. The output enhanced text features are then sent to the neck processing module and the head processing module together with the image features extracted by the image encoder to execute the detector process, and finally output the detection results.

[0049] The ITFB structure provided in the embodiment of the present invention adopts a new attention mechanism LG_MCA, which controls the text feature Token to perform cross-attention operations only with the image category features of the corresponding category, and ignores the image features of other categories, so that the text features can be strengthened in a more targeted manner. On the other hand, the three reasoning processes can be used freely, and both the reasoning process of the original open set target detection backbone model and the additional ITFB reasoning process can be used. By combining image category features with text features, the ITFB structure can make the text features carry image semantic information, thereby reducing the impact of semantic ambiguity and improving the model's zero-sample and few-sample reasoning capabilities. In addition, it also supports the function of inferring images with images. The function of inferring images with images is more straightforward and efficient, and the dual-modal prompt combination of image and text enhances the reasoning effect.

[0050] Based on the above embodiment, the open set target detection model is trained based on the following method: Step 110, obtaining a training data set; Step 111, constructing an image category feature library based on the training data set; Step 112, extracting a second image category feature vector from an image category feature library according to the category label of the sample image; Step 113, according to the hybrid module, the text features of the sample text and the second image category feature vector are enhanced to obtain a second enhanced text feature; Step 114, determining a loss function according to the second enhanced text feature and the image feature of the sample image; Step 115, according to the loss function and gradient back propagation, the structural parameters of the hybrid module are updated to obtain an open set target detection model.

[0051] The embodiment of the present invention mainly adopts a local fine-tuning method to perform model training based on a detection data set. During the training process, the original open-set target detection model parameters are frozen, and only the newly added structure ITFB is locally fine-tuned. The training steps of the open-set target detection model are as follows: Figure 7 As shown, specifically including: Step 1. Training data set: Preparation of detection data set: Compared with grounding, rec and image-text-pair datasets, detection datasets have weaker text information, but the category label Label is a unique single category, which is conducive to forming a unique one-hot mapping relationship. The data mapped by the Label will not change with the different image data of each batch, and always maintain a fixed corresponding mapping relationship. For example, the 80 categories in the COCO dataset are mapped into a list of [0,1,…79], and the mapping relationship between the category label and the value in the list is always maintained constant.

[0052] Step 2: Build an image category feature library offline; According to the category label Label and the corresponding Bbox (Bounding box) annotated area in the existing training data set, for each Label label, K corresponding Bbox areas are randomly selected and sent to the image feature extractor for feature extraction to generate image category feature vectors. The image category feature library stores data in the form of a dictionary, where the key of the dictionary is Label and the key value is the image category feature vector.

[0053] In one embodiment, an image category feature library is constructed according to a training data set, including: loading the training data set and initializing an empty dictionary as an initial image category feature library; traversing all annotated bounding box region coordinate information belonging to label category i in the training data set to generate a coordinate list; randomly selecting K bounding box region coordinate information from the coordinate list, extracting image category features of the areas where the K bounding box region coordinate information are located, to generate an image category feature matrix of label category i; adding the image category feature matrix of label category i to the initial image category feature library; and repeatedly performing the traversal operation of the category labels in the training data set until all the category labels in the training data set are traversed to generate a complete image category feature library.

[0054] For example, the detailed image category feature library construction process takes the COCO dataset as an example. The corresponding process is as follows Figure 8 As shown: (1) Load the COCO dataset and initialize an empty dictionary {} as the image category feature library M; (2) There are 80 labels in the COCO dataset, and the corresponding category Label mapping interval is [0,79]. For any integer i in the mapping interval, traverse the coordinate information of all the labeled Bbox areas belonging to category Label i in the dataset to generate a coordinate list Bbox=[(x1,y1,w1,h1),(x2,y2,w2,h2)…], where the x and y elements represent the upper left corner coordinates of the object labeled area, and the w and h elements represent the width and height of the object labeled area; (3) Shuffle the order of the coordinate list Bbox, determine its width and height, retain only the labeled coordinate information whose width and height are both greater than 64 pixels, and randomly select K coordinate information; (4) According to the selected K coordinate information, the image category features are extracted based on the image feature extraction module. The image feature extraction module of the embodiment of the present invention is composed of an image encoding module in the open set target detection model superimposed with an average pooling layer. The average pooling layer has no parameters and does not require training; and the image feature extraction directly uses the encoding module parameters of the open set target detection model. Parameter reuse can also reduce the difficulty of subsequent feature alignment. However, for these image encoding modules, multi-scale features are generally extracted and downsampled according to the original input image size. For example, when the Swin Transformer is used as the image encoding module for 4-scale feature extraction, the image will be scaled to 1 / 64 of the original size. To ensure the effectiveness of feature extraction, the size of the input image should be greater than 64 pixels. Use the K coordinate information to obtain the corresponding ROI (region of interest) area on the corresponding original image, that is, the region of interest, and save it as a picture and send it to the image feature extraction module for image category feature extraction. First, the image is passed through the image encoder to output multi-scale image features, then the multi-scale features are flattened and spliced, and the dimensions are compressed through the average pooling layer to generate the corresponding image category feature matrix Mi=(K,1,256); (5) According to category i, add the corresponding image category feature matrix Mi to the image category feature library M, that is, add element M[i]=Mi(K,1,256) to the empty dictionary.

[0055] (6) Repeat steps (2)-(5), and continuously add the image category feature matrix of the corresponding category to the image category feature library until all categories in the training data set are traversed.

[0056] Step 3: The open set target detection model introduces an additional ITFB structure.

[0057] Step 4: Based on each batch of image and text input, image features and text features are extracted through the image encoder and text encoder.

[0058] Step 5: According to the category label of each batch of image data, randomly extract the corresponding image category feature vector from the image category feature library.

[0059] Step 6: Input the text features and randomly extracted image category feature vectors into the IFTB structure, and output the enhanced text features.

[0060] The schematic diagram of the forward process of ITFB is as follows Figure 2 As shown: (1) According to the true category label of each batch of image data, random extraction is performed in the image category feature library M. For example, taking the extraction of image category feature vectors of 5 categories as an example: when the true category of the image data is greater than 5 categories, 5 categories are randomly selected from the existing true categories, and then index extraction is performed in the image category feature library based on the randomly selected 5 categories; when the true category of the image data is less than 5 categories, indexing is performed in the image category feature library according to the existing number of categories, and the remaining categories are randomly supplemented to 5 categories. Finally, the category feature vector of the batch image data is obtained.

[0061] (2) The existing text feature token and category feature vector are input into the ITFB structure for feature enhancement, and new text features are output. The ITFB structure mainly adopts two attention structures: multi-head self-attention mechanism (MSA) and a new attention mechanism - multi-head cross-attention mechanism based on category guidance (LG_MCA). In the forward process of ITFB, the category feature vector is enhanced by MSA, and then input into LG_MCA together with the text feature for text feature enhancement. Compared with the standard MCA, LG_MCA adds an additional category guidance mechanism. The unique category mask formed by the category guidance mechanism makes the vector in the text feature token only perform cross-attention operation with the corresponding category feature in the image category feature, avoiding cross-attention operation with vectors of other categories. It can be understood that it is equivalent to matching the single category feature in the text feature token with the image category feature vectors of 5 different categories one by one. If the corresponding category mask is 1 during matching, the category token in the text feature is used to perform MCA operation with the corresponding category vector in the image category feature vector. It is equivalent to adding a matching process to the standard MCA.

[0062] Step 7: The enhanced text features and image features are sent to the subsequent neck processing module and head processing module to execute the detector process and calculate the loss function Loss.

[0063] Step 8: Loss and gradient back propagation to update ITFB structure parameters.

[0064] The training method is to fine-tune the newly added ITFB structure block and freeze all the other original parts. This training method can ensure the plug-and-play usage mode of the ITFB structure block and will not destroy the zero-sample and few-sample reasoning capabilities of the original open-set object detection model. The loss functions used in training can include classification loss, positioning loss, and confidence loss.

[0065] The training method of the embodiment of the present invention relies on the detection type data set to perform training fine-tuning of the additional text enhancement structure ITFB. Without destroying the parameters of the original open set target detection backbone model, the image category features and text features of the detection category are pre-extracted for mixed enhancement, so that the text feature Token of the corresponding category carries the image semantic information of the corresponding category, which can alleviate the problem of text input ambiguity to a certain extent; and reduce the difficulty of aligning subsequent image features and text features in the neck and head modules, thereby enhancing the corresponding zero-sample and few-sample reasoning capabilities of the model.

[0066] The target detection device provided by the present invention is described below. The target detection device described below and the target detection method described above can be referenced to each other.

[0067] refer to Fig. 9 , the target detection device provided by the present invention includes: an acquisition module 901 and a detection module 902.

[0068] The acquisition module 901 is used to acquire the text to be detected and the image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; The detection module 902 is used to input the text to be detected and the image to be detected into an open set target detection model to obtain the detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer category-guided multi-head cross-attention mechanism LG_MCA, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain a first enhanced text feature.

[0069] The target detection device provided by the embodiment of the present invention obtains the text to be detected and the image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; the text to be detected and the image to be detected are input into the open set target detection model to obtain the detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer multi-head cross-attention mechanism LG_MCA based on category guidance, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain the first enhanced text feature. The present invention introduces a hybrid module in the open set target detection model to combine the image category feature vector with the text feature, so that the text feature carries the image semantic information, thereby reducing the impact of semantic ambiguity and improving the zero-sample and few-sample reasoning capabilities of the model.

[0070] In one embodiment, the open set target detection model also includes a text encoding module, an image encoding module, a neck processing module and a head processing module; the text encoding module is used to extract text features of the text to be detected and input the text features into the mixing module; the image encoding module is used to extract image features of the image to be detected and input the image features into the neck processing module; the mixing module is used to perform an MSA operation on the first image category feature vector, and perform an LG_MCA operation on the MSA operation result and the text feature to obtain the first enhanced text feature, and input the first enhanced text feature into the neck processing module; the neck processing module is used to perform feature fusion on the image feature and the first enhanced text feature, and input the fused feature into the head processing module; the head processing module is used to classify and locate the fused feature and output the detection result.

[0071] In one embodiment, the detection module 902 is further used to: retrieve the image category of the image to be detected in the image category feature library; if the image category is retrieved in the image category feature library, extract the first image category feature vector corresponding to the image category from the image category feature library.

[0072] In one embodiment, the detection module 902 is further used to: retrieve the image category of the image to be detected in the image category feature library; if the image category is not retrieved in the image category feature library, extract the image category features of multiple prompt images and store them in the image category feature library; the prompt image has the same image category as the image to be detected; and extract the first image category feature vector corresponding to the image category from the image category feature library.

[0073] In one embodiment, the detection module 902 is also used to: obtain a training data set; construct an image category feature library based on the training data set; extract a second image category feature vector from the image category feature library based on the category label of the sample image; enhance the text features of the sample text and the second image category feature vector according to the hybrid module to obtain a second enhanced text feature; determine a loss function based on the second enhanced text feature and the image features of the sample image; and update the structural parameters of the hybrid module according to the loss function and gradient back propagation to obtain the open set target detection model.

[0074] In one embodiment, the detection module 902 is further used to: load a training data set and initialize an empty dictionary as an initial image category feature library; traverse all annotated bounding box area coordinate information belonging to label category i in the training data set to generate a coordinate list; randomly select K bounding box area coordinate information from the coordinate list, extract image category features of the area where the K bounding box area coordinate information is located, to generate an image category feature matrix of label category i; add the image category feature matrix of label category i to the initial image category feature library; repeatedly perform the traversal operation of the category labels in the training data set until all category labels in the training data set are traversed to generate a complete image category feature library.

[0075] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the target detection method, which includes: obtaining a text to be detected and an image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; the text to be detected and the image to be detected are input into an open set target detection model to obtain a detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer multi-head cross attention mechanism LG_MCA based on category guidance, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain a first enhanced text feature.

[0076] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0077] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target detection method provided by the above methods, which method includes: obtaining a text to be detected and an image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; the text to be detected and the image to be detected are input into an open set target detection model to obtain a detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer category-guided multi-head cross-attention mechanism LG_MCA, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain a first enhanced text feature.

[0078] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the target detection method provided by the above-mentioned methods, the method comprising: obtaining text to be detected and an image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; the text to be detected and the image to be detected are input into an open set target detection model to obtain a detection result output by the open set target detection model; wherein the open set target detection model comprises a hybrid module of image information and text information, the hybrid module comprises a multi-layer multi-head self-attention mechanism MSA and a multi-layer category-guided multi-head cross-attention mechanism LG_MCA, the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain a first enhanced text feature.

[0079] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection method, characterized in that: include: Acquire a text to be detected and an image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; Inputting the to-be-detected text and the to-be-detected image into an open-set object detection model to obtain a detection result output by the open-set object detection model; Among them, the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer category-guided multi-head cross-attention mechanism LG_MCA, and the hybrid module is used to enhance the first image category feature vector and the text features of the text to be detected to obtain a first enhanced text feature.

2. The target detection method according to claim 1, characterized in that: The open set object detection model also includes a text encoding module, an image encoding module, a neck processing module and a head processing module; The text encoding module is used to extract text features of the text to be detected and input the text features into the mixing module; The image encoding module is used to extract image features of the image to be detected and input the image features into the neck processing module; The mixing module is used to perform an MSA operation on the first image category feature vector, and perform an LG_MCA operation on the MSA operation result and the text feature to obtain the first enhanced text feature, and input the first enhanced text feature into the neck processing module; The neck processing module is used to perform feature fusion on the image feature and the first enhanced text feature, and input the fused feature into the head processing module; The head processing module is used to classify and locate the fused features and output the detection results.

3. The target detection method according to claim 2, characterized in that: Obtaining the first image category feature vector includes: Retrieving the image category of the image to be detected in the image category feature library; If the image category is retrieved in the image category feature library, a first image category feature vector corresponding to the image category is extracted from the image category feature library.

4. The target detection method according to claim 2, characterized in that: Acquiring the first image category feature vector also includes: Retrieving the image category of the image to be detected in the image category feature library; If the image category is not retrieved in the image category feature library, extracting image category features of a plurality of prompt images and storing them in the image category feature library; the prompt image has the same image category as the image to be detected; A first image category feature vector corresponding to the image category is extracted from the image category feature library.

5. The target detection method according to claim 1, characterized in that: The open set object detection model is trained based on the following method: Get the training dataset; Constructing an image category feature library according to the training data set; Extracting a second image category feature vector from the image category feature library according to the category label of the sample image; According to the hybrid module, the text features of the sample text and the second image category feature vector are enhanced to obtain a second enhanced text feature; Determining a loss function according to the second enhanced text feature and the image feature of the sample image; According to the loss function and gradient back propagation, the structural parameters of the hybrid module are updated to obtain the open set target detection model.

6. The target detection method according to claim 5, characterized in that: The step of constructing an image category feature library according to the training data set includes: Load the training dataset and initialize an empty dictionary as the initial image category feature library; Traversing the coordinate information of all annotated bounding box regions belonging to label category i in the training data set to generate a coordinate list; Randomly select K bounding box region coordinate information from the coordinate list, extract image category features of the regions where the K bounding box region coordinate information are located, so as to generate an image category feature matrix of the label category i; Adding the image category feature matrix of the label category i to the initial image category feature library; The traversal operation of the category labels in the training data set is repeatedly performed until all the category labels in the training data set are traversed to generate a complete image category feature library.

7. A target detection device, characterized in that: include: An acquisition module, used to acquire the text to be detected and the image to be detected; the text to be detected is at least one of the annotation information, category information and description information of the image to be detected; A detection module is used to input the text to be detected and the image to be detected into an open set target detection model to obtain a detection result output by the open set target detection model; wherein the open set target detection model includes a hybrid module of image information and text information, the hybrid module includes a multi-layer multi-head self-attention mechanism MSA and a multi-layer category-guided multi-head cross-attention mechanism LG_MCA, and the hybrid module is used to enhance the first image category feature vector and the text feature of the text to be detected to obtain a first enhanced text feature.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the target detection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the target detection method according to any one of claims 1 to 6 is implemented.

Images