A prompt-free object detection method and device, a terminal and a storage medium

Abstract

The application provides a prompt-free object detection method and device, a terminal and a storage medium. The method comprises the following steps: inputting a to-be-detected image into a trained general visual prompt model to obtain a plurality of first detection boxes; performing category name prediction on each first detection box to obtain a first category of each first detection box; obtaining an image feature of the to-be-detected image, taking the first category as a text prompt, and performing target detection on the image feature to obtain a second detection box and a corresponding second category; and determining a final detection box and a category name corresponding to the final detection box based on the first detection box and the second detection box. The application does not require any prior information input, and the initial classification is obtained first, and then the initial classification is taken as a text prompt of a target detection model to perform target detection, thereby improving the accuracy of the detection result and realizing output of the final detection box and the category name of all objects in the image without any prompt.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and computer vision, and in particular to a prompt-free object detection method and device, a terminal and a storage medium. BACKGROUND

[0002] In the field of image detection and recognition, traditional detection methods usually rely on predefined categories and training data sets, which means that the system can only detect object categories known in the training phase. For example, existing deep learning models such as Faster R-CNN, YOLO, DETR, etc. are trained through supervised learning, requiring a large number of labeled image datasets to cover all possible detection targets. However, this method has significant limitations: when applied to open scenarios or unseen objects, detection performance drops dramatically. While prompt-based multi-modal models such as OpenAI's CLIP can perform image classification in a zero-shot scenario, prompt-based multi-modal models have been developed.

[0003] However, prompt-based multi-modal models usually rely on user-provided category names as prompt inputs. That is, the current technology still requires human input prompts or predefined categories for object detection when facing complex and open environments, making it difficult to effectively adapt to dynamic and unknown scenarios. In particular, in applications such as autonomous driving, intelligent monitoring, and agricultural monitoring, the types of objects are diverse and unpredictable, posing a great challenge to traditional methods. Therefore, the existing technology is difficult to achieve zero-shot object detection without any prompt input, especially in practical application scenarios with a large number of object categories and complex scenarios.

[0004] Therefore, the existing technology has defects and needs to be improved and developed. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a prompt-free object detection method, device, terminal and storage medium to solve the problem of difficulty in achieving zero-shot object detection without any prompt input in the prior art.

[0006] The technical solution adopted by the present application to solve the technical problem is as follows:

[0007] A prompt-free object detection method, wherein the method comprises:

[0008] inputting a to-be-detected image into a trained general visual prompt model to obtain a plurality of first detection boxes;

[0009] performing category name prediction on each of the first detection boxes to obtain a first category of each of the first detection boxes;

[0010] obtaining an image feature of the image to be detected, taking the first category as a text prompt, performing target detection on the image feature to obtain a second detection box and a corresponding second category;

[0011] determining a final detection box and a category name corresponding to the final detection box based on the first detection box and the second detection box.

[0012] In an embodiment of the present application, the image to be detected is input into a trained general visual prompt model to obtain a plurality of first detection boxes, including:

[0013] The image to be detected is input into the trained general visual prompt model, and a multi-scale feature of the image to be detected is extracted based on a backbone network in the general visual prompt model;

[0014] The multi-scale feature is flattened and input into an encoder in the general visual prompt model to obtain an image feature of the image to be detected;

[0015] obtaining a query vector learned in a training process of the general visual prompt model, inputting the query vector and the image feature into a decoder of the general visual prompt model to obtain an updated query vector;

[0016] obtaining a preset general visual prompt learned in the training process of the general visual prompt model, calculating a similarity between the updated query vector and the preset general visual prompt to obtain a plurality of first detection boxes.

[0017] In an embodiment of the present application, a category name prediction is performed on each of the first detection boxes to obtain a first category of each of the first detection boxes, including:

[0018] extracting a regional image feature corresponding to each of the first detection boxes;

[0019] inputting the regional image feature into a category prediction model to obtain a first category of each of the first detection boxes.

[0020] In an embodiment of the present application, a final detection box and a category name corresponding to the final detection box are determined based on the first detection box and the second detection box, including:

[0021] calculating an intersection over union of the first detection box and the second detection box;

[0022] if the intersection over union is less than or equal to a preset threshold, taking the first detection box and the first category and the second detection box and the second category as the final detection box and the corresponding category name.

[0023] In an embodiment of the present application, determining the final detection frame and the category name corresponding to the final detection frame based on the first detection frame and the second detection frame comprises:

[0024] calculating the intersection-over-union of the first detection frame and the second detection frame;

[0025] if the intersection-over-union is greater than a preset threshold, taking the second detection frame and the second category as the final detection frame and the corresponding category name.

[0026] In an embodiment of the present application, determining the final detection frame and the category name corresponding to the final detection frame based on the first detection frame and the second detection frame comprises:

[0027] determining the final detection frame based on the first detection frame and the second detection frame, and obtaining the category name corresponding to the final detection frame.

[0028] In an embodiment of the present application, determining the final detection frame and the category name corresponding to the final detection frame based on the first detection frame and the second detection frame comprises:

[0029] if the first category and the second category name are the same, taking the average of the position coordinates of the first detection frame and the second detection frame as the position information of the final detection frame to determine the final detection frame, and taking the first category or the second category as the category name corresponding to the final detection frame.

[0030] In an embodiment of the present application, the training step of the general visual prompt model comprises:

[0031] obtaining a training data set, wherein the training data set comprises: a plurality of training images, and a plurality of training images are labeled with label frames corresponding to different objects, and the category labels corresponding to different objects are all a preset general category;

[0032] obtaining a pre-constructed initial general visual prompt model;

[0033] training the initial general visual prompt model based on the training data set to obtain a trained general visual prompt model.

[0034] In an embodiment of the present application, training the initial general visual prompt model based on the training data set to obtain a trained general visual prompt model comprises:

[0035] inputting the training image into the initial general visual prompt model, and extracting multi-scale image features of the training image based on the backbone network in the initial general visual prompt model;

[0036] The multi-scale image features are flattened and input into an encoder in the initial general visual prompt model to obtain image features of the training image.

[0037] The learnable query vector is randomly initialized, the randomly initialized query vector and the image features of the training image are input into a decoder of the initial general visual prompt model, and an updated query vector is obtained to obtain a predicted bounding box.

[0038] The similarity between the updated query vector and the general category prompt is calculated to determine a predicted category.

[0039] The initial general visual prompt model is trained based on a pre-constructed multi-task loss function with the generation of the general visual prompt as a training target to obtain a trained general visual prompt model.

[0040] In an embodiment of the present application, the multi-task loss function includes a bounding box regression loss, a category loss, and a region matching loss; the bounding box regression loss is a position loss between the predicted bounding box and a labeled bounding box, the category loss is a loss between the predicted category and a general category, and the region matching loss is a matching loss between the predicted bounding box and the labeled bounding box.

[0041] In an embodiment of the present application, the first category has multiple categories; after the category name prediction of each first bounding box is performed to obtain the first category of each first bounding box, the method further includes:

[0042] When a selection instruction of the first category is received, a target category corresponding to the selection instruction is obtained.

[0043] The image features of the image to be detected are obtained, the target category is taken as a text prompt, and target detection is performed on the image features to obtain a second bounding box and a corresponding second category.

[0044] The present application also provides an object detection device without prompt, wherein the device includes:

[0045] An image input module is configured to extract image features of an image to be detected.

[0046] A first detection module is configured to obtain a plurality of first bounding boxes, perform category name prediction on each first bounding box, and obtain a first category of each first bounding box.

[0047] A second detection module is configured to take the first category as a text prompt, perform target prediction on the image features, and obtain a second bounding box and a corresponding second category.

[0048] An output module is configured to determine a final detection frame and a class name corresponding to the final detection frame based on the first detection frame and the second detection frame.

[0049] The application further provides a terminal, which comprises a memory, a processor, and a non-prompted object detection program stored in the memory and executable on the processor, and the non-prompted object detection program implements the steps of the non-prompted object detection method when executed by the processor.

[0050] The application further provides a computer readable storage medium, which stores a computer program executable for implementing the steps of the non-prompted object detection method.

[0051] The application provides a non-prompted object detection method, device, terminal, and storage medium, which comprises the following steps: inputting a to-be-detected image into a trained general visual prompt model to obtain a plurality of first detection frames; performing class name prediction on each first detection frame to obtain a first class of each first detection frame; obtaining an image feature of the to-be-detected image, taking the first class as a text prompt, and performing target detection on the image feature to obtain a second detection frame and a corresponding second class; and determining a final detection frame and a class name corresponding to the final detection frame based on the first detection frame and the second detection frame. The trained general visual prompt model can automatically detect and output the first detection frame of all objects in an image and predict the first class of the first detection frame to obtain an initial classification without any prior information input or specific training data set for training a specific class. The name of the initial classification is automatically taken as a text prompt of an open-set target detection model to obtain a second detection frame, and the final detection frame is further confirmed based on the first detection frame and the second detection frame to output a final detection frame and a class name corresponding to the final detection frame as an output result. Through twice automatic target detection, the accuracy of the detection result can be improved, and the final detection frame and the class name of all objects in the image can be automatically output without any prompt. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 FIG. 1 is a flowchart of a preferred embodiment of the non-prompted object detection method in the application.

[0053] Figure 2 FIG. 2 is a logic architecture diagram of a preferred embodiment of the non-prompted object detection method in the application.

[0054] Figure 3 FIG. 3 is a functional principle block diagram of a preferred embodiment of the non-prompted object detection device in the application.

[0055] Figure 4 is a functional block diagram of a preferred embodiment of the terminal in the present application. DETAILED DESCRIPTION

[0056] In order to make the objects, technical solutions and advantages of the present application clearer and more explicit, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0057] The existing object detection technology mainly relies on pre-defined categories and a large amount of labeled data, which leads to poor performance in specific open scenarios and unknown category object detection. The following are the main shortcomings of the prior art:

[0058] First, prior prompts or pre-defined categories are required. Traditional object detection methods (such as YOLO, Faster R-CNN, DETR, etc.) rely on pre-defined category labels in the training stage, and the target detection model can only detect known object categories in the training set. Once encountering an unseen object, the detection performance will drop sharply. In addition, detection methods based on multi-modal models (such as CLIP) usually require users to input category prompts, otherwise the detection effect is limited, which is difficult to apply in dynamic and open scenarios.

[0059] Second, accurate target detection cannot be achieved without prompts. Although existing technologies attempt to expand the detection capability of the model through multi-modal fusion (such as text and image), they still need to rely on user input text prompts to assist detection. This prompt-dependent approach cannot achieve automated and intervention-free object detection, especially in applications such as autonomous driving, agricultural monitoring, and intelligent security, where object categories are variable and unpredictable, making it difficult to meet actual needs.

[0060] Third, the object detection accuracy is limited by the initial detection stage. Traditional detection methods are prone to inaccurate detection boundaries when predicting the initial detection box, and lack of subsequent result optimization mechanism, resulting in insufficient accuracy and recall rate of the final detection result. This is particularly evident in complex scenarios, such as in multi-object occlusion and dense scenarios, where existing methods are difficult to accurately identify all targets.

[0061] In view of the above defects of the prior art, the present application can improve the accuracy of the detection result through two automated target detections, and can automatically output the final detection box and its category name of all objects in the image without any prompts.

[0062] The application belongs to the cross field of artificial intelligence, computer vision and natural language processing, and particularly relates to design and optimization of a multimodal large language model (MLLM) applied to visual perception, image understanding, object detection, region recognition and human-computer interaction scenes.

[0063] Please refer to Figure 1 , Figure 1 is a flowchart of the object detection method without prompting in the application. As shown in Figure 1 , the object detection method without prompting described in the embodiment of the application comprises the following steps:

[0064] Step S100, inputting an image to be detected into a trained general visual prompt model to obtain a plurality of first detection boxes.

[0065] In an embodiment, the category of each of the first detection boxes is a same preset general category. The preset general category does not refer to a specific object, but refers to any object. The preset general category is a broad concept, which can be defined as "object" "general" and the like, and is not limited in the application. The object in the application includes any thing, target detection object such as human, animal, plant, fruit and workpiece. The features of each category name are learned by using the trained general visual prompt model to obtain the features of a general category.

[0066] By using the trained general visual prompt model, the application can automatically detect and output the first detection boxes of all objects in the image without any prior information input, and generate a prompt covering multiple object categories by using a multimodal fusion technology to ensure efficient detection of all potential objects in the scene.

[0067] In the embodiment of the application, the step S100 specifically comprises:

[0068] Step S110, inputting the image to be detected into the trained general visual prompt model, and extracting multi-scale features of the image to be detected based on a backbone network in the general visual prompt model;

[0069] Step S120, inputting the multi-scale features after flattening into an encoder in the general visual prompt model to obtain image features of the image to be detected;

[0070] Step S130, acquiring a query vector learned in a training process of the general visual prompt model, inputting the query vector and the image features into a decoder of the general visual prompt model to obtain an updated query vector;

[0071] Step S140, obtaining a preset general visual cue learned in a general visual cue model training process, calculating similarity between the updated query vector and the preset general visual cue, and obtaining a plurality of first detection boxes.

[0072] The input data of the general visual cue model is an image to be detected, and the output data is a candidate region (Bounding Boxes) preliminarily detected, i.e., a first detection box. Specifically, as shown in Figure 2 The image to be detected is first subjected to a backbone network (Backbone Network), and the backbone network can use Swin Transformer (or EVA02) to extract multi-scale features; the extracted multi-scale features are flattened and input into an encoder (for example, a Transformer) to obtain image features of the image to be detected, so as to capture global context information in the image. Without predefining a class input, a query vector (Query Embeddings) learned in a general visual cue model training process is obtained, the query vector is input into a Transformer decoder, and the decoder outputs a plurality of first detection boxes and a preliminary confidence score, which are used to predict whether there is a detection target in the first detection box in the image. The confidence is the similarity between the query vector and the general visual cue, and each predicted detection box corresponds to a confidence score, which is used to represent the possibility that the detection box has a detection target. If the confidence score is greater than a threshold, the class name prediction model of the detection box is further predicted.

[0073] As shown in Figure 1 The method described in the embodiments of the present application further includes:

[0074] Step S200, predicting a class name for each of the first detection boxes to obtain a first class of each of the first detection boxes.

[0075] Specifically, the class prediction model inputs the first detection box output by the decoder of step S140 and the multi-scale features into the class prediction model, which can automatically detect and output a class name of an object, i.e., a first class. The class prediction model can be an autoregressive model. If step S100 obtains a plurality of first detection boxes, the class prediction model outputs a class name corresponding to each first detection box, and then step S200 obtains a plurality of first classes. For example, there are two first detection boxes, which are a cat and a dog, and the output first classes include "cat" and "dog".

[0076] In the embodiments of the present application, the step S200 specifically includes:

[0077] Step S210, extracting a region image feature corresponding to each of the first detection boxes;

[0078] Step S220: Input the region image features into the category prediction model to obtain the first category of each of the first detection boxes.

[0079] Specifically, for the initially generated candidate regions, i.e. the first detection boxes, the ROI Align technique is used to extract the region image features corresponding to each of the first detection boxes from the image features of the image to be detected. The extracted region features are converted into query vectors and input into the category prediction model (e.g., Open Pre-trained Transformer autoregressive model).

[0080] The category prediction model predicts the category of the region image features corresponding to each first detection box. An autoregressive approach is used to progressively generate the first category name for each detection box. The number of categories in the first category is less than or equal to the number of query vectors; however, this application does not limit the number of query vectors (it can be up to 900). In this embodiment, the category prediction model can be a general category prediction model, where the first category belongs to an open vocabulary. The category prediction model can predict the probability of each detection box in all categories of the open vocabulary, and the category with the highest probability is taken as the first category.

[0081] like Figure 1 As shown, the method described in this embodiment of the invention further includes:

[0082] Step S300: Obtain the image features of the image to be detected, use the first category as a text prompt, perform target detection on the image features, and obtain a second detection box and the corresponding second category.

[0083] This application uses the first category as a text prompt word to automatically input a category-guided open-set object detection model for secondary detection and refinement. Step S300 is category-based secondary detection, which can be performed using an open-set object detection model, which can be a zero-shot object detection model. Specifically, the first category obtained in the region recognition stage is used as the prompt input and input into the open-set object detection model. The open-set object detection model architecture includes a backbone network and a Transformer encoder-decoder. The backbone network can use a Swin Transformer (or EVA02) to extract image features.

[0084] The image features can be those obtained from step S120. The Transformer encoder-decoder then uses the first category as a text prompt to guide the output of a second detection box and the corresponding second category.

[0085] The application improves the precision and recall rate of object detection by taking the first category as a text prompt of an open set object detection model to obtain a second detection box, thereby ensuring the accuracy of the final detection result.

[0086] As shown in Figure 1 The method also includes the following steps:

[0087] In step S400, a final detection box is determined based on the first detection box and the second detection box, and the final detection box corresponds to a category name.

[0088] Specifically, the application can perform zero-prompt detection on any image without pre-defined object category labels or prior information input. The first detection box and category prediction are obtained through general object detection, and the second detection box is obtained through object detection based on the predicted category, which can automatically realize a closed-loop process from the first detection box obtained through general object detection to accurate object detection and then to detection result optimization, thereby greatly improving the robustness and generalization ability of object detection, achieving high-precision detection and classification of all objects in the image, and improving the object detection and recognition ability of the multi-modal large language model in complex scenes. The application can automatically detect and recognize all objects in the image without prior input and any pre-defined prompt, improve intelligent perception ability in various complex environments, significantly improve the wide adaptability and precision of object detection, and is widely applicable to various artificial intelligence application scenarios such as autonomous driving, intelligent robots, augmented reality (AR), image search, intelligent question-answering systems, intelligent city monitoring, agricultural monitoring, medical image analysis, natural disaster warning, logistics automation, etc.

[0089] In the embodiment of the application, the step S400 specifically includes the following steps:

[0090] The intersection over union of the first detection box and the second detection box is calculated.

[0091] If the intersection over union is less than or equal to a preset threshold, the first detection box and the first category, and the second detection box and the second category are taken as the final detection box and the corresponding category name.

[0092] The application outputs the object bounding box and category name after secondary detection optimization, ensuring more accurate object recognition in complex scenes. Specifically, if the intersection over union of the second detection box and the first detection box is less than or equal to a preset threshold (i.e., the first detection box and the second detection box are not the same detection target), the first detection box and the first category, and the second detection box and the second category are retained to avoid missing detection.

[0093] In one embodiment of the application, the step S400 includes the following steps:

[0094] calculating an intersection over union of the first bounding box and the second bounding box;

[0095] if the intersection over union is greater than a preset threshold, taking the second bounding box and the second category as a final bounding box and a corresponding category name.

[0096] Specifically, the second bounding box is merged with the first bounding box, if the intersection over union is greater than the preset threshold, that is, the first bounding box and the second bounding box are the same detection target, and the second bounding box is obtained based on the text prompt and is more accurate than the first bounding box, then only the second bounding box and the corresponding second category are retained.

[0097] The application automatically eliminates false detection (category false detection) and repeated detection by comparing with the first bounding box, avoids missing detection, and improves the accuracy and recall rate of the overall detection.

[0098] In an embodiment of the application, another implementation manner of the step S400 is: determining a final bounding box based on the first bounding box and the second bounding box, and obtaining a category name corresponding to the final bounding box.

[0099] Specifically, the second bounding box can be directly determined as the final bounding box, and a category name corresponding to the final bounding box is obtained, because the second bounding box is obtained by target detection with the first category of the first bounding box as the text prompt, and the detection result is also more accurate.

[0100] It can be understood that in other embodiments, more strategies can be set based on the first bounding box and the second bounding box to determine the final bounding box, for example, if the first category and the second category name are the same, the average value of the position coordinates of the first bounding box and the second bounding box is taken as the position information of the final bounding box to determine the final bounding box, and the first category or the second category is taken as the category name corresponding to the final bounding box.

[0101] In an embodiment of the application, the training step of the general visual prompt model comprises:

[0102] Step S10, acquiring a training data set, the training data set comprising: a plurality of training images, a plurality of training images having an annotation box corresponding to different objects, and a category label corresponding to different objects being a same preset general category;

[0103] Step S20, acquiring a pre-constructed initial general visual prompt model;

[0104] Step S30, training the initial general visual prompt model based on the training data set to obtain a trained general visual prompt model.

[0105] The present application trains a general visual prompt model by integrating multiple different data sets, so that the model can automatically detect all objects in an image without pre-defined class prompts.

[0106] Specifically, in order to ensure the wide applicability of the general visual prompt, the present application integrates multiple different types of data sets, including instance-level data sets and part-level data sets. The instance-level data set contains accurate object instance annotations. The part-level data set is used to capture the detailed parts and fine-grained features of the object.

[0107] The class labels in all integrated data sets are standardized, for example, all are classified as "object", that is, all specific classes such as "dog", "car", "table", etc. are abstracted into general object class labels, i.e. preset general classes, and then the training data set is obtained. That is, the preset general class does not refer to a specific object, but refers to any object. The preset general class is a broad concept, which can be defined as "object" "general" and the like, which is not limited in the present application. This class generalization processing makes the model no longer dependent on specific class names, thereby realizing the zero-shot detection capability of the general prompt.

[0108] In an embodiment of the present application, the step S30 specifically comprises:

[0109] Step S31, inputting the training image into the initial general visual prompt model, and extracting multi-scale image features of the training image based on the backbone network in the initial general visual prompt model;

[0110] Step S32, flattening the multi-scale image features and inputting them into the encoder in the initial general visual prompt model to obtain image features of the training image;

[0111] Step S33, randomly initializing a learnable query vector, inputting the randomly initialized query vector and the image features of the training image into the decoder of the initial general visual prompt model to obtain an updated query vector to obtain a predicted detection box;

[0112] Step S34, calculating the similarity between the updated query vector and the general class prompt to determine the predicted class;

[0113] Step S35, taking the generation of the general visual prompt as the training target, training the initial general visual prompt model based on the pre-constructed multi-task loss function to obtain a trained general visual prompt model.

[0114] Specifically, the training target of the universal visual prompt model provided in the present application is to enable the model to automatically generate universal prompts without explicit category prompts, and to detect the bounding boxes of all objects in the image using these prompts.

[0115] The query vector is a learnable parameter. During the model training process, the training of the query vector adopts a query mechanism similar to that in DETR (Detection Transformer). Specifically, the query vector is initialized by a learnable nn.Embedding layer and is continuously adjusted and optimized during the training process. Initially, the query vector is randomly initialized as a fixed-length vector sequence (such as 900 vectors used in the present application), and each vector is used as the input of the decoder to detect objects in the image. During the training process, the query vector interacts with the image features through multiple levels of Transformer decoder, thereby learning semantic information related to objects. With the training of a large number of instance-level datasets and part-level datasets, the query vector gradually understands the object categories and can effectively capture the features and boundary information of the objects. Finally, the query vector can not only detect the positions of the objects, but also provide preliminary category prompts for the region recognition stage, thereby providing a basis for subsequent autoregressive category prediction.

[0116] In an embodiment of the present application, the multi-task loss function includes a bounding box regression loss, a category loss, and a region matching loss; the bounding box regression loss is the position loss between the predicted bounding box and the labeled bounding box, the category loss is the loss between the predicted category and the universal category, and the region matching loss is the matching loss between the predicted bounding box and the labeled bounding box.

[0117] Specifically, the bounding box is a four-dimensional coordinate used to determine the position of the bounding box in the image. When calculating the matching loss between the predicted bounding box and the labeled bounding box, for example, if the number of query vectors is 900 and the number of predicted bounding boxes is 900, and the number of labeled bounding boxes is 10, the matching loss is to find 10 target bounding boxes from the 900 predicted bounding boxes, so that the sum of the bounding box regression loss and the category loss of the 10 target bounding boxes and the predicted bounding boxes is minimized.

[0118] In an embodiment of the present application, the first category has multiple categories; after the step S200, the method further includes: when receiving a selection instruction of the first category from a user, obtaining a target category corresponding to the selection instruction; obtaining image features of the image to be detected, taking the target category as a text prompt, and performing target detection on the image features to obtain a second bounding box and a corresponding second category.

[0119] Specifically, the first category has multiple, the instruction is "select N categories", then get the corresponding N target categories, N is greater than or equal to 1, less than or equal to the number of first categories of the first detection frame. Through interaction with the user, the interactivity with the user is improved.

[0120] The present application achieves the following beneficial effects:

[0121] First, the real zero prompt object detection is realized. In view of the problem that the prior art needs to predefine categories or user input prompts in object detection, the present application automatically detects all objects in the image through a general visual prompt model without human intervention.

[0122] Second, the accuracy and flexibility of object detection are improved. The existing method is easily limited in the initial candidate region generation stage, resulting in inaccurate detection boundaries. The present application optimizes the initial detection result through region recognition and secondary detection mechanism based on category guidance, ensuring high precision and high recall rate of object detection, especially in complex and variable scenes.

[0123] Third, the dependence on large-scale labeled data is reduced. In view of the problem that the existing method highly depends on labeled data, the present application reduces the dependence on manual labeling through automatic prompt generation and multi-stage detection process, reduces the cost of training data set construction, improves the generalization ability of the model, thereby improving the adaptability of the system in an open environment, especially suitable for unmanned driving, intelligent monitoring and agricultural monitoring fields.

[0124] Fourth, the detection precision and recall rate are significantly improved. Through the secondary detection based on category guidance, the present application can further optimize the object boundary and category recognition based on the preliminary detection. Compared with the traditional detection method, the present application effectively reduces the false detection and missed detection phenomenon in the secondary detection, improves the boundary precision and classification accuracy of object detection.

[0125] Fifth, the robustness in multi-object complex scene is improved. The present application adopts a multi-stage detection and optimization strategy, which can efficiently detect and recognize multiple objects even in complex scenes with dense objects and severe occlusion. Compared with the traditional method, the present application has stronger robustness in detecting and classifying diversified objects.

[0126] Sixth, it is flexible and adaptable to various application scenarios. Since it does not depend on specific category input and prior knowledge, the present application has wide applicability in various practical application scenarios, including unmanned driving, intelligent monitoring, agricultural monitoring and medical image analysis, providing more intelligent and automated object detection solutions for various industries.

[0127] Further, as Figure 3As shown, based on the above object detection method without prompting, the application also correspondingly provides an object detection device without prompting, comprising:

[0128] An image input module 100 is configured to extract image features of an image to be detected.

[0129] A first detection module 200 is configured to obtain a plurality of first detection boxes and identify the class names of each of the first detection boxes to obtain a first class of each of the first detection boxes.

[0130] A second detection module 300 is configured to use the first class as a text prompt to perform target prediction on the image features to obtain a second detection box and a corresponding second class.

[0131] An output module 400 is configured to determine a final detection box and a class name corresponding to the final detection box based on the first detection box and the second detection box.

[0132] Figure 4 A structure schematic diagram of a terminal provided by an embodiment of the application is provided. The terminal can include:

[0133] A memory 501, a processor 502, and a computer program stored in the memory 501 and executable on the processor 502.

[0134] The processor 502 implements the object detection method without prompting provided in the above embodiments when executing the program.

[0135] Further, the terminal also includes:

[0136] A communication interface 503 is configured to communicate between the memory 501 and the processor 502.

[0137] The memory 501 is configured to store a computer program executable on the processor 502.

[0138] The memory 501 can include a high-speed RAM memory and can also include a non-volatile memory, such as at least one disk memory.

[0139] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0140] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete communication between each other through an internal interface.

[0141] The processor 502 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0142] The embodiment also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the above object detection method without prompting.

[0143] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0144] Moreover, the terms "first", "second", "third", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of instances indicated. Thus, features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0145] Any process or method descriptions or blocks in flow charts herein, and elsewhere, can be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of the preferred embodiments of the present application in which additional functionality can be added or some functionality can be removed, by, for example, adding one or more steps performing a similar or reciprocal function, combining two or more steps into a single step, or splitting one step into two or more steps.

[0146] The logic and / or steps represented in flow charts herein, and elsewhere, can be considered as a sequence of executable instructions, for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch instructions from a instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can specifically include the following: an electronic connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium upon which the program can be printed, as the program can be electronically captured, via the optical scan of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in the computer memory.

[0147] It should be understood that parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. If realized in hardware and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0148] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by a program instructing the relevant hardware, which can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0149] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above integrated module can be realized in the form of hardware or in the form of a software functional module. The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0150] The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

[0151] To sum up, the object detection method, device, terminal and storage medium disclosed by the application, the method comprises: inputting a to-be-detected image into a trained general visual prompt model to obtain a plurality of first detection boxes; performing category name prediction on each first detection box to obtain a first category of each first detection box; obtaining image features of the to-be-detected image, taking the first category as a text prompt, performing target detection on the image features to obtain a second detection box and a corresponding second category; and determining a final detection box and a category name corresponding to the final detection box based on the first detection box and the second detection box. Through the trained general visual prompt model, without any prior information input, all objects in the image can be automatically detected and the first detection box of the object can be output, and the first category of the first detection box can be predicted to obtain an initial classification. The initial classification is taken as a text prompt of an open-set target detection model to obtain a second detection box, and the final detection box is further confirmed based on the first detection box and the second detection box to output the final detection box. The final detection box and the category name corresponding to the final detection box are taken as output results to improve the accuracy of the detection results, and the final detection box of all objects in the image and the category name of the final detection box can be output without any prompt.

[0152] It should be understood that the application of the present application is not limited to the above examples, and can be improved or changed according to the above description for those skilled in the art. All these improvements and changes shall belong to the protection scope of the appended claims of the present application.

Claims

1. A method of object detection without prompting, characterized by, The method comprises: inputting the image to be detected into the trained general visual prompt model to obtain a plurality of first detection boxes, the category of each first detection box being a same preset general category, the preset general category referring to any object; performing category name prediction on each first detection box through a category prediction model to obtain a first category of each first detection box, the first category belonging to an open vocabulary; obtaining image features of the image to be detected, taking the first category as a text prompt, and performing target detection on the image features to obtain a second detection box and a corresponding second category; determining a final detection box and a category name corresponding to the final detection box based on the first detection box and the second detection box; the training data set of the general visual prompt model comprises: a plurality of training images, a plurality of training images being labeled with a label box corresponding to different objects, and a category label corresponding to different objects being a same preset general category; the training target of the general visual prompt model is that the model automatically generates a general visual prompt without explicit category prompt; the inputting the image to be detected into the trained general visual prompt model to obtain a plurality of first detection boxes comprises: inputting the image to be detected into the trained general visual prompt model to detect all objects in the image to be detected through the general visual prompt to obtain a first detection box.

2. The unobtrusive object detection method of claim 1, wherein, the inputting the image to be detected into the trained general visual prompt model to obtain a plurality of first detection boxes comprises: inputting the image to be detected into the trained general visual prompt model, extracting multi-scale features of the image to be detected based on a backbone network in the general visual prompt model; inputting the flattened multi-scale features into an encoder in the general visual prompt model to obtain image features of the image to be detected; obtaining a query vector learned in a general visual prompt model training process, inputting the query vector and the image features into a decoder of the general visual prompt model to obtain an updated query vector; obtaining a preset general visual prompt learned in the general visual prompt model training process, calculating a similarity between the updated query vector and the preset general visual prompt to obtain a plurality of first detection boxes.

3. The unobtrusive object detection method of claim 1, wherein, the performing category name prediction on each first detection box to obtain a first category of each first detection box comprises: extracting regional image features corresponding to each first detection box; inputting the regional image features into a category prediction model to obtain a first category of each first detection box.

4. The unobtrusive object detection method of claim 1, wherein, the determining a final detection box and a category name corresponding to the final detection box based on the first detection box and the second detection box comprises: calculating an intersection over union of the first detection box and the second detection box; if the intersection over union is less than or equal to a preset threshold, taking the first detection box and the first category and the second detection box and the second category as the final detection box and the corresponding category name.

5. The unobtrusive object detection method of claim 1, wherein, the determining a final detection box and a category name corresponding to the final detection box based on the first detection box and the second detection box comprises: calculating an intersection over union of the first detection box and the second detection box; If the intersection over union is greater than a preset threshold, the second detection frame and the second category are taken as a final detection frame and a corresponding category name.

6. The unobtrusive object detection method of claim 1, wherein, The method further includes: The method further includes:

7. The unobtrusive object detection method of claim 1, wherein, The method further includes: The method further includes:

8. The unobtrusive object detection method of claim 1, wherein, If the first category and the second category name are the same, an average value of position coordinates of the first detection frame and the second detection frame is taken as position information of the final detection frame to determine the final detection frame, and the first category or the second category is taken as the category name corresponding to the final detection frame. The training step of the general visual prompt model includes: obtaining a training data set, the training data set including a plurality of training images, and a plurality of training images having a plurality of labeled boxes corresponding to different objects, and a plurality of category labels corresponding to different objects being a same preset general category; obtaining a pre-constructed initial general visual prompt model; 9. The unobtrusive object detection method of claim 8, wherein, training the initial general visual prompt model based on the training data set to obtain a trained general visual prompt model. The training step of the general visual prompt model includes: inputting the training image into the initial general visual prompt model, and extracting a multi-scale image feature of the training image based on a backbone network in the initial general visual prompt model; inputting the flattened multi-scale image feature into an encoder in the initial general visual prompt model to obtain an image feature of the training image; randomly initializing a learnable query vector, inputting the randomly initialized query vector and the image feature of the training image into a decoder of the initial general visual prompt model to obtain an updated query vector to obtain a predicted detection frame; calculating a similarity between the updated query vector and the general category prompt to determine a predicted category; 10. The unobtrusive object detection method of claim 9, wherein, training the initial general visual prompt model based on a pre-constructed multi-task loss function to obtain a trained general visual prompt model, with the general visual prompt as a training target.

11. The unobtrusive object detection method of claim 1, wherein, The multi-task loss function includes a detection frame regression loss, a category loss, and a region matching loss; the detection frame regression loss is a position loss between the predicted detection frame and the labeled box, the category loss is a loss between the predicted category and the general category, and the region matching loss is a matching loss between the predicted detection frame and the labeled box. The first category has a plurality of categories; after obtaining the first category of each first detection frame by predicting the category name of each first detection frame, the method further includes: when receiving a selection instruction of the first category from a user, obtaining a target category corresponding to the selection instruction; 12. An object detection apparatus without prompting, characterized by, obtaining an image feature of the image to be detected, taking the target category as a text prompt, and performing target detection on the image feature to obtain a second detection frame and a corresponding second category. The device includes: An image input module is configured to extract image features of a to-be-detected image. A first detection module is configured to obtain a plurality of first bounding boxes, each of the first bounding boxes is of a same preset general category, the preset general category refers to any object, and the first detection module is further configured to perform category name prediction on each of the first bounding boxes by using a category prediction model to obtain a first category of each of the first bounding boxes, the first category belongs to an open vocabulary. A second detection module is configured to perform target prediction on the image features by taking the first category as a text prompt to obtain second bounding boxes and corresponding second categories. An output module is configured to determine a final bounding box and a category name corresponding to the final bounding box based on the first bounding boxes and the second bounding boxes. The image input module is further configured to input the to-be-detected image into a trained general visual prompt model, and the first detection module is further configured to detect first bounding boxes of all objects in the to-be-detected image by using the general visual prompt. A training data set of the general visual prompt model includes a plurality of training images, and the training images are labeled with bounding boxes corresponding to different objects, and category labels corresponding to the different objects are all of a same preset general category. A training target of the general visual prompt model is to automatically generate general visual prompts without explicit category prompts.

13. A terminal, characterized by comprising: The computer readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the object detection method without prompts according to any one of claims 1-11. The computer readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the object detection method without prompts according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, ​

Images