Method and device for improving open word list target detection model

Abstract

The invention discloses an improvement method and device for an open word list target detection model, and belongs to the technical field of computer vision, and the method comprises the following steps: obtaining target training data which comprises a target training image, a target text label and a first target frame corresponding to the target text label; randomly augmenting the target training image to obtain a target augmented image, and judging whether the target text tag is unique in a training set to which the target training data belongs; and generating a prompt vector according to a judgment result, and inputting the prompt vector, the target training image and the target augmented image into an open word list target detection model for training to obtain an improved open word list target detection model. According to the application, on the premise of not increasing the original pre-training data set and not modifying the network structure, the verification index effect of the YOLOWORLD network model on the original verification set is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an improved method and device for an open vocabulary target detection model. Background Art

[0002] In the current field of computer vision, open vocabulary object detection technology is gradually becoming a research hotspot. The core of this technology is to expand the fixed number of detection categories in the traditional object detection algorithm to an unlimited number of category spaces through multimodal contrast learning of images and texts, thereby significantly improving the generalization ability and practicality of the algorithm.

[0003] Previously, most of the research on this technology was focused on attempts based on the Transformer network structure. Although this has achieved remarkable results to a certain extent, it generally faces the problem of huge computing resource overhead, which in turn limits its promotion in practical applications. Therefore, the industry has been exploring more efficient and low-cost solutions.

[0004] In 2024, at the top visual conference on Computer Vision and Pattern Recognition (CVPR), the YOLO_WORLD algorithm was proposed as a cutting-edge research result. The algorithm uses an efficient convolutional structure as the main body of the model, which not only effectively reduces the demand for computing resources, but also provides new ideas for the development of open vocabulary object detection technology.

[0005] However, the YOLO_WORLD algorithm also has certain limitations. Specifically, in the pre-training stage of the algorithm, for each image area, only a single text phrase is used as the corresponding prompt label for regional contrast learning. These text phrases are usually added manually or automatically annotated by the algorithm, and the semantic information contained is relatively one-sided and limited. For example, for an image area containing a puppy, the text label is mostly simply labeled as "dog", but in fact, this image area can also be described in many ways, such as "animals with tails", "animals with four legs", etc. This limitation of the label may cause deviations in the model when recognizing and understanding image content.

[0006] In addition, the YOLO_WORLD algorithm mainly focuses on the similarity between images and texts, but ignores the similarity between images. In the scenario of open vocabulary target detection, the similarity between images often contains rich contextual information and category associations, and is of great significance to improving the recognition accuracy and generalization ability of the model. Therefore, how to make full use of the similarity between images for training has also become one of the current problems to be solved.

[0007] In summary, although the YOLO_WORLD algorithm has achieved certain results in the field of open vocabulary object detection, there are problems such as one-sided semantic information and ignoring the similarity between images in the network pre-training stage, which requires further improvement and optimization. Summary of the invention

[0008] The purpose of the present invention is to provide an improved method and device for an open vocabulary object detection model to solve the problems of one-sided semantic information and ignoring the similarity between images in the YOLO_WORLD algorithm in the prior art during the network pre-training stage.

[0009] To achieve the above objectives, this application adopts the following technical solutions:

[0010] The present invention provides an improved method for an open vocabulary object detection model, comprising the following steps:

[0011] Acquire target training data, wherein the target training data includes a target training image, a target text label, and a first target frame corresponding to the target text label;

[0012] Randomly augmenting the target training image to obtain a target augmented image, and determining whether the target text label is unique in the training set to which the target training data belongs;

[0013] A prompt vector is generated according to the judgment result, and the prompt vector is respectively input into an open vocabulary object detection model with the target training image and the target augmented image for training to obtain an improved open vocabulary object detection model.

[0014] Preferably, the method further comprises:

[0015] The original target frame information of each image in the training set is stored in a first list in the form of a dictionary, wherein the original target frame information includes an identifier of the original target frame and a first text label corresponding to the original target frame;

[0016] Each dictionary in the first list is traversed, and a tag index table with the same name is constructed according to the traversal result, with the first text tag as the key and the second list consisting of identifiers of all original target frames corresponding to the first text tag as the value.

[0017] Preferably, the determining whether the target text label is unique in the training set to which the target training data belongs comprises:

[0018] Searching for an entry whose key is the target text label in the same-name label index table to obtain a second target list, wherein the second target list is a second list composed of identifiers of all original target frames corresponding to the target text label;

[0019] It is determined whether the identifier of the first target box exists uniquely in the target second list. If so, the target text label is unique in the training set to which the target training data belongs. Otherwise, it is not unique.

[0020] Preferably, generating a prompt vector according to the judgment result includes:

[0021] When the judgment result is yes, calculating the text feature vector of the target text label, and using the text feature vector as a prompt vector;

[0022] When the judgment result is no, a prompt vector is generated according to the target text label and other corresponding original target frames except the first target frame.

[0023] Preferably, generating a prompt vector according to the target text label and other corresponding original target frames except the first target frame includes:

[0024] Generate a random number, and compare the random number with a set threshold;

[0025] When the random number is greater than the set threshold, randomly extract any original target frame in the target second list except the first target frame as the second target frame, and crop a local target image according to the second target frame;

[0026] The local target image is converted into a local image feature vector, and the local image feature vector is fused with the text feature vector to obtain a prompt vector.

[0027] Preferably, converting the local target image into a local image feature vector comprises:

[0028] Convert the channel order of the local target image into RGB, and convert the image matrix after the channel order conversion into a tensor data format;

[0029] Inputting the image in tensor data format into the visual encoder for feature extraction to obtain a feature matrix, and calculating the Euclidean norm of each vector in the last dimension of the feature matrix;

[0030] Each vector in the feature matrix is ​​divided by its Euclidean norm to obtain a normalized feature matrix, and all dimensions except the last dimension in the normalized feature matrix are merged into one dimension to obtain a local image feature vector.

[0031] Preferably, the method further comprises:

[0032] Using the target augmented graph to perform self-supervised learning of the open vocabulary target detection model;

[0033] Wherein, the self-supervised learning includes:

[0034] A new loss function is added to the original loss function of the open vocabulary object detection model. The new loss function expression is:

[0035] ,

[0036] ,

[0037] in, ∈{0,1} is the indicator function, T is the temperature coefficient, x i and x j are the feature vectors of the target training image and the target augmented image obtained through network inference, the sim function is a function for calculating the similarity between two vectors, and N represents the number of images in a batch during model training.

[0038] An improved device for an open vocabulary object detection model, comprising:

[0039] An acquisition module, used to acquire target training data, wherein the target training data includes a target training image, a target text label, and a first target frame corresponding to the target text label;

[0040] A judgment module, used for randomly augmenting the target training image to obtain a target augmented image, and judging whether the target text label is unique in the training set to which the target training data belongs;

[0041] The training module is used to generate a prompt vector according to the judgment result, and input the prompt vector, the target training image and the target augmented image into an open vocabulary object detection model for training to obtain an improved open vocabulary object detection model.

[0042] An electronic device comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement an improved method for an open vocabulary object detection model as described in any one of the above.

[0043] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement a method for improving an open vocabulary object detection model as described in any one of the above when executed.

[0044] The present invention has the following beneficial effects:

[0045] In this application, image content similar to the target box area in the image is used, and the text label of the target box is integrated to perform image area comparative learning, thereby enriching the semantic information of the training label. At the same time, on the basis of image-text comparative learning, image-image comparative learning and image self-supervised learning are added to enhance the model's ability to represent images, and by learning the similarities between images, the model's ability to understand images is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of an improved method for an open vocabulary object detection model provided in an embodiment of the present application;

[0047] Figure 2 is a model training flow chart provided in an embodiment of the present application;

[0048] Figure 3 This is the improved model architecture diagram in this application;

[0049] Figure 4 It is a structural schematic diagram of an improved device for an open vocabulary object detection model provided in an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of an electronic device for implementing an improved method for an open vocabulary target detection model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the technical solution of the present application clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The terms "first", "second", etc. in the claims and specification of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is only to describe the distinction method used when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that a process, method, system, product or device containing a series of units is not necessarily limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] In some embodiments, Figure 1 , an improved method for an open vocabulary object detection model is provided, comprising the following steps:

[0053] S110, obtaining target training data, where the target training data includes a target training image, a target text label, and a first target frame corresponding to the target text label;

[0054] S120, randomly augmenting the target training image to obtain a target augmented image, and determining whether the target text label is unique in the training set to which the target training data belongs;

[0055] S130 , generating a prompt vector according to the judgment result, and inputting the prompt vector, the target training image and the target augmented image into an open vocabulary object detection model for training, thereby obtaining an improved open vocabulary object detection model.

[0056] First, a set of training data is extracted from the training set of the open vocabulary target detection model as the target training data. The target training data includes a target training image, a target text label, and a first target box corresponding to the target text label. It should be noted here that each set of training data in the training set actually includes a training image, a text label, and a target box corresponding to the text label. Among them, the training image is the image data used in the model training process, which contains the visual features that the model needs to learn. Inputting the training image into the model can help the model learn the shape, color, texture and other visual features of different objects, thereby realizing image classification, detection and other tasks; the text label is the text information associated with the training image, usually indicating the object in the image. The category or attribute of the object provides supervision information for the model, so that the model can know the category or attribute corresponding to each image during the training process, and thus learn how to classify or recognize the image; the target box, also known as the bounding box or the circumscribed rectangular box, is a rectangular area marked on the training image, which is used to describe the position and size of the object in the image. It provides the model with the location information of the object, so that the model can learn how to accurately locate the object in the image during the training process; at the same time, in order to distinguish the content contained in the target training data from the content contained in the training data, the training image, the text label and the target box corresponding to the text label in the training data are respectively referred to as the original training image, the first text label and the original target box in this embodiment.

[0057] Open Vocabulary Object Detection (OVOD) is a target detection task. Its core idea is to use the vision-language joint modeling method to associate visual features with language features, so as to realize the detection of unseen object categories. Currently, common open vocabulary target detection models include OVR-CNN, DetCLIPv3, GLEE and YOLO_WORLD. In this embodiment, YOLO_WORLD is preferred. In view of the fact that a single text phrase is usually used as a prompt label in the YOLO_WORLD algorithm, the target text label here is specifically the target text phrase label.

[0058] Next, the target training image is randomly augmented to obtain a target augmented map, which is similar to the target training image for subsequent self-supervised learning. The augmentation method can be clarity change, color space change or random masking, etc., which is not specifically limited here.

[0059] Then determine whether the target text label is unique in the training set. If the target text label is unique in the training set, it means that another image containing a similar target cannot be found in the training set through the dataset annotation. Then calculate the text feature vector of the target text label and use the text feature vector directly as the prompt vector. It is input into the YOLO_WORLD network model backbone with the target training image and the target augmented image for model forward reasoning. Backbone is a key concept in the field of deep learning, especially in convolutional neural networks (CNNs) that process images, videos, or other types of data. It refers to the main component of the network, responsible for extracting the main features of the data and preliminary processing. The backbone of the network model usually contains a series of convolutional layers, pooling layers, activation functions, and possible normalization layers, etc. These layers are stacked and connected to form the deep structure of the network. In this embodiment, the backbone of the YOLO_WORLD network model is a Darknet network based on YOLOv8; if the target text label is not unique in the training set, a prompt vector is generated based on the target text label and its corresponding original target boxes other than the first target box, and then the prompt vector is respectively input into the backbone of the YOLO_WORLD network model with the target training image and the target augmented image for iterative training, and finally the improved YOLO_WORLD network model is output. The training process of the model is as follows Figure 2 shown.

[0060] The YOLO_WORLD network model training method provided in this embodiment can improve the verification index effect of the YOLO_WORLD network model on the original verification set LVIS data set without increasing the original pre-training data set and without modifying the network structure.

[0061] In some embodiments, the method further comprises:

[0062] The original target frame information of each image in the training set is stored in the first list in the form of a dictionary, where the original target frame information includes an identifier of the original target frame and a first text label corresponding to the original target frame;

[0063] Each dictionary in the first list is traversed, and a label index table with the same name is constructed according to the traversal result, with the first text label as the key and the second list consisting of the identifiers of all original target boxes corresponding to the first text label as the value.

[0064] YOLO_WORLD uses multiple mixed datasets during model pre-training. Take the cc3m_lite dataset as an example. Its annotation file stores the original target box information of all images in the dataset in the first list A. Each element in the first list A is a dictionary with the same fields, representing an original target box B. i The following is an example of information:

[0065] {

[0066] "image_id": 123461,

[0067] "id": 400967,

[0068] "category_id": 352,

[0069] "area": ​​18910.30967394961,

[0070] "bbox": [

[0071] 180.27377319335938,

[0072] 76.97242736816406,

[0073] 117.61227416992188, 160.78517150878506

[0075] ],

[0076] "iscrowd": 0,

[0077] "score": 0.3002203106880188,

[0078] "tokens": "vector illustration"

[0079] }

[0080] Among them, image_id is the unique identifier ID of the training image in the training set, id is the unique identifier ID of the original target box in the entire training set, bbox is the coordinate representation of the original target box on the training image, and tokens is the first text label corresponding to the original target box. These contents are also included in the original target box information.

[0081] Before model improvement training, each dictionary element in the first list A is traversed to construct a tag index table with the same name. The essence of the tag index table with the same name is also a dictionary, where each key is a first text tag C. i, the value corresponding to each key is a second list, which stores all tokens fields equal to C i The original target box B i In layman's terms, each second list is a collection of identifiers of all original target boxes corresponding to a first text label.

[0082] In this embodiment, it is determined whether the target text label is unique in the training set to which the target training data belongs, specifically including: searching the value corresponding to the key being the target text label in the same-name label index table, obtaining a second list consisting of identifiers of all original target frames corresponding to the target text label, and recording the second list as the target second list, and then comparing all identifiers contained in the target second list with the identifier of the first target frame. If there is only one first target frame identifier in the target second list, it means that the target text label is unique in the training set to which the target training data belongs. At this time, the text feature vector corresponding to the target text label is the prompt vector. If there are other original target frame identifiers in addition to the first target frame identifier in the target second list, it means that the target text label is not unique in the training set to which the target training data belongs. At this time, a prompt vector is generated according to the target text label and its corresponding other original target frames except the first target frame.

[0083] Further, a prompt vector is generated according to the target text label and its corresponding original target frames other than the first target frame, including:

[0084] Generate a random number and compare it with a set threshold;

[0085] When the random number is greater than a set threshold, any original target frame except the first target frame in the second target list is randomly selected as the second target frame, and a local target image is cropped according to the second target frame;

[0086] The local target image is converted into a local image feature vector, and the local image feature vector is fused with the text feature vector to obtain a prompt vector.

[0087] First, a random number P is generated. In this embodiment, a random number P between 0 and 1 is generated using the random module of the Python language. Other methods are also possible and are not specifically limited. Then, it is determined whether the random number P is greater than a set threshold. Here, the threshold is set to be 0.65, or it can be other specific values, as long as the value can ensure that a certain amount of training data uses the intact text as input. If the random number P is greater than 0.65, any original target frame other than the first target frame is randomly selected from all target frames corresponding to the target text label, and the frame is recorded as the second target frame. A local target image is then cropped according to the image_id field and the bbox field corresponding to the second target frame, and the local target image is converted into a local image feature vector. Then, the local image feature vector and the text feature vector are fused to obtain a prompt vector. Specifically, the prompt vector G=p*T+(1-p)*M, wherein T represents the text feature vector corresponding to the target text label, and M represents the local image feature vector corresponding to the local target image. If the random number P is less than 0.65, the text feature vector corresponding to the target text label is still used as the prompt vector.

[0088] Finally, the obtained prompt vector is input into the backbone of the network model for training with the target training image and the target augmented image to obtain the network prediction output. Then, the network backbone parameters are optimized according to the loss function, and it is continuously iterated to output the improved YOLO_WORLD network model. Its model architecture is as follows: Figure 3 shown.

[0089] Furthermore, the local target image is converted into a local image feature vector, including:

[0090] Convert the channel order of the local target image to RGB, and convert the image matrix after the channel order conversion to a tensor data format;

[0091] Input the image in tensor data format into the visual encoder for feature extraction to obtain a feature matrix, and calculate the Euclidean norm of each vector on the last dimension of the feature matrix;

[0092] Each vector in the feature matrix is ​​divided by its Euclidean norm to obtain a normalized feature matrix, and all dimensions except the last dimension in the normalized feature matrix are merged into one dimension to obtain a local image feature vector.

[0093] For an image to be encoded, its format may be ".jpg", ".png", ".jpeg", etc., and the channel order of images in these formats may not be "RGB". Therefore, in this embodiment, the channel order of the local target image is first converted to "RGB" using the PIL library of the python language, and then the image matrix after the channel order conversion is converted into a tensor data format, that is, a tensor data format. In computer science and deep learning, tensor is a core concept, especially in deep learning frameworks such as PyTorch. Tensor is the most basic unit of operation, which can represent various types of data, including scalar (0-dimensional tensor), vector (1-dimensional tensor), matrix (2-dimensional tensor) and higher-dimensional arrays. In deep learning, tensors are often used for storage and operation. The weights, activation values, gradients and other information in the neural network are obtained, and the image in the tensor data format is input into the visual encoder for feature extraction to obtain its feature matrix D. In this embodiment, the visual encoder is a visual encoding module split from the contrastive language-image pre-training (CLIP) model. It uses hundreds of millions of training data during pre-training to align the features of the image with the features of the text description. Therefore, the use of this visual encoding module can make the feature vector encoded from the local target image have a certain similarity with the original label text, and it is suitable for pictures in formats such as JPG, PNG, and JPEG. Of course, other improved versions of CLIP such as Sigclip visual encoder can also be used. The specific visual encoder used can be adjusted according to actual conditions.

[0094] Next, calculate the Euclidean norm, i.e., the L2 norm, of each vector in the last dimension of the feature matrix D. Assume that the feature matrix D is a matrix of shape (m, n), where m is the number of rows, representing the number of data points, and n is the number of columns, representing the number of features of each data point. Then, each vector in the last dimension of the feature matrix D corresponds to the value of each feature in all data points. The essence of the L2 norm is the square root of the sum of the squares of all elements of the vector, which reflects the overall distance of each component of the vector relative to the origin. Divide each vector in the feature matrix D by the L2 norm to obtain a normalized feature matrix. Finally, merge all dimensions except the last dimension in the normalized feature matrix into one dimension to obtain a one-dimensional feature vector. This one-dimensional feature vector is the local image feature vector.

[0095] In this embodiment, an image similar to the target box is encoded by the CLIP visual encoder and then integrated into the text label feature, which enriches the semantic information of the label text. At the same time, the network model pays more attention to the similarities between images, strengthens the model's ability to represent images, and improves the convergence effect of the network.

[0096] In some embodiments, the method further comprises:

[0097] Self-supervised learning of open vocabulary object detection models using object augmentation graphs;

[0098] Among them, self-supervised learning includes:

[0099] Add a new loss function to the original loss function of the open vocabulary object detection model. The new loss function expression is:

[0100] ,

[0101] ,

[0102] in, ∈{0,1} is the indicator function, T is the temperature coefficient, x i and x j are the feature vectors of the target training image and the target augmented image obtained through network inference, the sim function is a function for calculating the similarity between two vectors, and N represents the number of images in a batch during model training.

[0103] In this embodiment, the target augmented image is mainly used for self-supervised learning of the model, and the self-supervised learning method used here is also the self-supervised learning method used in other models such as simCLR and SLIP. The core idea of ​​this method is that the feature encodings obtained after two similar images are input into the same network are similar. Therefore, if a network model encodes two similar images and obtains image features that are very different, it will be considered that the representation ability of the model is not good. At the same time, the self-supervised learning method in this embodiment is specifically to add a new loss function to the original loss function of the YOLO_WORLD network model. The new loss function expression is as follows:

[0104] ,

[0105] ,

[0106] in, ∈{0,1} is the indicator function, T is the temperature coefficient, x i and x j are the feature vectors of the target training image and the target augmented image obtained through network inference, the sim function is a function for calculating the similarity between two vectors, and N represents the number of images in a batch during model training.

[0107] By optimizing the backbone parameters of the network model through the calculated loss function and then using the model with optimized parameters for iterative training, the performance of the model can be improved to make it more accurate.

[0108] In some embodiments, Figure 4 As shown, an improved device for an open vocabulary object detection model is provided, comprising:

[0109] An acquisition module 10 is used to acquire target training data, where the target training data includes a target training image, a target text label, and a first target frame corresponding to the target text label;

[0110] A judgment module 20 is used to randomly augment the target training image to obtain a target augmented image, and to judge whether the target text label is unique in the training set to which the target training data belongs;

[0111] The training module 30 is used to generate a prompt vector according to the judgment result, and input the prompt vector, the target training image and the target augmented image into the open vocabulary object detection model for training, so as to obtain an improved open vocabulary object detection model.

[0112] An implementation of the above-mentioned device may be: an acquisition module 10 acquires target training data, the target training data including a target training image, a target text label and a first target frame corresponding to the target text label; a judgment module 20 randomly augments the target training image obtained by the acquisition module 10 to obtain a target augmented map, and judges whether the target text label is unique in the training set to which the target training data belongs; a training module 30 generates a prompt vector according to the judgment result of the judgment module 20, and inputs the prompt vector, the target training image and the target augmented map into an open vocabulary target detection model for training, thereby obtaining an improved open vocabulary target detection model.

[0113] In some embodiments, Figure 5 As shown, an electronic device is provided, including a memory 501 and a processor 502, wherein the memory 501 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 502 to implement the above-mentioned improved method for an open vocabulary target detection model.

[0114] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0115] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement the above-mentioned method for improving an open vocabulary object detection model when executed.

[0116] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 501 and executed by the processor 502, and the input interface 505 and the output interface 506 complete the I / O interface transmission of data to complete the present invention. The one or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0117] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a memory 501 and a processor 502. Those skilled in the art may understand that this embodiment is only an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include an input device 507, a network access device, a bus, etc.

[0118] The processor 502 may be a central processing unit (CPU), or other general-purpose processors 502, digital signal processors 502 (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 502 may be a microprocessor 502 or the processor 502 may also be any conventional processor 502, etc.

[0119] The memory 501 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. The memory 501 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 501 may also include both an internal storage unit of the computer device and an external storage device. The memory 501 is used to store computer programs and other programs and data required by the computer device. The memory 501 may also be used to temporarily store in the output device 508, and the aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory ROM503, a random access memory RAM504, a disk or an optical disk.

[0120] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An improved method for an open vocabulary object detection model, characterized in that: The following steps are involved: Acquire target training data, wherein the target training data includes a target training image, a target text label, and a first target frame corresponding to the target text label; Randomly augmenting the target training image to obtain a target augmented image, and determining whether the target text label is unique in the training set to which the target training data belongs; A prompt vector is generated according to the judgment result, and the prompt vector is respectively input into an open vocabulary object detection model with the target training image and the target augmented image for training to obtain an improved open vocabulary object detection model.

2. The method for improving an open vocabulary object detection model according to claim 1, characterized in that: The method further comprises: The original target frame information of each image in the training set is stored in a first list in the form of a dictionary, wherein the original target frame information includes an identifier of the original target frame and a first text label corresponding to the original target frame; Each dictionary in the first list is traversed, and a tag index table with the same name is constructed according to the traversal result, with the first text tag as the key and the second list consisting of identifiers of all original target frames corresponding to the first text tag as the value.

3. The method for improving an open vocabulary object detection model according to claim 2, characterized in that: The determining whether the target text label is unique in the training set to which the target training data belongs includes: Searching for an entry whose key is the target text label in the same-name label index table to obtain a second target list, wherein the second target list is a second list composed of identifiers of all original target frames corresponding to the target text label; It is determined whether the identifier of the first target box exists uniquely in the target second list. If so, the target text label is unique in the training set to which the target training data belongs. Otherwise, it is not unique.

4. The method for improving an open vocabulary object detection model according to claim 3, characterized in that: Generating a prompt vector according to the judgment result includes: When the judgment result is yes, calculating the text feature vector of the target text label, and using the text feature vector as a prompt vector; When the judgment result is no, a prompt vector is generated according to the target text label and other corresponding original target frames except the first target frame.

5. The method for improving an open vocabulary object detection model according to claim 4, characterized in that: The generating a prompt vector according to the target text label and other corresponding original target frames except the first target frame includes: Generate a random number, and compare the random number with a set threshold; When the random number is greater than the set threshold, randomly extract any original target frame in the target second list except the first target frame as the second target frame, and crop a local target image according to the second target frame; The local target image is converted into a local image feature vector, and the local image feature vector is fused with the text feature vector to obtain a prompt vector.

6. The method for improving an open vocabulary object detection model according to claim 5, characterized in that: The converting the local target image into a local image feature vector comprises: Convert the channel order of the local target image into RGB, and convert the image matrix after the channel order conversion into a tensor data format; Inputting the image in tensor data format into the visual encoder for feature extraction to obtain a feature matrix, and calculating the Euclidean norm of each vector in the last dimension of the feature matrix; Each vector in the feature matrix is ​​divided by its Euclidean norm to obtain a normalized feature matrix, and all dimensions except the last dimension in the normalized feature matrix are merged into one dimension to obtain a local image feature vector.

7. The method for improving an open vocabulary object detection model according to claim 1, characterized in that: The method further comprises: Using the target augmented graph to perform self-supervised learning of the open vocabulary target detection model; Wherein, the self-supervised learning includes: A new loss function is added to the original loss function of the open vocabulary object detection model. The new loss function expression is: ， ， in, ∈{0,1} is the indicator function, T is the temperature coefficient, x i and x j are the feature vectors of the target training image and the target augmented image obtained through network inference, the sim function is a function for calculating the similarity between two vectors, and N represents the number of images in a batch during model training.

8. An improved device for an open vocabulary object detection model, characterized in that: include: An acquisition module, used to acquire target training data, wherein the target training data includes a target training image, a target text label, and a first target frame corresponding to the target text label; A judgment module, used for randomly augmenting the target training image to obtain a target augmented image, and judging whether the target text label is unique in the training set to which the target training data belongs; The training module is used to generate a prompt vector according to the judgment result, and input the prompt vector, the target training image and the target augmented image into an open vocabulary object detection model for training to obtain an improved open vocabulary object detection model.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement an improved method for an open vocabulary object detection model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to implement an improved method for an open vocabulary object detection model as described in any one of claims 1 to 7 when executed.

Images