Training method, target detection method, and storage medium

Abstract

The present disclosure relates to a training method, a target detection method and a storage medium related to an open vocabulary target detection model. According to an embodiment, the training method is used for training an open vocabulary target detection model. The model comprises a prompt enhancement module, an input image encoding module and a detection module. The method comprises: generating, by the prompt enhancement module, an enhanced prompt feature based on a text description and a prompt image of a prompt category of a current iteration training round; generating, by the input image encoding module, an image feature of an input image of the current iteration training round; determining, by the detection module, a target category and a target position of a target in the input image based on the enhanced prompt feature and the image feature; and adjusting parameters of the model based on the determined target category and target position.

Description

Technical Field

[0001] This disclosure relates generally to computer vision, and more specifically to training methods for training models for open-vocabulary object detection, object detection methods, and computer-readable non-transitory storage media storing programs. Background Technology

[0002] With the development of computer science and artificial intelligence, using computers to run neural network-based artificial intelligence models for information processing is becoming increasingly common and effective. Computer vision is an important application area of ​​artificial intelligence models.

[0003] One application of computer vision technology is object detection, which involves identifying the category and location of objects in an image.

[0004] In traditional object detection tasks, each model is trained for a specific scene and a fixed set of categories. Although generalization performance is an important metric for evaluating model capability, fine-tuning remains the only feasible approach when encountering new categories in closed-set object detection tasks. This means that detecting a completely new category requires acquiring images of that category and performing very careful manual annotation on the dataset. Furthermore, when adding new categories, the distribution of the dataset must be carefully considered, as improper fine-tuning can lead to severe forgetting, causing the model to focus only on newly learned categories while ignoring previously learned ones.

[0005] Compared to traditional closed-set object detection, Open Vocabulary Object Detection (OVOD) can not only identify objects within the training categories but also detect objects outside the training categories. This enables the model to recognize objects it has never seen before, effectively alleviating the problem of repeatedly training the model to adapt to different detection tasks in different application scenarios. Summary of the Invention

[0006] A brief overview of this disclosure is provided below to offer a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the disclosure. It is not intended to identify key or essential parts of the disclosure, nor is it intended to limit its scope. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0007] The inventors proposed this invention after studying conventional open vocabulary target detection models.

[0008] According to one aspect of this disclosure, a training method is provided for training a model for open-vocabulary object detection. The model includes a cue enhancement module, an input image encoding module, and a detection module. The method includes: the cue enhancement module generating enhanced cue features based on a text description of the cue category and a cue image in the current training iteration; the input image encoding module generating image features of the input image in the current training iteration; the detection module determining the target category and target location of an object in the input image based on the enhanced cue features and the image features; and adjusting the parameters of the model based on the determined target category and target location.

[0009] According to one aspect of this disclosure, an object detection method is provided. The method includes: determining the category and location of an object in an image using an open-vocabulary object detection model trained using the training method described above.

[0010] According to another aspect of this disclosure, a computer-readable non-transitory storage medium storing a program is provided. When the program is executed by a computer, it enables the computer to implement the aforementioned training method followed by a target detection method.

[0011] The beneficial effects of the methods and storage media disclosed herein include: improving the performance of open vocabulary object detection models. Attached Figure Description

[0012] The embodiments of this disclosure are described below with reference to the accompanying drawings, which will help to more easily understand the above and other objects, features, and advantages of this disclosure. The drawings are only for illustrating the principles of this disclosure. The dimensions and relative positions of the elements are not necessarily drawn to scale in the drawings. The same reference numerals may denote the same features. In the drawings:

[0013] Figure 1 An exemplary flowchart of a training method for training an open vocabulary object detection model according to an embodiment of the present disclosure is shown;

[0014] Figure 2 An exemplary structural block diagram of an open vocabulary object detection model according to an embodiment of the present disclosure is shown;

[0015] Figure 3 An exemplary block diagram of an open vocabulary object detection model according to an embodiment of the present disclosure is shown;

[0016] Figure 4 An exemplary block diagram of a target detection apparatus according to an embodiment of the present disclosure is shown; and

[0017] Figure 5 This is an exemplary block diagram of an information processing device according to one embodiment of the present disclosure. Detailed Implementation

[0018] Exemplary embodiments of this disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0019] It should also be noted that, in order to avoid obscuring the contents of this disclosure with unnecessary details, only the device structure closely related to the solution according to this disclosure is shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0020] It should be understood that this disclosure is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0021] Computer program code used to perform the operations of various aspects of embodiments of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0022] The method disclosed herein can be implemented using circuitry with corresponding functional configurations. The circuitry includes circuitry for a processor.

[0023] One aspect of this disclosure relates to a training method for an Open Vocabulary Object Detection (OVOD) model. This method can be implemented using a computer. The OVOD model is a neural network model. The neural network model undergoes multiple rounds of iterative training to adjust network parameters and optimize performance. This model can be applied, for example, to fields such as autonomous driving, smart security, smart retail, robotics, image and video content understanding, and medical image analysis.

[0024] The following is for reference. Figure 1 and 2 The method is described exemplarily.

[0025] Figure 1 An exemplary flowchart of a training method 100 for training an open vocabulary object detection model according to an embodiment of the present disclosure is shown. Figure 2An exemplary structural block diagram of an open-vocabulary object detection model 200 according to one embodiment of the present disclosure is shown. Figure 2 As shown, model 200 includes a cue enhancement module 201, an input image encoding module 203, and a detection module 205. In one example, method 100 is implemented by a computer running a corresponding computer program.

[0026] In operation Op101, the cue enhancement module 201 generates enhanced cue features ehpF based on the text description tD0 of the cue category and the cue image pIm (also known as a visual cue, which can be represented as vP0 in this case) of the cue category in the current iteration of training. The cue image pIm is an image about the cue category. The cue category refers to the corresponding category of the target that the cue model wants to detect from the input image. In one example, the text description tD0 is expressed in natural language. For example, the text description tD0 can be the name of the cue category expressed in natural language. It is understood that during the training phase, the image pIm is the cue image used for training, so it has been labeled. The label includes: the target category of the target in the image (e.g., "cat") and the corresponding target location (e.g., the coordinates of the top-left and bottom-right corners of the bounding box of the target "cat" in the image). The cue image pIm can include multiple targets, and the target categories of the multiple targets can be different. Of course, in the testing phase (i.e., the phase where the model is used for inference after training), this cue image can be used as the visual cue input for the model for the same cue category.

[0027] Before training begins, the input image encoding module 203 and detection module 205 can be initialized as the input image encoding and detection modules of a pre-trained model of a conventional open-vocabulary object detection model. The pre-trained model is, for example, the LaMI-DETR (LaMI: Language Model Instruction; DETR: Detection Transformers) model. As a conventional open-vocabulary object detection model, LaMI-DETR classifies and locates objects in an image based on image features and cue features of the input image. The cue features are obtained by instructing GPT 3.5 to output a description of the target with the name of the category of interest, and then converting this description into features as cue features. GPT 3.5 is not open-source; the descriptions generated in this way are random, non-template-based, uncontrollable, and to some extent uninterpretable.

[0028] In operation Op103, the input image encoding module 203 generates the image features imF of the input image inIm for the current iteration of training. The input image inIm is the image to be detected. During training, the input image inIm is a sample image in the training set, which has also been labeled. The labeling includes: the target category of the object in the image (e.g., "cat") and the corresponding target location (e.g., the coordinates of the top-left and bottom-right corners of the bounding box of the target "cat" in the image). The input image imIm can include multiple targets, and the target categories of the multiple targets can be different.

[0029] In operation Op105, the detection module 205 determines the target category Ob.Cat and target location Ob.Loc of the target Ob in the input image inIm based on the enhanced cue feature ehpF and the image feature imF.

[0030] In operation Op107, the model parameters (including weight parameters) are adjusted based on the determined target category Ob.Cat and target location Ob.Loc. These parameters preferably include: parameters of the cue enhancement module, parameters of the input image encoding module, and parameters of the detection module. It is understood that this operation may, for example, include: determining the loss based on the determined target category Ob.Cat and target location Ob.Loc and the labeled content of the input image, and adjusting the model parameters based on the loss using gradient descent.

[0031] Training method 100 enhances the cueing ability of cue features by introducing visual cues on top of the textual description of the category and generating cue features based on both (i.e., generating enhanced cue features), thereby improving the detection of targets in some categories that are “difficult to describe clearly in text” (e.g., mechanical equipment with complex structures).

[0032] The following is for reference. Figure 3 Further details of training method 100 are described by way of example. Figure 3 An exemplary block diagram of an open vocabulary object detection model 300 according to an embodiment of the present disclosure is shown. The open vocabulary object detection model 300 includes a cue enhancement module 310, an input image encoding module 330, and a detection module 350. Additionally, Figure 3 At least one of the language model 301, text encoder 303 and visual encoder 305 shown can also be used as part of the object detection model, but the parameters of these parts are frozen during the training process, that is, the parameters of these parts are not adjusted.

[0033] To obtain the enhanced cue feature ehpF, the text description tD0 needs to be transformed into text cue features. A common transformation method is to provide the text cue, serving as the "target category name," to a language model (e.g., GPT, Generative Pre-trained Transformer), instructing the language model to generate a description of the object category (commonly, a description of the appearance of objects within that category), and then using an encoder to convert the generated description into cue features. In one embodiment of the invention, the generation of the enhanced cue feature ehpF by the cue enhancement module based on the text description tD0 and cue image pIm of the cue category in the current training iteration includes: the cue enhancement module sending a first instruction Int11 to the language model (e.g., GPT, more specifically, GPT 3.5): generating multiple text cue features describing the target category from multiple different target attributes based on the text description; and the cue enhancement module receiving the multiple text cue features from the language model. The target attribute can be represented as Pp[i], and the target attribute set consisting of multiple target attributes can be represented as: "{Pp[i]|i∈[iStart,iEnd]}" (or abbreviated as {Pp[i]}), where i is a natural number. The text prompt generated for the target attribute Pp[i] can be represented as "tP[i]", and the text prompt set consisting of multiple text prompts can be represented as {tP[i]}. The aforementioned operations can be performed by... Figure 3 The text prompt generation module 311 executes the following: The text prompt generation module 311 sends a first instruction to the language model 301 (e.g., GPT) to generate multiple text prompts for the target category, each describing a different target attribute, based on a text description, and receives the first instruction Int11 for the multiple text prompts from the language model. The text prompt generation module 311 receives a set of multiple text prompts {tP[i]} from the language model 301. In one example, the multiple different target attributes include: size, color, texture, shape, structural components, and common environment. The multiple different target attributes are multiple predetermined attributes that do not change with the prompt category. During training, the same set of target attributes is used to generate the text prompt set for different prompt categories. Compared to simply providing the category name to the language model without requiring the generated text description to specifically describe the target attribute category, generating text descriptions of target categories from multiple specified target attributes is beneficial for unifying and standardizing text descriptions, obtaining more sub-features of the target category, improving the object detection accuracy of the model, and improving model performance.

[0034] In one example, considering that the encoder used to transform text descriptions into text features may have token length limitations, the first instruction Int1 includes a requirement for the token length of each text prompt tP[i] in a set of multiple text prompts. For example, for the CLIP (Contrastive Language-Image Pre-training) text encoder, it requires that the input text tokens not exceed 77. Therefore, if the selected text encoder is the CLIP text encoder, the first instruction Int1 could include: the token length of the generated text prompts does not exceed 77. This prevents valuable information included in the generated text prompts from being truncated by the text encoder, thereby reducing model performance.

[0035] In one example, considering that the encoder used to transform text descriptions into text features may have a preference for the way the input text is represented, the first instruction Int1 includes a requirement for the encoder applicable to each of the multiple text prompts. The first instruction Int1 may include: the generated text prompts should be suitable for the CLIP text encoder. That is, for example, the CLIP text encoder is selected as the encoder for transforming the text prompts.

[0036] In one example, considering that the encoder used to transform the text description into text features may have a preference for the format of the input text, the first instruction Int1 includes requirements for the representation format of each of the multiple text prompts. The first instruction Int1 may include: generating text prompts in JSON (JavaScript Object Notation) format.

[0037] In one example, the first instruction Int1 includes: ensuring that each text prompt in a series of text prompts is concise and truthful.

[0038] A specific example of an instruction provided to GPT for the object category name "category_name" is shown in Table 1.

[0039] Table 1 provides example instructions to GPT for the object category name "category_name".

[0040]

[0041]

[0042] Another specific example of the instructions provided to GPT for the object category name "category_name" is shown in Table 2.

[0043] Table 2 provides example instructions to GPT for the object category name "category_name".

[0044]

[0045]

[0046] In one example, there are six different target attributes: size, color, texture, shape, structural components, and common environment; the cue category name is "cat". In this case, the six text cues generated by the language model GPT are shown in Table 3.

[0047] Table 3 provides six example text hints for the six target attributes.

[0048]

[0049] In one embodiment, the process of generating enhanced cue features ehpF by the cue enhancement module based on the text description tD0 of the cue category and the cue image pIm in the current training iteration round further includes: the cue enhancement module invoking the text encoder and receiving multiple corresponding text cue features of multiple text cuees generated by the text encoder. Each corresponding text cue feature can be represented as tpF[i], and the text cue feature set can be represented as {tpF[i]}. The aforementioned operations can be performed by... Figure 3 The text prompt feature generation module 312 executes the process. The text prompt feature generation module 312 sends a second instruction Int2 to the text encoder 303 to generate multiple corresponding text prompt features for multiple text prompts, and receives the generated multiple corresponding text prompt features from the text encoder 303. The text encoder 303 can be a CLIP text encoder, which can transform natural language text into vector features. During training, the network parameters of the text encoder 303 are frozen; that is, the network parameters of the text encoder 303 do not need to be adjusted.

[0050] In one embodiment, the generation of enhanced cue features by the cue enhancement module based on the text description of the cue category and the cue image in the current training iteration further includes: the cue enhancement module invoking the visual encoder and receiving at least one visual cue feature based on the cue image generated by the visual encoder. The visual cue feature can be represented as vpF. The aforementioned operation can be performed by... Figure 3The visual cue feature generation module 313 executes the process. The visual cue feature generation module 313 sends a third instruction Int3 to the visual encoder 305 to generate at least one visual cue feature, and receives at least one visual cue feature from the visual encoder, the at least one visual cue feature being based on the cue image pIm. The visual encoder 305 is, for example, a CLIP visual encoder, which can transform an image into vector features. During training, the network parameters of the visual encoder 305 are frozen, i.e., the network parameters of the visual encoder 305 do not need to be adjusted. The vector features generated by the text encoder 303 and the visual encoder 305 belong to the same feature space and have the same specifications.

[0051] In one embodiment, generating enhanced cue features by the cue enhancement module based on the text description and cue image of the cue category in the current iteration training round further includes: generating multiple embedded cue features by the embedding module of the cue enhancement module (e.g., based on a cross-attention mechanism) regarding multiple corresponding text cue features and at least one visual cue feature. The embedded cue features can be represented as epF[i], and the set of embedded cue features can be represented as {epF[i]}. The aforementioned operation can be performed by... Figure 3 The embedding module 314 is executed. The embedding module 314 generates, for example, multiple embedded cue features ({epF[i]}) about multiple corresponding text cue features and at least one visual cue feature based on a cross-attention mechanism. During embedding, tpF[i] acts as the "query," and the visual cue feature vpF acts as both the "key" and the "value." In this way, text cue features from different target attributes can focus on different parts of the visual cue feature, thereby achieving clear and targeted enhancement.

[0052] When multiple different target attributes include common environmental target attributes, in order to obtain more environmental cue information, the image used to generate the visual cue feature for that target attribute preferably includes more background area. In one embodiment, the multiple different target attributes include common environmental target attributes and a first other target attribute. The multiple corresponding text cue features received by the text cue generation module 311 include a first text cue feature corresponding to the first other target attribute and a second text cue feature corresponding to the common environmental target attribute. The visual encoder generates a first visual cue feature based on a first cropped image patch of the target in the cue image. The visual encoder generates a second visual cue feature based on a second cropped image patch of the target in the cue image. The second cropped image patch includes more background area than the first cropped image patch. For example, when multiple target attributes include "common environmental target attributes" and other target attributes, the visual cue feature generation module 313 can crop two sizes of image patches including the target from the cue image pIm. The first image patch of the first size is smaller, with the target in the foreground and less background area, while the second image patch of the second size is larger, with the target in the foreground and more background area than the background area of ​​the first image frame. For example, the area of ​​the second image patch is twice the area of ​​the first image patch. The visual cue feature generation module 313 aligns the centers of the first and second image patches and scales them to the same size (denoted as "first cue image" and "second cue image"), and then provides them to the visual encoder 305 to generate a first visual cue feature vpF1 for other target attributes and a second visual cue feature vpF2 for common environmental target attributes. The embedding module 314 generates a text cue feature tpF[i] for "other target attributes" (e.g., "color target attribute") and a first embedded cue feature epF[i] for "other target attributes" of the first visual cue feature vpF1 based on a cross-attention mechanism; the embedding module 314 also generates a text cue feature tpF[i'] for "common environmental target attributes" and a second embedded cue feature epF[i'] for "common environmental target attributes" of the second visual cue feature vpF2 based on a cross-attention mechanism. All other target attribute embedded cue features can be generated based on the same visual cue feature (e.g., the first visual cue image), relative to the second visual cue image used to generate the second embedded cue feature, which contains more background information.

[0053] In one embodiment, the process of generating enhanced cue features ehpF by the cue enhancement module based on the text description tD0 of the cue category and the cue image pIm in the current iteration of training rounds further includes: the cue enhancement module determining the enhanced cue features by fusing multiple embedded cue features. The aforementioned operations can be performed by... Figure 3The fusion module 315 is executed. The fusion module 315 determines the enhanced cue feature ehpF by fusing multiple embedded cue features ({epF[i]}). For example, the fusion module 315 may use a weighted sum of multiple embedded cue features as the enhanced cue feature ehpF, or calculate the arithmetic mean of multiple features from the fusion module 315 as the enhanced cue feature ehpF.

[0054] In this invention, image features can be generated based on a transformer encoder algorithm. In one embodiment, generating image features imF of the input image inIm for the current iteration training round by the input image encoding module includes: the image encoding module of the input image encoding module calling the visual encoder and receiving the visual features vF of the input image generated by the visual encoder; and the transformer encoder of the input image encoding module transforming the visual features into image features. Figure 3 As shown, the image encoding module 331 sends a fourth instruction Int4 to the visual encoder 305 to generate visual features vF of the input image inIm; and receives visual features vF from the visual encoder 305. The Transformer Encoder 333 of the input image encoding module 330 transforms the visual features vF into image features imF.

[0055] In one embodiment, the detection module determines the target category and target location of a target in the input image based on enhanced cue features and image features, including: a feature injection module of the detection module (see...). Figure 3 The injection module 351 injects the enhanced cue feature ehpF into the image feature imF as the injected feature itF; the transformer decoder of the detection module (see Figure 3 The Transformer Decoder (353) in the module transforms the injected feature itF as the transformer feature tF (i.e., the injected feature itF is considered a "query" for the Transformer Decoder); the classification module of the detection module (see...) Figure 3 The classification module 355 determines the target category Ob.Cat of the target in the input image inIm based on the transformer feature tF; and the localization module of the detection module (see Figure 3The localization module 357 determines the target location Ob.Loc of the target in the input image based on the transformer features. For example, the localization module 357 is a feedforward neural network (FNN) as a multilayer perceptron (MLP). For example, this feedforward neural network includes three fully connected layers. This feedforward neural network can determine the target location of the target in the input image based on the transformer features tF. For example, the classification module 355 can determine the target category by performing a similarity calculation operation and a normalization operation. In the similarity calculation operation, the classification module 355 calculates the similarity between the transformer features tF and the pre-injection features A, for example, by using an inner product dot product operation. The pre-injection features A are pooled features of some or all of the enhanced cue features ehpF, multiple text cue features, or one or more text cue features. In the normalization operation, the classification module 355 normalizes the calculated similarity using a softmax function to obtain the scores of each candidate category, and then determines the category of the target in the input image based on these scores.

[0056] In one embodiment, enhanced cue features can be used to select the most relevant image patches in the input image. In one example, the detection module is configured to: select a subset of image patches from multiple image patches in the cue image based on enhanced cue features; and classify the targets in the subset of image patches. For example, the detection module selects a subset of image patches from multiple candidate image patches in the cue image for classification and localization based on the similarity between enhanced cue features and image features. For example, image features imF include a first number (e.g., 5000) of candidate image patch features corresponding to candidate image patches; injection module 351 calculates the similarity between these candidate image patch features and enhanced cue features (or a weighted sum of pure text cue features) (high similarity means that the candidate image patch corresponding to the candidate image patch feature may contain targets of that category); injection module 351 selects the features of the top N (e.g., 900) candidate image patches with similarity greater than a predetermined similarity threshold and injects them with enhanced cue features ehpF as the query input itF of converter decoder 353 (i.e., equivalent to discarding candidate image patches with low similarity); this selection can improve computational efficiency and improve the final classification and localization performance.

[0057] In one embodiment, the fusion module can determine the enhanced cue feature by fusing multiple embedded cue features through average pooling, or by fusing multiple embedded cue features through weighted pooling, or by fusing multiple embedded cue features through attention pooling. The weights for weighted pooling are determined using a multilayer perceptron (MLP). The parameters of the attention pooling algorithm can be determined through learning.

[0058] In one embodiment, the classification module can be configured to classify based solely on the visual cue feature vpF, one feature from the text cue feature set {tpF[i]}, and the average pooling feature of some or all of the features in the text cue feature set {tpF[i]}, rather than based on all the information included in the transformer features. In one example, the classification module classifies by comparing only one feature from the text cue feature set {tpF[i]}, the average pooling feature of some or all of the features in the text cue feature set {tpF[i]}, and the image features included in the transformer features, to improve the stability of the training process.

[0059] In one embodiment, the injection module can obtain the injected feature itF by directly summing the enhanced cue feature ehpF and the image feature imF, or by summing the enhanced cue feature ehpF and the image feature imF based on a gating mechanism. The gating ratio can be calculated by embedding image patch features from the image features and the cue into the input MLP.

[0060] The trained OVOD model can be used for open-vocabulary object detection. One aspect of the invention relates to an object detection method. The method includes: determining the category and location of an object in an image using an open-vocabulary object detection model trained using the training method of the invention. In one example, during detection, the model's input includes: the image to be detected, a textual description of the cue category, and a cue image. The image can come from one of the following domains: autonomous driving, smart security, smart retail, robotics, image and video content understanding, and medical image analysis.

[0061] Alternatively, during the inference phase (i.e., the phase of using the trained model to detect the input image), the cue enhancement module 310 of the present invention can first calculate and save the enhanced cue features of all categories based on the text descriptions and cue images of all known categories during model initialization. This can be used to select the corresponding saved enhanced cue features for classification and localization during the subsequent inference and detection process for a known category, instead of calculating the corresponding enhanced cue features after each image to be detected is input into the model.

[0062] This invention also relates to a computer-readable non-transitory storage medium storing a program. When the program is executed by a computer, it causes the computer to implement the aforementioned training method or target detection method of this invention.

[0063] According to one aspect of the present invention, a target detection device is provided. Figure 4 An exemplary block diagram of a target detection apparatus 400 according to one embodiment of the present disclosure is shown.

[0064] The target detection apparatus 400 includes: a memory 401 storing instructions Inst; and at least one processor 403. The at least one processor is configured to execute the instructions Inst to implement the target detection method of the present invention.

[0065] According to one aspect of this disclosure, an information processing device is also provided.

[0066] Figure 5 This is an exemplary block diagram of an information processing device 500 according to one embodiment of the present disclosure. Figure 5 In this system, the central processing unit 501 performs various processes based on programs stored in the read-only memory 502 or programs loaded from the storage device 508 into the random access memory 503. The RAM 503 also stores data required by the CPU 501 for performing various processes, as needed.

[0067] CPU 501, ROM 502 and RAM 503 are interconnected via bus 504. Input / output interface 505 is also connected to bus 504.

[0068] The following components are connected to the input / output interface 505: an input device 506 including a soft keyboard, etc.; an output device 507 including a display such as an LCD and a speaker, etc.; a storage device 508 such as a hard disk; and a communication device 509 including a network interface card such as a LAN card, a modem, etc. The communication device 509 performs communication processing via a network such as the Internet, a local area network, a mobile network, or a combination thereof.

[0069] The driver 510 is also connected to the input / output interface 505 as needed. A removable medium 511, such as a semiconductor memory, is installed on the driver 510 as needed, so that programs read from it can be installed into the storage device 508 as needed.

[0070] CPU501 can run programs corresponding to the training method of this disclosure, or programs corresponding to the object detection method of this disclosure.

[0071] The inventors tested the model trained using the training method of this invention, and the test results are shown in Table 4. The dataset used was the open-source dataset LVIS for open vocabulary object detection. As shown in Table 4, the model achieved the best performance when textual cues for the six target attributes and cue images were used together.

[0072] Table 4. Effects of the present invention

[0073]

[0074]

[0075] In addition, the inventors also tested the models trained by the training method of this invention using their own datasets ("Brand Package Dataset" and "Garbage Overflow Dataset"), and the average accuracy reached 72.15 and 71.19, respectively.

[0076] The training method based on this disclosure can improve the performance of open vocabulary object detection models.

[0077] As described above, the principles of object detection have been disclosed according to this disclosure. It should be noted that the effects of the solutions disclosed herein are not necessarily limited to those described above, and any effect shown in this specification or other effects that can be understood from this specification can be achieved in addition to or in place of the effects described in the preceding paragraphs.

[0078] Although the invention has been disclosed above through a description of specific embodiments, it should be understood that those skilled in the art can devise various modifications (including, where applicable, combinations or substitutions of features between embodiments), improvements, or equivalents to the invention within the scope of the appended claims. These modifications, improvements, or equivalents should also be considered to be included within the scope of this disclosure.

[0079] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0080] Furthermore, the methods of the various embodiments of the present invention are not limited to being performed in the chronological order described in the specification or shown in the drawings, but may also be performed in other chronological orders, in parallel, or independently. Therefore, the execution order of the methods described in this specification does not constitute a limitation on the technical scope of the present invention.

[0081] Postscript

[0082] This disclosure includes, but is not limited to, the following schemes.

[0083] 1. A training method for training a model for open-vocabulary object detection, said model comprising a cue enhancement module, an input image encoding module, and a detection module, characterized in that the training method comprises:

[0084] The prompt enhancement module generates enhanced prompt features based on the text description and prompt image of the prompt category in the current iteration training round;

[0085] The input image encoding module generates the image features of the input image for the current iteration of training round;

[0086] The detection module determines the target category and target location in the input image based on the enhanced cue features and the image features; and

[0087] The parameters of the model are adjusted based on the determined target category and target location.

[0088] 2. According to the training method described in Appendix 1, the generation of enhanced prompt features by the prompt enhancement module includes:

[0089] The prompt enhancement module sends a first instruction to the language model to generate multiple text prompts for the target of the prompt category based on the text description, and receives the multiple text prompts from the language model.

[0090] 3. The training method according to Appendix 2, wherein the first instruction includes a requirement for the token length of each of the plurality of text prompts.

[0091] 4. The training method according to Appendix 2, wherein the first instruction includes requirements for the encoder applicable to each of the plurality of text prompts.

[0092] 5. The training method according to Appendix 2, wherein the first instruction includes requirements for the expression format of each of the plurality of text prompts.

[0093] 6. The training method according to Appendix 5, wherein the expression format is JSON format.

[0094] 7. The training method according to Appendix 4, wherein the encoder is a CLIP text encoder.

[0095] 8. The training method according to Appendix 2, wherein the plurality of different target attributes include: size, color, texture, shape, structural components and common environment.

[0096] 9. The training method according to Appendix 2, wherein the first instruction includes: ensuring that each of the plurality of text prompts is concise and authentic.

[0097] 10. According to the training method described in Appendix 2, the generation of enhanced prompt features by the prompt enhancement module further includes:

[0098] The prompt enhancement module sends a second instruction to the text encoder to generate multiple corresponding text prompt features for the multiple text prompts, and receives the multiple corresponding text prompt features from the text encoder.

[0099] 11. According to the training method described in Appendix 10, the generation of enhanced prompt features by the prompt enhancement module further includes:

[0100] The visual cue feature generation module of the cue enhancement module sends a third instruction to the visual encoder to generate at least one visual cue feature, and receives the at least one visual cue feature from the visual encoder.

[0101] The at least one visual cue feature is based on the cue image.

[0102] 12. The training method according to Appendix 11, wherein the plurality of different target attributes include common environmental target attributes and a first other target attribute;

[0103] The visual cue feature generation module sends a first cue image based on a first cropped image block of a target in the cue image to the visual encoder so that the visual encoder generates a first visual cue feature based on the first cue image.

[0104] The visual cue feature generation module sends a second cue image to the visual encoder based on a second cropped image patch of a target in the cue image, so that the visual encoder generates a second visual cue feature based on the second cue image; and

[0105] The second cropped image block includes more background area than the first cropped image block.

[0106] 13. According to the training method described in Appendix 11, the generation of enhanced prompt features by the prompt enhancement module further includes:

[0107] The embedding module of the aforementioned prompt enhancement module generates multiple embedded prompt features based on a cross-attention mechanism, which relate to the multiple corresponding text prompt features and at least one visual prompt feature.

[0108] 14. The training method according to Appendix 13, wherein the plurality of different target attributes include common environmental target attributes and a first other target attribute;

[0109] The plurality of corresponding text prompt features include a first text prompt feature corresponding to the first other target attribute and a second text prompt feature corresponding to the common environmental target attribute;

[0110] The visual cue feature generation module sends a first cue image based on a first cropped image block of a target in the cue image to the visual encoder so that the visual encoder generates a first visual cue feature based on the first cue image.

[0111] The visual cue feature generation module sends a second cue image based on a second cropped image block of a target in the cue image to the visual encoder so that the visual encoder generates a second visual cue feature based on the second cue image.

[0112] The second cropped image block includes more background area than the first cropped image block;

[0113] The embedding module generates a first embedded cue feature based on the cross-attention mechanism, relating the first text cue feature and the first visual cue feature; and

[0114] The embedding module generates a second embedded cue feature based on the cross-attention mechanism, relating the second text cue feature and the second visual cue feature.

[0115] 15. The training method according to Appendix 13, wherein the generation of enhanced cue features by the cue enhancement module further includes:

[0116] The fusion module of the aforementioned prompt enhancement module determines the enhanced prompt feature by fusing the multiple embedded prompt features.

[0117] 16. According to the training method described in Appendix 1, the image features of the input image generated by the input image encoding module for the current iteration training round include:

[0118] The image encoding module of the input image encoding module sends a fourth instruction to the visual encoder to generate visual features of the input image, and receives the visual features from the visual encoder; and

[0119] The image encoding module's converter encoder transforms the visual features into image features.

[0120] 17. According to the training method described in Appendix 1, wherein the determination of the target category and target location of the target in the input image by the detection module based on the enhanced cue features and the image features includes:

[0121] The injection module of the detection module injects the enhanced prompt features into the image features as injected features;

[0122] The converter decoder of the detection module transforms the injected features into converter features;

[0123] The classification module of the detection module determines the target category of the target in the input image based on the transformer features; and

[0124] The localization module of the detection module determines the target position of the target in the input image based on the transformer features.

[0125] 18. The training method according to Appendix 17, wherein the detection module is configured to:

[0126] Based on the enhanced prompt features, select a portion of image patches from multiple image patches in the prompt image; and

[0127] The targets in the aforementioned image blocks are classified.

[0128] 19. A target detection method, comprising:

[0129] The open-vocabulary object detection model, trained according to the training method described in Appendix 1, is used to determine the category and location of objects in an image.

[0130] 20. A computer-readable non-transitory storage medium storing a program, characterized in that, when the program is executed by a computer, the program causes the computer to implement the training method according to Appendix 1 or the detection method according to Appendix 19.

Claims

1. A training method for training a model for open-vocabulary object detection, said model comprising a cue enhancement module, an input image encoding module, and a detection module, characterized in that, The training method includes: The prompt enhancement module generates enhanced prompt features based on the text description and prompt image of the prompt category in the current iteration training round; The input image encoding module generates the image features of the input image for the current iteration of training round; The detection module determines the target category and target location in the input image based on the enhanced cue features and the image features; and The parameters of the model are adjusted based on the determined target category and target location.

2. The training method according to claim 1, wherein, The enhanced prompt features generated by the prompt enhancement module include: The prompt enhancement module sends a first instruction to the language model to generate multiple text prompts for the target of the prompt category based on the text description, and receives the multiple text prompts from the language model.

3. The training method according to claim 2, wherein, The enhanced prompt features generated by the prompt enhancement module also include: The prompt enhancement module sends a second instruction to the text encoder to generate multiple corresponding text prompt features for the multiple text prompts, and receives the multiple corresponding text prompt features from the text encoder.

4. The training method according to claim 3, wherein, The enhanced prompt features generated by the prompt enhancement module also include: The visual cue feature generation module of the cue enhancement module sends a third instruction to the visual encoder to generate at least one visual cue feature, and receives the at least one visual cue feature from the visual encoder. The at least one visual cue feature is based on the cue image.

5. The training method according to claim 4, wherein the plurality of different target attributes include common environmental target attributes and a first other target attribute; The visual cue feature generation module sends a first cue image based on a first cropped image block of a target in the cue image to the visual encoder so that the visual encoder generates a first visual cue feature based on the first cue image. The visual cue feature generation module sends a second cue image to the visual encoder based on a second cropped image patch of a target in the cue image, so that the visual encoder generates a second visual cue feature based on the second cue image; and The second cropped image block includes more background area than the first cropped image block.

6. The training method according to claim 4, wherein the generation of enhanced cue features by the cue enhancement module further includes: The embedding module of the aforementioned prompt enhancement module generates multiple embedded prompt features based on a cross-attention mechanism, which relate to the multiple corresponding text prompt features and at least one visual prompt feature.

7. The training method according to claim 6, wherein the plurality of different target attributes include common environmental target attributes and a first other target attribute; The plurality of corresponding text prompt features include a first text prompt feature corresponding to the first other target attribute and a second text prompt feature corresponding to the common environmental target attribute; The visual cue feature generation module sends a first cue image based on a first cropped image block of a target in the cue image to the visual encoder so that the visual encoder generates a first visual cue feature based on the first cue image. The visual cue feature generation module sends a second cue image based on a second cropped image block of a target in the cue image to the visual encoder so that the visual encoder generates a second visual cue feature based on the second cue image. The second cropped image block includes a larger background area than the first cropped image block; The embedding module generates a first embedded cue feature based on the cross-attention mechanism, relating the first text cue feature and the first visual cue feature; and The embedding module generates a second embedded cue feature based on the cross-attention mechanism, relating the second text cue feature and the second visual cue feature.

8. The training method according to claim 6, wherein, The enhanced prompt features generated by the prompt enhancement module also include: The fusion module of the aforementioned prompt enhancement module determines the enhanced prompt feature by fusing the multiple embedded prompt features.

9. A target detection method, characterized in that, include: The open-vocabulary object detection model trained using the training method described in claim 1 determines the category and location of objects in an image.

10. A computer-readable non-transitory storage medium storing a program, characterized in that, When the program is executed by a computer, the program causes the computer to implement the training method according to claim 1 or the detection method according to claim 9.

Images