Image detection method and device, model training method and device, equipment, storage medium and product

Abstract

Embodiments of the present application disclose image detection methods, model training methods and devices, equipment, storage media and products. The image detection method comprises: obtaining a to-be-detected image and a query text, on the one hand, performing open-vocabulary object detection on the to-be-detected image based on the query text to obtain a set of object position indication information, and on the other hand, extracting global features of the to-be-detected image; then, according to the obtained global features and the set of object position indication information, local features of the to-be-detected image are extracted, and according to the global features, the local features and the query text, a target detection result of the query text in the to-be-detected image is obtained. It can be seen that by decoupling low-level positioning (obtaining the set of object position indication information) and high-level understanding (extracting global features), and performing open-vocabulary object detection on the to-be-detected image in the low-level positioning process, the object position indication information in the set of object position indication information can not be limited to fixed categories, thereby realizing target detection of the image.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an image detection method, a model training method, an image detection device, a model training device, a computer device, a computer-readable storage medium, and an image detection product and a model training product. Background Technology

[0002] With advancements in scientific research, machine learning and artificial intelligence technologies have developed rapidly and are widely applied in various aspects of daily life, such as intelligent question answering, artistic creation, and image detection. In the field of image detection, image detection models can quickly identify the objects to be detected (such as people, objects, etc.) from images. Research has found that image detection models are typically based on visual localization techniques, making it difficult to perform object detection in images (e.g., difficulty in identifying objects that are not present in the image). Summary of the Invention

[0003] This application provides an image detection method, a model training method, an apparatus, a device, a computer-readable storage medium, and a product capable of performing target detection on images.

[0004] On one hand, embodiments of this application provide an image detection method, including:

[0005] Obtain the image to be detected and the query text, wherein the query text is associated with at least one associated object;

[0006] Open-vocabulary object detection is performed on the image to be detected based on the query text, resulting in a set of object location indication information; each object location indication information in the set corresponds to an associated object.

[0007] Extract global features from the image to be detected, and extract local features from the image to be detected based on the global features and the object location indication information set;

[0008] Based on global features, local features, and query text, the target detection results of the query text in the image to be detected are obtained.

[0009] In this embodiment, the process involves acquiring the image to be detected and the query text. Firstly, open-vocabulary object detection is performed on the image to be detected based on the query text to obtain a set of object location indication information. Secondly, global features of the image to be detected are extracted. Then, based on the obtained global features and the set of object location indication information, local features of the image to be detected are extracted. Finally, based on the global features, local features, and the query text, the object detection result of the query text in the image to be detected is obtained. It is evident that by decoupling low-level localization (obtaining the set of object location indication information) and high-level understanding (extracting global features), and by performing open-vocabulary object detection on the image to be detected during the low-level localization process, the object location indication information in the set of object location indication information is not limited to a fixed category, thereby achieving object detection in the image.

[0010] On the one hand, embodiments of this application provide a model training method, including:

[0011] Obtain the sample image, sample query text, and the corresponding verification result of the sample query text. The sample query text is associated with at least one associated object.

[0012] Open-vocabulary object detection is performed on sample images based on sample query text to obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object.

[0013] Extract global features from the sample image, and extract local features from the sample image based on the global features and the object location indication information set;

[0014] The model to be trained is invoked to use the sample query text as the detection condition. Based on global and local features, open domain target detection is performed on the sample image to obtain the target detection results of the sample query text in the sample image.

[0015] Based on the target detection results and the verification results, the model to be trained is adjusted to obtain the image detection model.

[0016] In this embodiment, a sample image, sample query text, and corresponding verification results are obtained. On one hand, open-domain object detection is performed on the sample image based on the sample query text to obtain a set of object location indication information. On the other hand, global features of the sample image are extracted. Then, based on the obtained global features and the set of object location indication information, local features of the sample image are extracted. The model to be trained is then called, using the sample query text as a detection condition. Open-domain object detection is performed on the sample image based on the global and local features to obtain the object detection results of the sample query text in the sample image. Based on the object detection results and verification results, the model to be trained is adjusted to obtain an image detection model. It can be seen that during model training, by decoupling low-level localization (obtaining the set of object location indication information) and high-level understanding (extracting global features), the model to be trained can be trained more specifically. By performing open-domain object detection on the image to be detected during low-level localization, the object location indication information in the set of object location indication information is not limited to fixed categories, enabling the image detection model to possess object detection capabilities.

[0017] On one hand, embodiments of this application provide an image detection device, which includes:

[0018] The acquisition unit is used to acquire the image to be detected and the query text, wherein the query text is associated with at least one associated object;

[0019] The processing unit is used to perform open-vocabulary target detection on the image to be detected based on the query text, and obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object;

[0020] And to extract global features of the image to be detected, and to extract local features of the image to be detected based on the global features and the object position indication information set;

[0021] It is also used to obtain the target detection results of the query text in the image to be detected based on global features, local features and query text.

[0022] In one implementation, the processing unit is configured to perform open-vocabulary target detection on the image to be detected based on the query text, and obtain a set of object location indication information, specifically configured to:

[0023] Perform semantic analysis on the query text to obtain the vocabulary corresponding to each associated object;

[0024] Based on the semantic features of each word and the image to be detected, candidate location indication information and the confidence level of each candidate location indication information are determined, with each word corresponding to at least one candidate location indication information.

[0025] The candidate location indication information is filtered by a confidence threshold to obtain a set of object location indication information.

[0026] In one implementation, the global features of the image to be detected include global features of the image at at least two scaling scales; the local features of the image to be detected correspond one-to-one with the object location indication information in the object location indication information set; the processing unit is used to extract the local features of the image to be detected based on the global features and the object location indication information set, specifically for:

[0027] From the global features at each scaling scale, extract the feature components corresponding to the position indication information of each object to obtain the feature components corresponding to the position indication information of each object at different scaling scales.

[0028] The feature components corresponding to the location indication information of each object at different scaling scales are fused to obtain the local features corresponding to each object location indication information.

[0029] In one implementation, the processing unit is configured to extract feature components corresponding to each object position indication from the global features at each scaling scale, thereby obtaining the feature components corresponding to each object position indication at different scaling scales, specifically configured to:

[0030] The global features at the target scaling scale are segmented to obtain M feature units. The target scaling scale is any one of at least two scaling scales, and M is an integer greater than 1.

[0031] If the area indicated by the target object location indication information is contained within a feature unit, then based on that feature unit, the feature component corresponding to the target object location indication information at the target scaling scale is determined, and the target object location indication information is any one of the object location indication information in the set of object location indication information.

[0032] If the area indicated by the target object location indication information is contained within at least two feature units, then based on the at least two feature units, the feature components corresponding to the target object location indication information at the target scaling scale are determined.

[0033] In one implementation, the processing unit is configured to obtain the target detection result of the query text in the image to be detected based on global features, local features, and query text, specifically for:

[0034] The input sequence of the image to be detected is generated based on global and local features;

[0035] By using the query text as the detection condition, open domain target detection is performed on the input sequence to obtain the target detection results of the query text in the image to be detected.

[0036] In one implementation, the input sequence includes N tokens, each token corresponding to a local feature, where N is an integer greater than 1. The target detection result is obtained based on the prediction results of the N tokens. The process by which the processing unit uses the query text as a detection condition to perform open-domain target detection on the input sequence and obtain the target detection result of the query text in the image to be detected includes:

[0037] Based on global features and the prediction results of the first i tokens, the prediction result of the (i+1)th token is generated, where i is a positive integer less than N.

[0038] In one implementation, the global features of the image to be detected include global features of the image at at least two scaling scales; the processing unit is configured to generate an input sequence of the image to be detected based on the global features and local features, specifically for:

[0039] The global feature projection layer projects the global features of the image to be detected at the target scaling scale into the text feature space to obtain the first projection feature. The target scaling scale is any one of at least two scaling scales.

[0040] The local feature projection layer projects local features onto the text feature space to obtain second projected features. Each second projected feature corresponds to a local feature. The projection parameters of the local feature projection layer are different from those of the global feature projection layer.

[0041] The first projection feature and the second projection feature are combined to obtain the input sequence of the image to be detected.

[0042] In one implementation, the query text is descriptive information of the query object; the target detection result of the query text in the image to be detected is used to indicate whether the image to be detected contains the query object;

[0043] If the target detection result indicates that the image to be detected contains a query object, the target detection result also includes object location indication information corresponding to the query object, which is used to indicate the position of the query object in the image to be detected.

[0044] On one hand, embodiments of this application provide a model training apparatus, which includes:

[0045] The acquisition unit is used to acquire a sample image, a sample query text, and the corresponding verification result of the sample query text. The sample query text is associated with at least one associated object.

[0046] The processing unit is used to perform open-vocabulary target detection on the sample image based on the sample query text to obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object.

[0047] And to extract global features from sample images, and to extract local features from sample images based on the set of global features and object location indication information;

[0048] And it is used to call the model to be trained to use the sample query text as the detection condition, perform open domain target detection on the sample image based on global and local features, and obtain the target detection result of the sample query text in the sample image;

[0049] And it is used to adjust the training model based on the target detection results and verification results to obtain the image detection model.

[0050] In one implementation, the processing unit is configured to acquire a sample image, sample query text, and the corresponding verification result of the sample query text, specifically for:

[0051] Obtain a set of region indication information for the sample image and the sample object. The set of region indication information is used to indicate the region where the sample object is located in the sample image.

[0052] Based on the sample image, generate descriptive information about the content of the target region, where the target region is the region indicated by any one of the region indication information in the set of region indication information;

[0053] The description information is broken down into conditions to obtain a list of conditions for the description information;

[0054] Using sample images, a set of region indication information, and a list of conditions, sample query text and corresponding verification results are generated.

[0055] In one implementation, the processing unit is configured to generate sample query text and corresponding verification results from the sample image, the set of region indication information, and the condition list, specifically for:

[0056] The region indicated by each region indication information is matched and detected with the conditions in the condition list to obtain the matching result of each region under each condition. Each matching result has a corresponding judgment basis.

[0057] If the matching result of any region is true under all conditions, then the region indication information corresponding to that region is added to the sample region indication information set corresponding to the description information.

[0058] The sample query text is generated based on the descriptive information, and the corresponding verification result is generated based on the sample region indication information set.

[0059] In one implementation, the target detection result is obtained based on the prediction results of N tokens, where the prediction result of each token indicates whether the corresponding local features satisfy the detection condition, and N is an integer greater than 1; the processing unit is used to adjust the model to be trained based on the target detection result and the verification result to obtain an image detection model, specifically for:

[0060] Based on the verification results and the prediction results of N tokens, the conditional probability of each token is calculated.

[0061] Construct the cross-entropy loss function using the conditional probabilities of N tokens;

[0062] The image detection model is obtained by adjusting the model to be trained based on the cross-entropy loss function.

[0063] Accordingly, this application provides a computer device comprising:

[0064] Memory, which stores computer programs;

[0065] A processor is used to load computer programs to implement the image detection method or the model training method described above.

[0066] Accordingly, this application provides a computer-readable storage medium storing a computer program that is adapted to be loaded by a processor and executed by the above-described image detection method, or adapted to be loaded by a processor and executed by the above-described model training method.

[0067] Accordingly, this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned image detection method or the aforementioned model training method.

[0068] In the image detection process, this application decouples low-level localization (obtaining the object location indication information set) from high-level understanding (extracting global features). Furthermore, by performing open-vocabulary object detection on the image to be detected during low-level localization, the object location indication information in the set is not limited to fixed categories, thus achieving object detection in the image. During model training, decoupling low-level localization (obtaining the object location indication information set) from high-level understanding (extracting global features) allows for more targeted training of the model. Performing open-vocabulary object detection on the image to be detected during low-level localization ensures that the object location indication information in the set is not limited to fixed categories, enabling the image detection model to possess object detection capabilities. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 An image detection scene diagram provided in an embodiment of this application;

[0071] Figure 2 A flowchart of an image detection method provided in an embodiment of this application;

[0072] Figure 3a This is a schematic diagram of open vocabulary target detection provided in an embodiment of this application;

[0073] Figure 3b This application provides a schematic diagram illustrating the positional relationship between a feature unit and an area indicated by object location indication information, as provided in an embodiment of the present application.

[0074] Figure 3c This application provides an image detection model architecture diagram.

[0075] Figure 3d This is a schematic diagram of an image detection application scenario provided in an embodiment of this application;

[0076] Figure 3e This is a schematic diagram of a video detection scenario provided in an embodiment of this application;

[0077] Figure 4 A flowchart illustrating a model training method provided in this application embodiment;

[0078] Figure 5 A flowchart illustrating an automated annotation process for target detection data based on description, as provided in this application embodiment;

[0079] Figure 6 This is a schematic diagram of the structure of an image detection device provided in an embodiment of this application;

[0080] Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0081] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0083] This application relates to technologies related to large language models and image detection. The following is a brief introduction to the relevant terms and concepts of large language models and image detection:

[0084] Transformer is a deep learning model architecture that forms the basis of large language models. It is primarily used to process sequence data, especially in natural language processing tasks. The core innovation of Transformer is its self-attention mechanism, which allows the model to focus on information at different positions when processing the input sequence, thereby capturing long-distance dependencies.

[0085] Large Language Model (LLM): A natural language processing model built on the Transformer architecture. This model, trained on large-scale text data, can understand and generate natural language, performing various language tasks such as text generation, translation, question answering, and dialogue. A key characteristic of LLMs is their massive parameter count, typically containing hundreds of millions to trillions of parameters, giving them powerful language understanding and generation capabilities. In this embodiment, the LLM is used as a foundation, and further trained using sample images, sample query text, and the corresponding verification results to obtain an image detection model (a multimodal LLM).

[0086] Multimodal Large Language Model (MLLM) is a large-scale language model trained on a large-scale text dataset, combined with other modalities (such as images and audio). These models not only possess powerful natural language processing capabilities but also can understand and generate information related to other modalities, thereby enabling richer and more diverse interactions and application scenarios.

[0087] Contrastive Language-Image Pre-training (CLIP) trains both image and text encoders simultaneously, making the matched images and texts close to each other in the same feature space, thereby achieving cross-modal understanding and generation.

[0088] Visual grounding is used to associate visual information with linguistic information. The goal of visual grounding is to find the corresponding object location indication information (bounding box) in an image based on a given natural language description (such as a sentence or phrase).

[0089] Object detection is a technique in computer vision designed to identify specific objects in images or videos and determine their locations. Object detection not only needs to identify the categories of objects present in an image but also needs to mark the specific location of each object within the image using object location indicators. This technology is widely used in fields such as autonomous driving, content moderation, and image search. The difference between object detection and visual localization is that visual localization assumes the target object must exist in the image, while object detection may search for objects that do not exist in the image; therefore, object detection has a wider range of applications than visual localization.

[0090] Open Vocabulary Detection: Open Vocabulary Detection aims to identify and locate objects in images, even if the categories of these objects do not appear in the model's training data. Traditional object detection relies on a fixed list of object categories, while Open Vocabulary Detection uses an open set of object categories, not limited to fixed categories. In this embodiment, Open Vocabulary Detection is primarily used for low-level localization (obtaining a set of object location indication information).

[0091] A multi-layer perceptron (MLP) is a type of feedforward neural network composed of multiple layers of neurons, including an input layer, one or more hidden layers, and an output layer. Each neuron is connected to the neurons in the previous layer through an activation function. MLPs can learn complex function mappings through nonlinear transformations and are widely used in classification and regression tasks.

[0092] Pre-training refers to the process of initially training a model using a large-scale dataset before a specific task. Through pre-training, the model learns general features and knowledge, which can then be fine-tuned for subsequent tasks to improve model performance. Pre-training is commonly used in fields such as natural language processing and computer vision, significantly reducing the need for labeled data and accelerating the model training process.

[0093] Instruction-tuning is a model training method that aims to optimize the performance of a pre-trained model by using a dataset containing explicit instructions or task descriptions. This approach is commonly used in natural language processing, especially in large language models.

[0094] Based on the aforementioned technologies related to large language models and image detection, embodiments of this application provide an image detection scheme and a model training scheme, which can perform object detection on images. Figure 1 An image detection scene diagram provided in an embodiment of this application, such as Figure 1 As shown, the image detection scenario provided in this application includes a terminal device 101 and a server 102. The image detection scheme and model training scheme provided in this application can be executed by either the terminal device 101 or the server 102. When both the image detection scheme and model training scheme provided in this application are executed by the terminal device 101, the image detection scenario may not include the server 102. The terminal device may include, but is not limited to: smartphones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile internet devices (MIDs), smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc., and this application embodiment does not limit this. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and this application embodiment does not limit this.

[0095] It should be noted that, Figure 1 The number of terminal devices and servers is for illustrative purposes only and does not constitute an actual limitation of this application. Terminal device 101 and server 102 can be connected via wired or wireless means, and this application does not impose any restrictions on this.

[0096] Image detection schemes and model training schemes can be independent of each other or combined; for example, the model training scheme can be executed in server 102, and the trained image detection model can be deployed in terminal device 101, and then the image detection scheme can be executed in terminal device 101; or, for another example, the model training scheme and the image detection scheme can be executed in server 102.

[0097] The general process of an image detection scheme is as follows:

[0098] (1) Server 102 obtains the image to be detected and the query text provided by terminal device 101; wherein, the query text is associated with at least one associated object, which may be contained in the query text or may be derived from the query text; for example, the associated objects of the query text “a person riding a horse” include “horse” and “person”; for another example, the associated objects of the query text “walks sideways, has eight legs and a pair of pincers” include “crab”.

[0099] (2) Server 102 performs open-vocabulary target detection on the image to be detected based on the query text, and obtains a set of object location indication information; wherein, each object location indication information in the set of object location indication information corresponds to an associated object. Since the associated object of the query text may not be contained in the image, the number of object location indication information in the set of object location indication information is less than or equal to the number of associated objects.

[0100] (3) Server 102 extracts global features of the image to be detected, and extracts local features of the image to be detected based on the global features and the set of object position indication information. In one embodiment, after extracting the global features of the image to be detected, server 102 divides the global features into at least two feature units, and determines at least one feature unit corresponding to each object position indication information based on the position of each object position indication information in the image to be detected. Then, server 102 determines the local features corresponding to each object position indication information based on the at least one feature unit corresponding to each object position indication information.

[0101] (4) Server 102 obtains the target detection result of the query text in the image to be detected based on global features, local features, and query text. This process can be implemented using an image detection model. In one embodiment, server 102 generates an input sequence of the image to be detected based on global features and local features, where each local feature corresponds to a token in the input sequence. The server then calls an image detection model to perform open-domain target detection on the input sequence using the query text as a detection condition, thereby obtaining the target detection result of the query text in the image to be detected.

[0102] The general process of training an image detection model is as follows:

[0103] Server 102 acquires sample images, sample query text, and corresponding verification results for the sample query text. The sample query text is associated with at least one related object. The sample query text and its corresponding verification results can be generated through automated annotation. In one implementation, server 102 acquires a sample image and a set of region indication information for sample objects; wherein the region indication information set indicates the location of the sample object in the sample image. Then, server 102 can call a multimodal large language model to generate descriptive information about the content in the target region based on the sample image. The target region can be the region indicated by any region indication information in the region indication information set. The large language model decomposes the descriptive information into a conditional list. Then, the multimodal large language model is called again to generate the sample query text and its corresponding verification results using the sample image, the region indication information set, and the conditional list. As can be seen, following the above process, existing (multimodal) large language models can be used for automated annotation of sample data, saving the cost of manual data annotation.

[0104] The process of obtaining the target detection result of the sample query text in the sample image based on the sample image and sample query text can refer to the image detection scheme described above, and will not be repeated here. Further, the server 102 adjusts the model to be trained based on the target detection result and the verification result to obtain the image detection model. In one embodiment, the training stage of the model to be trained may include a pre-training stage and an instruction fine-tuning stage. In the pre-training stage, the server 102 may freeze some parameters in the model to be trained and perform pre-training on the model to be trained based on the target detection result and the verification result (adjusting the parameters in the model to be trained that are not frozen). In the instruction fine-tuning stage, the server 102 may perform overall parameter adjustment on the model to be trained based on the target detection result and the verification result to obtain the image detection model.

[0105] In the image detection process, this application decouples low-level localization (obtaining the object location indication information set) from high-level understanding (extracting global features). Furthermore, by performing open-vocabulary object detection on the image to be detected during low-level localization, the object location indication information in the set is not limited to fixed categories, thus achieving object detection in the image. During model training, decoupling low-level localization (obtaining the object location indication information set) from high-level understanding (extracting global features) allows for more targeted training of the model. Performing open-vocabulary object detection on the image to be detected during low-level localization ensures that the object location indication information in the set is not limited to fixed categories, enabling the image detection model to possess object detection capabilities.

[0106] Based on the above image detection scheme and model training scheme, this application proposes a more detailed image detection method and model training method. The image detection method and model training method proposed in this application will be described in detail below with reference to the accompanying drawings.

[0107] Please see Figure 2 , Figure 2 This is a flowchart illustrating an image detection method provided in an embodiment of this application. This image detection method can be executed by a computer device; for example, by... Figure 1 The terminal device 101 or server 102 shown executes this. For example... Figure 2 As shown, the image detection method may include the following steps S201-S204:

[0108] S201. Obtain the image to be detected and the query text.

[0109] The image to be detected can be a single image or a video frame; that is, the image detection method provided in this application can also be applied to video detection (by splitting the video into video frames for individual detection). The query text can be obtained not only from text information but also from images, audio, and video.

[0110] A query text association has at least one associated object. The associated object can be contained in the query text or inferred from the query text. For example, the associated objects of the query text "a person riding a horse" include "horse" and "person" (included in the query text). Another example is the associated object of the query text "walks sideways, has eight legs and a pair of pincers" which includes "crab" (inferred from the query text).

[0111] S202. Based on the query text, perform open-vocabulary target detection on the image to be detected to obtain a set of object location indication information.

[0112] Each object location indication information in the object location indication information set corresponds to an associated object, and each object location indication information is used to indicate the position of the corresponding associated object in the image to be detected; for example, the object location indication information can indicate the coordinate bounding box containing the corresponding associated object. In one embodiment, the computer device detects the associated objects of the query text one by one in the image to be detected. If any associated object is detected in the image to be detected, the computer device outputs the object location indication information of that associated object. Furthermore, the computing device can also assign a token (such as...) to the object location indication information. <a0> , <a1>(etc.), this token can indicate the correspondence between object location information and corresponding associated objects, facilitating subsequent processing and identification. The above process can be implemented using a query-based low-level locator, with input data being query text and the image to be detected; the query text can be used to extract associated objects through an object extractor before being input into the query-based low-level locator.

[0113] It is understandable that, since the associated objects of the query text may not be contained in the image to be detected, the number of object location indicators in the object location indicator set is less than or equal to the number of associated objects.

[0114] In another implementation, the computer device performs semantic analysis on the query text to obtain words corresponding to each associated object. These words can be directly extracted or inferred. Then, based on the semantic features of each word and the image to be detected, the computer device determines candidate location indications and the confidence level of each candidate location indication. Each word corresponds to at least one candidate location indication. For example, assuming the query text is "a person riding a horse," the words include "horse" and "person." The computer device detects "horse" and "person" in the image to be detected, obtaining candidate location indications for "horse" and "person," and simultaneously outputting the confidence level of each candidate location indication. Further, the computer device filters the candidate location indications using a confidence threshold to obtain a set of object location indications.

[0115] Figure 3a This is a schematic diagram illustrating an open vocabulary target detection method provided in an embodiment of this application. For example... Figure 3a As shown, the first query term is "helmet". The low-level locator based on the query detects that region 3011 in the image to be detected is a "helmet" with a confidence level of 95%. Assuming a confidence threshold of 80%, the low-level locator based on the query outputs the object location indication information of region 3011 (such as the coordinate box of region 3011). The second query term is "bird". The low-level locator based on the query detects that region 3012 in the image to be detected is a "bird" with a confidence level of 15%. Assuming a confidence threshold of 80%, the low-level locator based on the query outputs "None" to indicate that the image to be detected does not contain "bird".

[0116] It is evident that by decoupling the object location indication information determination (low-level localization) process, detection efficiency can be improved. While achieving efficient object localization, it also provides a foundation for subsequent processing (such as complex semantic understanding, high-level understanding analysis, and applications), enabling the image detection method provided in this application to flexibly respond to various query needs in diverse real-world scenarios.

[0117] S203. Extract the global features of the image to be detected, and extract the local features of the image to be detected based on the global features and the object position indication information set.

[0118] The global features of the image to be detected can be obtained through an image encoder. Specifically, the image encoder can be a deep learning model (CLIP Vision Transformer) pre-trained on image and text based on contrastive learning. The image encoder is used to capture important information in the image to be detected and transform it into a high-dimensional feature representation (i.e., global features).

[0119] A local feature is obtained based on a global feature and an object location indication; that is, the local features of the image to be detected correspond one-to-one with the object location indication information in the object location indication information set.

[0120] In one implementation, the global features include global features of the image to be detected at at least two scaling scales. The computer device extracts feature components corresponding to each object location indication from the global features at each scaling scale, obtaining feature components corresponding to each object location indication at different scaling scales. In one embodiment, the computer device segments the global features (e.g., feature map) at a target scaling scale to obtain M feature units, where the target scaling scale is any one of at least two scaling scales, and M is an integer greater than 1.

[0121] If the area indicated by the target object location indication information is contained within a single feature unit, the computer device determines the feature component corresponding to the target object location indication information at the target scaling scale based on that feature unit; wherein, the target object location indication information is any object location indication information in the set of object location indication information. For example, the computer device can directly use the feature information carried by the feature unit as the feature component corresponding to the target object location indication information at the target scaling scale; or, it can perform interpolation calculation on the feature information carried by the feature unit to obtain the feature component corresponding to the target object location indication information at the target scaling scale. If the area indicated by the target object location indication information is contained within at least two feature units, the computer device determines the feature component corresponding to the target object location indication information at the target scaling scale based on at least two feature units. For example, the computer device can perform mean calculation on the feature information carried by at least two feature units to obtain the feature component corresponding to the target object location indication information at the target scaling scale; or, it can perform interpolation calculation on the feature information carried by at least two feature units to obtain the feature component corresponding to the target object location indication information at the target scaling scale.

[0122] Figure 3b This is a schematic diagram illustrating the positional relationship between a feature unit and an area indicated by object location indication information, provided as an embodiment of this application. For example... Figure 3b As shown, the computer device can segment the feature map of the image to be detected to obtain several feature units. Region 3021 is contained in one feature unit. The computer device can directly use the feature information carried by the feature unit as the feature component corresponding to the object position indication information of region 3021 at the current scaling scale; or, it can perform interpolation calculation on the feature information carried by the feature unit to obtain the feature component corresponding to the object position indication information of region 3021 at the current scaling scale. Region 3022 is contained in four feature units. The computer device can perform mean calculation on the feature information carried by these four feature units to obtain the feature component corresponding to the object position indication information of region 3022 at the current scaling scale; or, it can perform interpolation calculation on the feature information carried by these four feature units to obtain the feature component corresponding to the object position indication information of region 3022 at the current scaling scale.

[0123] Furthermore, the computer device fuses the feature components corresponding to each object location indication information at different scaling scales to obtain the local features corresponding to each object location indication information.

[0124] In another implementation, the global features are the global features of the image to be detected at a scaling scale. The computer device extracts the local features corresponding to each object location indication from the global features. In one embodiment, the computer device segments the global features (such as a feature map) to obtain M feature units, where M is an integer greater than 1. Then, based on the positional relationship between the region indicated by each object location indication and each feature unit, the computer device determines the local features corresponding to each object location indication. For specific implementation details, please refer to... Figure 3b This will not be elaborated upon here.

[0125] S204. Based on global features, local features, and query text, obtain the target detection results of the query text in the image to be detected.

[0126] In one implementation, a computer device generates an input sequence of an image to be detected based on global and local features. In one embodiment, the computer device projects the global features of the image to be detected at a target scaling scale into a text feature space through a global feature projection layer to obtain first projected features. When the global features include global features of the image to be detected at at least two scaling scales, the target scaling scale is any one of the at least two scaling scales; that is, the computer device does not need to project all global features at all scaling scales into the text feature space. For example, assuming the global features include global features of the image to be detected at L scaling scales, the computer device projects the global features at the (L-1)th scaling scale into the text feature space to obtain the first projected features, where L is an integer greater than 1.

[0127] On the other hand, the computer device projects local features into the text feature space through a local feature projection layer, obtaining second projected features. Each second projected feature corresponds to a local feature, and the projection parameters of the local feature projection layer are different from those of the global feature projection layer. It can be understood that the purpose of feature projection is to ensure that global and local features can be effectively integrated with the query text when performing target detection subsequently.

[0128] After obtaining the first projection feature and the second projection feature, the computer device combines the first projection feature and the second projection feature to obtain the input sequence of the image to be detected. In one embodiment, the input sequence of the image to be detected can be represented as:

[0129] Image:<image>Regions:<region><a0><region><a1>…<region><a N-1 >

[0130] Where <image> represents the first projected feature (obtained by projecting global features through a global feature projection layer), and <region> represents the second projected feature (obtained by projecting local features through a local feature projection layer). 0～N-1 > represents the region anchor token assigned to each second projection feature. This region anchor token can be used to determine the object location indication information corresponding to each second projection feature, where N is an integer greater than 1. It can be understood that this input sequence enables multi-granularity visual feature input of the image to be processed, allowing the computer device to perform a more in-depth and comprehensive analysis based on these multi-granularity visual features during subsequent object detection, thereby improving the accuracy of the detection results.

[0131] Further, the computer device uses the query text as a detection condition to perform open-domain object detection on the input sequence, obtaining the object detection result of the query text in the image to be detected. In one embodiment, the process of the computer device using the query text as a detection condition includes: performing language annotation on the query text to obtain the annotation result of the query text (including objects, conditions, etc. in the query text). In another embodiment, the input sequence includes N tokens, each token corresponding to a local feature, where N is an integer greater than 1; the object detection result is obtained based on the prediction results of the N tokens. The process of the computer device using the query text as a detection condition to perform open-domain object detection on the input sequence to obtain the object detection result of the query text in the image to be detected includes: generating the prediction result of the (i+1)th token based on the global features and the prediction results of the first i tokens, where i is a positive integer less than N. That is, except for the first token, the prediction result of each token is generated based on the previously obtained prediction results, and the prediction result of the first token is generated based on the global features.

[0132] Figure 3c This is a diagram illustrating an image detection model architecture provided in an embodiment of this application. Figure 3c As shown, the input data of the image detection model are the image to be detected and the query text. The output data of the image detection model can be the position of the query object in the image to be detected, or "XX not found (XX is any query object)". The image detection model includes a large language model, a global feature projection layer, a local feature projection layer, an image encoder, a local feature extractor, a language tagger, and a query-based low-level locator. The global and local feature projection layers project the global and local features of the image to be detected into the text feature space, respectively. The image encoder extracts the global features of the image to be detected, and the local feature extractor extracts the local features. The language tagger can tag not only objects contained in the query text but also conditions in the query text. The query-based low-level locator is used to achieve open-vocabulary object detection. The large language model, based on its large number of parameters and rich pre-trained knowledge, performs high-level understanding within the model to output decision results.

[0133] For example, suppose the query text is "a person riding a horse wearing a helmet". In this case, the low-level locator based on the query will generate a set of object location indications, including the object location indications for "helmet", "horse", and "person". The local feature extractor, based on the global features extracted by the image encoder and the set of object location indications, obtains the local features of the image to be detected, and projects these local features into the text feature space through a local feature projection layer. Furthermore, the global features extracted by the image encoder are also projected into the text feature space through a global feature projection layer; the query text is labeled by a language tagger, and the labeled results are used as query conditions by the large language model. The large language model analyzes the global and local features of the image to be detected, combines this with its semantic understanding of the query text, and finally determines that "person" in the image meets the query conditions, outputting a "person" token. <a2>Each token corresponds to an object location indicator, thus enabling the efficient acquisition of location information (such as a bounding box). Conversely, if the query text is "walking person" and the image to be detected does not contain a walking person, the large language model will identify that there is no object matching the query condition in the image to be detected and output "None" to indicate that no relevant object was detected. This mechanism effectively utilizes the powerful capabilities of the large language model in language understanding and reasoning, transforming the complex object detection problem into a candidate object selection problem for the large language model.

[0134] In practical applications, Figure 3c The image detection model architecture shown can be trained and deployed for inference on a Linux-based Graphics Processing Unit (GPU) cloud server. This environment leverages high-performance GPUs for accelerated computation, effectively handling large-scale datasets and complex model structures, significantly improving training speed and inference efficiency. The server configuration includes multiple high-performance GPUs, ample memory, and storage to support the training needs of large models. Furthermore, the deep learning framework PyTorch is used for model development, combined with containerization technology to achieve environment isolation and portability, ensuring model consistency and stability across different environments. Through this efficient runtime environment, the model can be rapidly iterated and optimized to meet the real-time inference requirements of real-world business scenarios.

[0135] In one embodiment, the query text is a description of the query object, and the target detection result of the query text in the image to be detected is used to indicate whether the image to be detected contains the query object. If the target detection result indicates that the image to be detected contains the query object, the target detection result also includes object location indication information corresponding to the query object, which is used to indicate the position of the query object in the image to be detected. Correspondingly, if the target detection result indicates that the image to be detected does not contain the query object, the target detection result can be "None" to indicate that the image to be detected does not contain the query object.

[0136] Figure 3d This is a schematic diagram illustrating an image detection application scenario provided in an embodiment of this application. For example... Figure 3d As shown, when the query text is "bananas on a plate", the image detection model outputs target location indication information, which indicates the position of region 3031 in the image to be detected. When the query text is "apple in a bowl", the image detection model outputs "None", indicating that the image to be detected does not contain an object that matches the query condition.

[0137] Figure 3e This is a schematic diagram of a video detection scene provided in an embodiment of this application. Figure 3e As shown, the query text can be extracted based on audio, or it can be extracted based on an image carrying text information; this application does not limit this. When detecting a video, each video frame of the video to be detected can be input into the image detection model as an image to be detected for target detection. After obtaining the target detection results for each video frame, the target detection result of the video to be detected is obtained based on the target detection results of all video frames. For example, if the video is a fruit introduction video and the query text is "peeled banana," assuming that at least one video frame in the fruit introduction video contains "peeled banana," the computer device determines that the video meets the detection conditions based on the target detection results of all video frames. Optionally, when detecting certain dynamic conditions (such as walking, flying, etc.), the computer device can also call the image detection model to make a judgment based on multiple adjacent video frames.

[0138] In this embodiment, the process involves acquiring the image to be detected and the query text. Firstly, open-vocabulary object detection is performed on the image to be detected based on the query text to obtain a set of object location indication information. Secondly, global features of the image to be detected are extracted. Then, based on the obtained global features and the set of object location indication information, local features of the image to be detected are extracted. Finally, based on the global features, local features, and the query text, the object detection result of the query text in the image to be detected is obtained. It is evident that by decoupling low-level localization (obtaining the set of object location indication information) and high-level understanding (extracting global features), and by performing open-vocabulary object detection on the image to be detected during low-level localization, the object location indication information in the set of object location indication information is not limited to fixed categories. Furthermore, the understanding and reasoning capabilities of a large language model are used to filter out the target objects that ultimately meet the requirements, thereby improving object localization performance and achieving object detection in the image, thus enhancing the generalization ability of object detection in various scenarios.

[0139] Please see Figure 4 , Figure 4 A flowchart illustrating a model training method provided in this application embodiment, the model training method being executed by a computer device; for example, by... Figure 1 The terminal device 101 or server 102 shown executes this. For example... Figure 4 As shown, the model training method may include the following steps S401-S405:

[0140] S401 is used to obtain sample images, sample query text, and the corresponding verification results of the sample query text.

[0141] The sample query text is associated with at least one associated object. In one implementation, a computer device can automatically generate labeled data based on the sample image, thereby obtaining the sample query text and the corresponding verification result.

[0142] In one embodiment, the automated description of the target detection data annotation process includes: a computer device acquiring a sample image and a set of region indication information for sample objects. The set of region indication information is used to indicate the region where the sample object is located in the sample image. For example, assuming the sample object is "pizza," each region indication information is used to indicate the region where one "pizza" is located in the sample image, and different region indication information indicates different regions. The difference between region indication information and the aforementioned object location indication information is that region indication information is acquired known information (such as labeled data), while object location indication information is obtained by performing target detection on the image to be detected using a developed vocabulary.

[0143] Next, the computer device can invoke a pre-trained multimodal large language model to generate descriptive information about the content of the target region based on the sample image. The target region is the region indicated by any region indicator in the region indicator information set. For example, if the target region is a pizza with cheese and vegetables, the computer device will generate the descriptive information "a pizza with cheese and vegetables" based on the target region. After generating the descriptive information about the content of the target region, the computer device can invoke the pre-trained large language model to decompose the descriptive information into a condition list; for example, "a pizza with cheese and vegetables" can be decomposed into: "Condition 1: The object is a pizza", "Condition 2: The object contains cheese", and "Condition 3: The object contains vegetables". Conditions 1-3 together constitute the condition list. Further, the computer device uses the sample image, the region indicator information set, and the condition list to generate sample query text and the corresponding verification results.

[0144] In one implementation, the computer device can invoke a pre-trained multimodal large language model to perform matching detection between the region indicated by each region indication information and the conditions in the condition list, obtaining the matching result of each region under each condition. In one embodiment, each matching result is obtained using a thought chain technique; that is, each matching result corresponds to a judgment criterion. For example, assuming region 1 is a pizza with tomatoes, condition 1 is "the object is a pizza", and condition 2 is "the object contains cheese", then the judgment criterion for region 1 and condition 1 is generated as follows: "the image recognition result of region 1 is a pizza", and the matching result is: region 1 matches condition 1; the judgment criterion for region 1 and condition 2 is: "the image of region 1 does not contain cheese", and the matching result is: region 1 does not match condition 2.

[0145] Furthermore, if the matching result of any region is true under all conditions, the computer device adds the region indication information corresponding to that region to the sample region indication information set corresponding to the description information. For example, suppose region 1 corresponding to region indication information 1 is a pizza with cheese and vegetables. In the condition list of description information A, "Condition 1: The object is a pizza", "Condition 2: The object contains cheese", and "Condition 3: The object contains vegetables". Region 1 meets all the conditions in the condition list, so the computer device adds region indication information 1 to the sample region indication information set corresponding to description information A. Following the above method, the computer device can obtain all region indication information in the region indication information set that matches the description information (these region indication information collectively constitute the sample region indication information set corresponding to the description information).

[0146] Next, the computer device generates sample query text based on the description information and generates the corresponding verification result based on the sample region indication information set; for example, the computer device can directly use the description information as the sample query text and the sample region indication information set as the verification result corresponding to this query text.

[0147] Figure 5 This document provides a flowchart for automated annotation of target detection data based on description, as an embodiment of this application. Figure 5 As shown, the initial data includes sample images and a set of region indication information for objects within the sample images. Figure 5 The object in the middle is "banana", and the set of area indication information includes area indication information 5011 and area indication information 5012 (displayed in the form of coordinate boxes).

[0148] After obtaining the initial data, the first step is to generate (content) description information for the indicated regions. This process involves combining sample images with region indication information from a set of region indication information. Figure 5 Taking the region indication information 5012 as an example, it is used as the input of the multimodal large language model, and the model is called to generate the (content) description information of the region indicated by the region indication information 5012; for example, the description information corresponding to the region indication information 5012 is "bananas on a plate".

[0149] Next, the generated description information is decoupled from the conditions. This process involves using the description information as input to a large language model and calling the model to decompose the input description information to obtain a condition list (which can be represented as Condition_list). Taking the description information "bananas on a plate" as an example, the decomposed condition list includes: "Condition 1: The object is a banana" and "Condition 2: The object is on a plate".

[0150] Then, using the condition list as query conditions, the system checks whether the regions indicated by each region indicator in the region indicator information set meet all the conditions in the query list. When a region indicator information indicates a region that meets all the conditions in the query list, the computer device adds that region indicator information to the sample region indicator information set corresponding to the description information. This process uses the region indicator information set, sample images, and condition list as input to a multimodal large language model, and calls the model to output the region indicator information set that meets all the conditions in the query list.

[0151] Optionally, the determination of whether the region indicated by the region indication information meets the query conditions can be based on the thinking chain technique. This technique can enhance the reasoning ability of the multimodal large language model, thereby improving the accuracy of the labeled data and reducing labeling noise. In other words, before obtaining the matching result for each condition, the multimodal large language model needs to provide the basis (reason) for the judgment. For example, assuming the current query is "Does the object in region indication information 5011 match condition 2?", the multimodal large language model first provides the basis for the judgment: "The region does not include plates," and then gives the answer: "No match."

[0152] Finally, the annotation results are the sample image, descriptive information, and a set of sample region indication information corresponding to the descriptive information (including the region indication information corresponding to all regions in the sample image that match the descriptive information). Based on the descriptive information and the set of sample region indication information corresponding to the descriptive information, the sample query text of the sample image (obtained based on the descriptive information) and the verification result corresponding to the sample query text (obtained based on the set of sample region indication information corresponding to the descriptive information) can be obtained.

[0153] As can be seen, this automated annotation process can transform object (category) based object detection datasets into description-based object detection datasets for training models, enabling the trained image detection models to perform description-based object detection tasks. Description-based object detection tasks contain richer and more complex semantic information about objects compared to general category-based object detection tasks. For example, object (category) based object detection can only detect coarse-grained object classifications such as "person," "bicycle," and "airplane," while description-based object detection can detect fine-grained object classifications such as "person wearing a baseball cap," "bicycle parked on the roadside," and "airplane landed on the ground."

[0154] The automated object detection data annotation process utilizes existing (multimodal) large language models to annotate sample images, obtaining descriptive information associated with the sample images and indication information of sample regions, thereby generating sample query text and corresponding verification results. Compared to manual annotation, this reduces data annotation costs and improves data annotation efficiency.

[0155] S402. Based on the sample query text, perform open-vocabulary target detection on the sample image to obtain a set of object coordinate box object location indication information.

[0156] S403. Extract the global features of the sample image, and extract the local features of the sample image based on the global features and the object position indication information set.

[0157] S404. Call the model to be trained to use the sample query text as the detection condition, and perform open domain target detection on the sample image based on global and local features to obtain the target detection results of the sample query text in the sample image.

[0158] For detailed implementation methods of steps S402-S404, please refer to [link / reference]. Figure 2 The implementation methods in steps S202-S204 will not be described in detail here.

[0159] S405. Based on the target detection results and verification results, adjust the model to be trained to obtain the image detection model.

[0160] The architecture of the model to be trained can be referenced. Figure 3c In one embodiment, the training phase of the model to be trained may include a pre-training phase and an instruction fine-tuning phase. (Combined) Figure 3c In the pre-training phase, the computer can freeze the parameters of the image encoder and the large language model, and adjust the parameters of the global feature projection layer and the local feature projection layer based on the object detection results and verification results. In the instruction fine-tuning phase, the computer can adjust all parameters except those of the image encoder (such as the large language model parameters, object extractor parameters, query-based low-level locator parameters, and local feature extractor parameters) based on the object detection results and verification results to obtain the image detection model.

[0161] In one implementation, the target detection result is obtained based on the prediction results of N tokens, where the prediction result of each token indicates whether the corresponding local feature satisfies the detection condition, and N is an integer greater than 1. The computer device calculates the conditional probability of each token based on the verification result and the prediction results of the N tokens, and constructs a cross-entropy loss function using the conditional probabilities of the N tokens, which can be specifically expressed as:

[0162]

[0163] Where N is the number of tokens in the input sequence, and the total length of the input sequence is N+1. t is the t-th token in the input sequence. p(x t |x ＜t ) is the probability that the model generates the t-th token given the first t-1 tokens.

[0164] For example, a computer device obtains an input sequence "Image: <image> Regions: <region> <a0> <region> <a1> ... <region> <a N-1 The sample query text was "Find all people riding horses wearing helmets". The test result was "<a 10 > <a 23 >”; where <a 10 > and <a 23 > These indicate the regions of two "people" in the sample image that meet the conditions of the sample query text. Then, the computer device uses the prediction results of N tokens to statistically analyze . 10 > and <a 23 The conditional probability of >, that is, the generation of <a 10 The probability of > and the already generated <a 10 When >, is generated. 23 The probability of > is calculated, and a cross-entropy loss function is constructed based on statistical probability.

[0165] After constructing the cross-entropy loss function, the computer device adjusts the training model based on the cross-entropy loss function (e.g., through gradient descent algorithm) to obtain the image detection model. The method of adjusting model parameters based on the cross-entropy loss function is applicable to both the pre-training and fine-tuning stages.

[0166] Optionally, the training process of the model to be trained can also be optimized based on at least one of the following methods:

[0167] (1) Generative Adversarial Networks (GANs) are used to generate candidate object regions that match the query text description. Specifically, the generator generates potential object regions based on the query text description, and the discriminator determines whether these object regions match objects in the sample images. Through adversarial training, the generator can learn how to generate more accurate object regions, thereby achieving description-based object detection.

[0168] (2) A reinforcement learning (RL) algorithm is used to train an agent to locate target objects in sample images based on query text descriptions. The agent learns how to select the optimal object region in the sample images through interaction with the environment. The reward mechanism can be designed based on the accuracy of the detection results, thereby guiding the agent to gradually optimize its localization strategy.

[0169] (3) Graph Neural Networks (GNNs) are used to model the relationships between objects in sample images, thereby achieving description-based object detection. Specifically, objects in the image are represented as nodes in a graph, and the relationships between objects are represented as edges in the graph. Through the propagation mechanism of GNNs, complex relationships between objects can be captured, thus locating target objects more accurately.

[0170] In this embodiment, a sample image, sample query text, and corresponding verification results are obtained. On one hand, open-domain object detection is performed on the sample image based on the sample query text to obtain a set of object location indication information. On the other hand, global features of the sample image are extracted. Then, based on the obtained global features and the set of object location indication information, local features of the sample image are extracted. The model to be trained is then called, using the sample query text as a detection condition. Open-domain object detection is performed on the sample image based on the global and local features to obtain the object detection results of the sample query text in the sample image. Based on the object detection results and verification results, the model to be trained is adjusted to obtain an image detection model. It can be seen that during model training, by decoupling low-level localization (obtaining the set of object location indication information) and high-level understanding (extracting global features), the model to be trained can be trained more specifically. By performing open-domain object detection on the image to be detected during low-level localization, the object location indication information in the set of object location indication information is not limited to fixed categories, enabling the image detection model to possess object detection capabilities. Furthermore, by utilizing existing (multimodal) large language models to annotate sample images, descriptive information associated with the sample images and sample region indication information can be obtained. This leads to the generation of sample query text and corresponding verification results, enabling automated annotation of target detection data. Compared to manual annotation, this reduces data annotation costs and improves data annotation efficiency.

[0171] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0172] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an image detection device provided in an embodiment of this application. Figure 6 The image detection device shown can be mounted in a computer device, which can be a terminal device or a server. Figure 6 The image detection device shown can be used to perform the above. Figure 2 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 6 The image detection device includes:

[0173] The acquisition unit 601 is used to acquire the image to be detected and the query text, wherein the query text is associated with at least one associated object;

[0174] Processing unit 602 is used to perform open-vocabulary target detection on the image to be detected based on the query text, and obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object;

[0175] And to extract global features of the image to be detected, and to extract local features of the image to be detected based on the global features and the object position indication information set;

[0176] It is also used to obtain the target detection results of the query text in the image to be detected based on global features, local features and query text.

[0177] In one implementation, the processing unit 602 is configured to perform open-vocabulary target detection on the image to be detected based on the query text, and obtain a set of object location indication information, specifically for:

[0178] Perform semantic analysis on the query text to obtain the vocabulary corresponding to each associated object;

[0179] Based on the semantic features of each word and the image to be detected, candidate location indication information and the confidence level of each candidate location indication information are determined, with each word corresponding to at least one candidate location indication information.

[0180] The candidate location indication information is filtered by a confidence threshold to obtain a set of object location indication information.

[0181] In one implementation, the global features of the image to be detected include global features of the image at at least two scaling scales; the local features of the image to be detected correspond one-to-one with the object position indication information in the object position indication information set; the processing unit 602 is used to extract the local features of the image to be detected based on the global features and the object position indication information set, specifically for:

[0182] From the global features at each scaling scale, extract the feature components corresponding to the position indication information of each object to obtain the feature components corresponding to the position indication information of each object at different scaling scales.

[0183] The feature components corresponding to the location indication information of each object at different scaling scales are fused to obtain the local features corresponding to each object location indication information.

[0184] In one embodiment, the processing unit 602 is configured to extract the feature components corresponding to each object position indication information from the global features at each scaling scale, thereby obtaining the feature components corresponding to each object position indication information at different scaling scales, specifically configured to:

[0185] The global features at the target scaling scale are segmented to obtain M feature units. The target scaling scale is any one of at least two scaling scales, and M is an integer greater than 1.

[0186] If the area indicated by the target object location indication information is contained within a feature unit, then based on that feature unit, the feature component corresponding to the target object location indication information at the target scaling scale is determined, and the target object location indication information is any one of the object location indication information in the set of object location indication information.

[0187] If the area indicated by the target object location indication information is contained within at least two feature units, then based on the at least two feature units, the feature components corresponding to the target object location indication information at the target scaling scale are determined.

[0188] In one embodiment, the processing unit 602 is configured to obtain the target detection result of the query text in the image to be detected based on global features, local features, and query text, specifically for:

[0189] The input sequence of the image to be detected is generated based on global and local features;

[0190] By using the query text as the detection condition, open domain target detection is performed on the input sequence to obtain the target detection results of the query text in the image to be detected.

[0191] In one implementation, the input sequence includes N tokens, each token corresponding to a local feature, where N is an integer greater than 1, and the target detection result is obtained based on the prediction results of the N tokens; the processing unit 602 uses the query text as a detection condition to perform open-domain target detection on the input sequence, and the process of obtaining the target detection result of the query text in the image to be detected includes:

[0192] Based on global features and the prediction results of the first i tokens, the prediction result of the (i+1)th token is generated, where i is a positive integer less than N.

[0193] In one implementation, the global features of the image to be detected include global features of the image at at least two scaling scales; the processing unit 602 is configured to generate an input sequence of the image to be detected based on the global features and local features, specifically for:

[0194] The global feature projection layer projects the global features of the image to be detected at the target scaling scale into the text feature space to obtain the first projection feature. The target scaling scale is any one of at least two scaling scales.

[0195] The local feature projection layer projects local features onto the text feature space to obtain second projected features. Each second projected feature corresponds to a local feature. The projection parameters of the local feature projection layer are different from those of the global feature projection layer.

[0196] The first projection feature and the second projection feature are combined to obtain the input sequence of the image to be detected.

[0197] In one implementation, the query text is descriptive information of the query object; the target detection result of the query text in the image to be detected is used to indicate whether the image to be detected contains the query object;

[0198] If the target detection result indicates that the image to be detected contains a query object, the target detection result also includes object location indication information corresponding to the query object, which is used to indicate the position of the query object in the image to be detected.

[0199] According to one embodiment of this application, Figure 2 The steps involved in the image detection method shown can be derived from... Figure 6 The image detection apparatus shown is executed by each unit. For example, Figure 2 Step S201 shown can be performed by Figure 6 The acquisition unit 601 shown is executed, and steps S202-S204 can be performed by... Figure 6 The processing unit 602 shown executes. Figure 6 The various units in the image detection device shown can be individually or entirely combined into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image detection device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0200] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 2 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 6 The image detection apparatus shown herein, and the image detection method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0201] Based on the same inventive concept, the principle and beneficial effects of the image detection device provided in the embodiments of this application are similar to the principle and beneficial effects of the image detection method in the embodiments of this application. For details, please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, these will not be repeated here.

[0202] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application. Figure 7 The model training device shown can be mounted on a computer device, which can be a terminal device or a server. Figure 7 The model training apparatus shown can be used to perform the above. Figure 4 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 7 The model training device includes:

[0203] The acquisition unit 701 is used to acquire a sample image, a sample query text and a verification result corresponding to the sample query text, wherein the sample query text is associated with at least one associated object.

[0204] Processing unit 702 is used to perform open-vocabulary target detection on sample images based on sample query text to obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object;

[0205] And to extract global features from sample images, and to extract local features from sample images based on the set of global features and object location indication information;

[0206] And it is used to call the model to be trained to use the sample query text as the detection condition, perform open domain target detection on the sample image based on global and local features, and obtain the target detection result of the sample query text in the sample image;

[0207] And it is used to adjust the training model based on the target detection results and verification results to obtain the image detection model.

[0208] In one embodiment, the processing unit 702 is configured to acquire a sample image, sample query text, and the verification result corresponding to the sample query text, specifically for:

[0209] Obtain a set of region indication information for the sample image and the sample object. The set of region indication information is used to indicate the region where the sample object is located in the sample image.

[0210] Based on the sample image, generate descriptive information about the content of the target region, where the target region is the region indicated by any one of the region indication information in the set of region indication information;

[0211] The description information is broken down into conditions to obtain a list of conditions for the description information;

[0212] Using sample images, a set of region indication information, and a list of conditions, sample query text and corresponding verification results are generated.

[0213] In one embodiment, the processing unit 702 is configured to generate sample query text and corresponding verification results from the sample image, the set of region indication information, and the condition list, specifically for:

[0214] The region indicated by each region indication information is matched and detected with the conditions in the condition list to obtain the matching result of each region under each condition. Each matching result has a corresponding judgment basis.

[0215] If the matching result of any region is true under all conditions, then the region indication information corresponding to that region is added to the sample region indication information set corresponding to the description information.

[0216] The sample query text is generated based on the descriptive information, and the corresponding verification result is generated based on the sample region indication information set.

[0217] In one implementation, the target detection result is obtained based on the prediction results of N tokens, where the prediction result of each token indicates whether the corresponding local features satisfy the detection condition, and N is an integer greater than 1; the processing unit 702 is used to adjust the training model based on the target detection result and the verification result to obtain an image detection model, specifically for:

[0218] Based on the verification results and the prediction results of N tokens, the conditional probability of each token is calculated.

[0219] Construct the cross-entropy loss function using the conditional probabilities of N tokens;

[0220] The image detection model is obtained by adjusting the model to be trained based on the cross-entropy loss function.

[0221] According to one embodiment of this application, Figure 4 The model training method shown can be partially implemented by... Figure 7 The individual units in the model training device shown are used to execute this. For example, Figure 4 Step S401 shown can be performed by Figure 7 The acquisition unit 701 shown is executed, and steps S402-S405 can be performed by... Figure 7 The processing unit 702 shown is executed. Figure 7 The units in the model training device shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the model training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0222] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 4 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 7 The model training apparatus shown herein, and the model training method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0223] Based on the same inventive concept, the principle and beneficial effects of the model training device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the model training method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0224] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a terminal device or a server. Figure 8 As shown, the computer device includes at least a processor 801, a communication interface 802, and a memory 803. The processor 801, communication interface 802, and memory 803 can be connected via a bus or other means. The processor 801 (or Central Processing Unit, CPU) is the computing and control core of the computer device. It can parse various instructions within the computer device and process various data. For example, the CPU can parse power-on / off commands issued by an object to the computer device and control the computer device to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device, and so on. The communication interface 802 may optionally include standard wired interfaces or wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), and can be used to send and receive data under the control of the processor 801; the communication interface 802 can also be used for data transmission and interaction within the computer device. The memory 803 is the storage device in the computer device, used to store programs and data. It can be understood that the memory 803 here can include the computer device's built-in memory, or it can include extended memory supported by the computer device. The memory 803 provides storage space for storing the operating system of the computer device, which may include, but is not limited to, Android, iOS, Windows Phone, etc. This application does not limit this.

[0225] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores computer programs suitable for loading and execution by the processor 801. It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0226] In one embodiment, processor 801 performs the following operations by running a computer program stored in memory 803:

[0227] Obtain the image to be detected and the query text, wherein the query text is associated with at least one associated object;

[0228] Open-vocabulary object detection is performed on the image to be detected based on the query text, resulting in a set of object location indication information; each object location indication information in the set corresponds to an associated object.

[0229] Extract global features from the image to be detected, and extract local features from the image to be detected based on the global features and the object location indication information set;

[0230] Based on global features, local features, and query text, the target detection results of the query text in the image to be detected are obtained.

[0231] As an optional embodiment, the processor 801 performs open-vocabulary target detection on the image to be detected based on the query text to obtain a set of object location indication information. A specific embodiment of this is as follows:

[0232] Perform semantic analysis on the query text to obtain the vocabulary corresponding to each associated object;

[0233] Based on the semantic features of each word and the image to be detected, candidate location indication information and the confidence level of each candidate location indication information are determined, with each word corresponding to at least one candidate location indication information.

[0234] The candidate location indication information is filtered by a confidence threshold to obtain a set of object location indication information.

[0235] As an optional embodiment, the global features of the image to be detected include global features of the image at at least two scaling scales; the local features of the image to be detected correspond one-to-one with the object position indication information in the object position indication information set; a specific embodiment of the processor extracting the local features of the image to be detected based on the global features and the object position indication information set is as follows:

[0236] From the global features at each scaling scale, extract the feature components corresponding to the position indication information of each object to obtain the feature components corresponding to the position indication information of each object at different scaling scales.

[0237] The feature components corresponding to the location indication information of each object at different scaling scales are fused to obtain the local features corresponding to each object location indication information.

[0238] As an optional embodiment, the processor 801 extracts the feature components corresponding to each object position indication information from the global features at various scaling scales. A specific embodiment of obtaining the feature components corresponding to each object position indication information at different scaling scales is as follows:

[0239] The global features at the target scaling scale are segmented to obtain M feature units. The target scaling scale is any one of at least two scaling scales, and M is an integer greater than 1.

[0240] If the area indicated by the target object location indication information is contained within a feature unit, then based on that feature unit, the feature component corresponding to the target object location indication information at the target scaling scale is determined, and the target object location indication information is any one of the object location indication information in the set of object location indication information.

[0241] If the area indicated by the target object location indication information is contained within at least two feature units, then based on the at least two feature units, the feature components corresponding to the target object location indication information at the target scaling scale are determined.

[0242] As an optional embodiment, the processor 801 obtains the target detection result of the query text in the image to be detected based on global features, local features, and query text. A specific embodiment of this is as follows:

[0243] The input sequence of the image to be detected is generated based on global and local features;

[0244] By using the query text as the detection condition, open domain target detection is performed on the input sequence to obtain the target detection results of the query text in the image to be detected.

[0245] As an optional embodiment, the input sequence includes N tokens, each token corresponding to a local feature, where N is an integer greater than 1, and the target detection result is obtained based on the prediction results of the N tokens; the processor 801 uses the query text as a detection condition to perform open-domain target detection on the input sequence, and the process of obtaining the target detection result of the query text in the image to be detected includes:

[0246] Based on global features and the prediction results of the first i tokens, the prediction result of the (i+1)th token is generated, where i is a positive integer less than N.

[0247] As an optional embodiment, the global features of the image to be detected include global features of the image at at least two scaling scales; a specific embodiment of the processor 801 generating the input sequence of the image to be detected based on the global features and local features is as follows:

[0248] The global feature projection layer projects the global features of the image to be detected at the target scaling scale into the text feature space to obtain the first projection feature. The target scaling scale is any one of at least two scaling scales.

[0249] The local feature projection layer projects local features onto the text feature space to obtain second projected features. Each second projected feature corresponds to a local feature. The projection parameters of the local feature projection layer are different from those of the global feature projection layer.

[0250] The first projection feature and the second projection feature are combined to obtain the input sequence of the image to be detected.

[0251] As an optional embodiment, the query text is a description of the query object; the target detection result of the query text in the image to be detected is used to indicate whether the image to be detected contains the query object;

[0252] If the target detection result indicates that the image to be detected contains a query object, the target detection result also includes object location indication information corresponding to the query object, which is used to indicate the position of the query object in the image to be detected.

[0253] In another embodiment, processor 801 performs the following operations by running a computer program stored in memory 803:

[0254] Obtain the sample image, sample query text, and the corresponding verification result of the sample query text. The sample query text is associated with at least one associated object.

[0255] Open-vocabulary object detection is performed on sample images based on sample query text to obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object.

[0256] Extract global features from the sample image, and extract local features from the sample image based on the global features and the object location indication information set;

[0257] The model to be trained is invoked to use the sample query text as the detection condition. Based on global and local features, open domain target detection is performed on the sample image to obtain the target detection results of the sample query text in the sample image.

[0258] Based on the target detection results and the verification results, the model to be trained is adjusted to obtain the image detection model.

[0259] As an optional embodiment, the processor 801 acquires the sample image, sample query text, and the corresponding verification result of the sample query text. A specific embodiment of this is as follows:

[0260] Obtain a set of region indication information for the sample image and the sample object. The set of region indication information is used to indicate the region where the sample object is located in the sample image.

[0261] Based on the sample image, generate descriptive information about the content of the target region, where the target region is the region indicated by any one of the region indication information in the set of region indication information;

[0262] The description information is broken down into conditions to obtain a list of conditions for the description information;

[0263] Using sample images, a set of region indication information, and a list of conditions, sample query text and corresponding verification results are generated.

[0264] As an optional embodiment, the processor 801 generates sample query text and corresponding verification results from the sample image, the set of region indication information, and the condition list. A specific embodiment of this is as follows:

[0265] The region indicated by each region indication information is matched and detected with the conditions in the condition list to obtain the matching result of each region under each condition. Each matching result has a corresponding judgment basis.

[0266] If the matching result of any region is true under all conditions, then the region indication information corresponding to that region is added to the sample region indication information set corresponding to the description information.

[0267] The sample query text is generated based on the descriptive information, and the corresponding verification result is generated based on the sample region indication information set.

[0268] As an optional embodiment, the object detection result is obtained based on the prediction results of N tokens, where the prediction result of each token indicates whether the corresponding local features meet the detection conditions, and N is an integer greater than 1; the processor 801 adjusts the training model based on the object detection result and the verification result, resulting in the following specific embodiment of the image detection model:

[0269] Based on the verification results and the prediction results of N tokens, the conditional probability of each token is calculated.

[0270] Construct the cross-entropy loss function using the conditional probabilities of N tokens;

[0271] The image detection model is obtained by adjusting the model to be trained based on the cross-entropy loss function.

[0272] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the image detection method and model training method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the implementation of the method. For the sake of brevity, they will not be repeated here.

[0273] This application also provides a computer-readable storage medium storing a computer program, which is adapted to be loaded by a processor and execute the image detection method of the above method embodiments, or adapted to be loaded by a processor and execute the model training method of the above method embodiments.

[0274] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image detection method or the model training method described above.

[0275] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0276] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0277] In the embodiments of this application, the term "module" or "unit" refers to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0278] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0279] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments and equivalent variations made in accordance with the claims of this application are still within the scope of this application. < / a0>

Claims

1. An image detection method, characterized in that, The method includes: Obtain the image to be detected and the query text, wherein the query text is associated with at least one associated object; Based on the query text, open vocabulary target detection is performed on the image to be detected to obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object. Extract the global features of the image to be detected, and extract the local features of the image to be detected based on the global features and the object position indication information set; Based on the global features, the local features, and the query text, the target detection result of the query text in the image to be detected is obtained.

2. The method as described in claim 1, characterized in that, The step of performing open-vocabulary target detection on the image to be detected based on the query text to obtain a set of object location indication information includes: Perform semantic analysis on the query text to obtain the vocabulary corresponding to each associated object; Based on the semantic features of each word and the image to be detected, candidate location indication information and the confidence level of each candidate location indication information are determined, with each word corresponding to at least one candidate location indication information. The candidate location indication information is filtered by a confidence threshold to obtain a set of object location indication information.

3. The method as described in claim 1, characterized in that, The global features of the image to be detected include global features of the image to be detected at at least two scaling scales; the local features of the image to be detected correspond one-to-one with the object position indication information in the object position indication information set; the step of extracting the local features of the image to be detected based on the global features and the object position indication information set includes: From the global features at each scaling scale, extract the feature components corresponding to the position indication information of each object to obtain the feature components corresponding to the position indication information of each object at different scaling scales. The feature components corresponding to the location indication information of each object at different scaling scales are fused to obtain the local features corresponding to each object location indication information.

4. The method as described in claim 3, characterized in that, The step of extracting feature components corresponding to each object position indication from the global features at each scaling scale to obtain the feature components corresponding to each object position indication at different scaling scales includes: The global features at the target scaling scale are segmented to obtain M feature units, where the target scaling scale is any one of the at least two scaling scales, and M is an integer greater than 1. If the area indicated by the target object location indication information is contained within a feature unit, then based on that feature unit, the feature component corresponding to the target object location indication information under the target scaling scale is determined, and the target object location indication information is any one of the object location indication information in the set of object location indication information; If the area indicated by the target object location indication information is contained within at least two feature units, then based on the at least two feature units, the feature components corresponding to the target object location indication information at the target scaling scale are determined.

5. The method as described in claim 1, characterized in that, The step of obtaining the target detection result of the query text in the image to be detected based on the global features, the local features, and the query text includes: The input sequence of the image to be detected is generated based on the global features and the local features; Using the query text as a detection condition, open domain target detection is performed on the input sequence to obtain the target detection result of the query text in the image to be detected.

6. The method as described in claim 5, characterized in that, The input sequence includes N tokens, each token corresponding to a local feature, where N is an integer greater than 1, and the target detection result is obtained based on the prediction results of the N tokens; The process of using the query text as a detection condition to perform open domain target detection on the input sequence, and obtaining the target detection result of the query text in the image to be detected, includes: Based on the global features and the prediction results of the first i tokens, the prediction result of the (i+1)th token is generated, where i is a positive integer less than N.

7. The method as described in claim 5, characterized in that, The global features of the image to be detected include global features of the image to be detected at at least two scaling scales; the step of generating the input sequence of the image to be detected based on the global features and the local features includes: The global feature projection layer projects the global features of the image to be detected at the target scaling scale into the text feature space to obtain the first projection feature. The target scaling scale is any one of the at least two scaling scales. The local features are projected onto the text feature space through the local feature projection layer to obtain the second projection features. Each second projection feature corresponds to a local feature. The projection parameters of the local feature projection layer are different from the projection parameters of the global feature projection layer. The first projection feature and the second projection feature are combined to obtain the input sequence of the image to be detected.

8. The method as described in claim 1, characterized in that, The query text is a description of the query object; the target detection result of the query text in the image to be detected is used to indicate whether the image to be detected contains the query object. If the target detection result indicates that the image to be detected contains the query object, the target detection result also includes object location indication information corresponding to the query object, which is used to indicate the position of the query object in the image to be detected.

9. A model training method, characterized in that, The method includes: Obtain a sample image, a sample query text, and the verification result corresponding to the sample query text, wherein the sample query text is associated with at least one associated object; Based on the sample query text, open vocabulary target detection is performed on the sample image to obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object. Extract the global features of the sample image, and extract the local features of the sample image based on the global features and the object location indication information set; The model to be trained is invoked to use the sample query text as the detection condition, and open domain target detection is performed on the sample image based on the global features and the local features to obtain the target detection result of the sample query text in the sample image. Based on the target detection results and the verification results, the model to be trained is adjusted to obtain the image detection model.

10. The method as described in claim 9, characterized in that, The acquisition of the sample image, sample query text, and the corresponding verification result of the sample query text includes: Obtain a set of region indication information for a sample image and a sample object, wherein the set of region indication information is used to indicate the region where the sample object is located in the sample image; Based on the sample image, descriptive information of the content of the target region is generated, wherein the target region is the region indicated by any one of the region indication information in the set of region indication information; The description information is decomposed into conditions to obtain a list of conditions for the description information; Using the sample image, the set of region indication information, and the condition list, a sample query text and a corresponding verification result are generated.

11. The method as described in claim 10, characterized in that, The step of generating sample query text and corresponding verification results using the sample image, the region indication information set, and the condition list includes: The region indicated by each region indication information is matched and detected with the conditions in the condition list respectively to obtain the matching result of each region under each condition. Each matching result has a corresponding judgment basis. If the matching result of any region is true under all conditions, then the region indication information corresponding to that region is added to the sample region indication information set corresponding to the description information. A sample query text is generated based on the description information, and a verification result corresponding to the sample query text is generated based on the sample region indication information set.

12. The method as described in claim 9, characterized in that, The target detection result is obtained based on the prediction results of N tokens. The prediction result of each token is used to indicate whether the corresponding local features satisfy the detection condition, where N is an integer greater than 1. The step of adjusting the model to be trained based on the target detection result and the verification result to obtain the image detection model includes: Based on the verification results and the prediction results of the N tokens, the conditional probability of each token is calculated. Construct a cross-entropy loss function using the conditional probabilities of the N tokens; The image detection model is obtained by adjusting the model to be trained based on the cross-entropy loss function.

13. An image detection device, characterized in that, The image detection device includes: An acquisition unit is used to acquire an image to be detected and query text, wherein the query text is associated with at least one associated object; The processing unit is configured to perform open-vocabulary target detection on the image to be detected based on the query text, and obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object; And for extracting global features of the image to be detected, and extracting local features of the image to be detected based on the global features and the object position indication information set; And for obtaining the target detection result of the query text in the image to be detected based on the global features, the local features and the query text.

14. A model training device, characterized in that, The model training device includes: The acquisition unit is used to acquire a sample image, a sample query text, and a verification result corresponding to the sample query text, wherein the sample query text is associated with at least one associated object; The processing unit is configured to perform open-vocabulary target detection on the sample image based on the sample query text to obtain a set of object location indication information; each object location indication information in the set of object location indication information corresponds to an associated object; And for extracting global features of the sample image, and extracting local features of the sample image based on the global features and the object location indication information set; And a function for calling the model to be trained to use the sample query text as a detection condition, and performing open domain target detection on the sample image based on the global features and the local features, to obtain the target detection result of the sample query text in the sample image; And to adjust the model to be trained based on the target detection result and the verification result to obtain an image detection model.

15. A computer device, characterized in that, include: A memory, wherein a computer program is stored; A processor is configured to load the computer program to implement the image detection method as described in any one of claims 1-8; or to load the computer program to implement the model training method as described in any one of claims 9-12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed by the image detection method as described in any one of claims 1-8; or adapted to be loaded by a processor and executed by the model training method as described in any one of claims 9-12.

17. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and executed as the image detection method as described in any one of claims 1-8; or adapted to be loaded by a processor and executed as the model training method as described in any one of claims 9-12.

Images