Multi-modal large model training method and device, target detection method and device and electronic equipment

Abstract

The embodiment of the invention provides a multi-modal large model training method and device, a target detection method and device and electronic equipment, and relates to the technical field of artificial intelligence, and the multi-modal large model training method comprises the steps: obtaining a plurality of sample images and a first sample description text of an object labeling box in the image; utilizing the first large language model and the first text prompt to extract a category name and an attribute of an object described by the first sample description text corresponding to each sample image, and combining at least one of the obtained category name and the attribute to obtain a second sample description text; determining whether each object labeling box in the sample image is matched with each attribute of a plurality of corresponding second sample description texts or not; constructing a sample question containing the second sample description text and a corresponding sample answer in a thinking chain form to obtain a question and answer sample pair; and training the multi-modal large model by using the constructed question and answer sample pair. The complexity of descriptive target detection can be reduced, and the detection efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal large model training method, target detection method, device and electronic equipment. Background Art

[0002] In some application scenarios (e.g., image retrieval or visual question answering), it is often necessary to locate specific objects mentioned in the natural language description text provided by the user in the image (which can be called descriptive target detection). For example, descriptive target detection can be achieved by integrating multiple models. For example, when it is necessary to detect "athletes on the green ground and wearing non-white stockings", the target detection model used to detect "people" can be used to locate the people in the image, and then the classification model that can determine "whether they are wearing stockings" can be used to screen them, and finally the scene recognition model can be used to confirm whether the target is on the green ground. However, this way of integrating multiple models increases the complexity of achieving descriptive target detection and has low detection efficiency. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a multimodal large model training method, target detection method, device and electronic device to reduce the complexity of achieving descriptive target detection and improve detection efficiency. The specific technical solution is as follows:

[0004] In a first aspect of the implementation of the present application, a multimodal large model training method is provided, the method comprising:

[0005] Acquire a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image;

[0006] For each sample image, the first large language model and the first text prompt are used to extract the category name of the object described in the first sample description text corresponding to the sample image and multiple attributes of the object, and the obtained category name is combined with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the first text prompt is used to indicate: extract the category name of the object described in the input description text and the attributes of the object;

[0007] For each object annotation box in the sample image, determining whether the object annotation box matches each attribute of a plurality of second sample description texts corresponding to the sample image that are combined;

[0008] Construct a sample question containing each second sample description text, and use the matching result of each attribute contained in the second sample description text and each object annotation box to construct a sample answer in the form of a thinking chain corresponding to the sample question, so as to obtain a question-answer sample pair; wherein, the sample question is used to indicate the position of the image area in the input image that matches the second sample description text; the sample answer contains a first reasoning process text for describing the multimodal large model reasoning process; the first reasoning process text includes: each reasoning step and a first execution order between each reasoning step; according to the first execution order, each reasoning step is respectively: extracting the category name and the attribute of the object described by the description text contained in the input question, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area and each extracted attribute, and determining the position of the image area that matches each extracted attribute;

[0009] Input the sample image and the sample question in each question-answer sample pair into the multimodal large model of the initial structure to obtain the predicted answer;

[0010] Based on the difference between the predicted answer and the sample answer in the corresponding question and answer sample pair, the parameters of the multimodal large model of the initial structure are adjusted until the preset convergence conditions are reached, thereby obtaining a trained multimodal large model.

[0011] Optionally, before extracting, for each sample image, the category name of the object described in the first sample description text corresponding to the sample image and a plurality of attributes of the object using the first large language model and the first text prompt, and combining the obtained category name with at least one of the obtained attributes to obtain a plurality of second sample description texts corresponding to the sample image, the method further includes:

[0012] For each sample image, obtaining the category name of the object in the object annotation box of the specified category in the sample image as the preset category name;

[0013] The method of extracting the category name of the object described in the first sample description text corresponding to the sample image and multiple attributes of the object by using the first large language model and the first text prompt for each sample image, and combining the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image includes:

[0014] For each sample image, the first text prompt, the first sample description text corresponding to the sample image and the preset category name are input into the first large language model to extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object, and the obtained category name is combined with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the obtained category name is consistent with the category represented by the preset category name.

[0015] Optionally, the step of obtaining a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image includes:

[0016] Obtain multiple sample images and the position of the object annotation box of the specified category in each sample image;

[0017] For each sample image, the sample image and the second text prompt are input into the second largest language model to obtain a first sample description text of an object annotation box of a specified category in the sample image; wherein the second text prompt is used to indicate: generate a description text of the object annotation box in the input image.

[0018] Optionally, for each sample image, using the first large language model and the first text prompt, extracting the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object, and combining the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image, including:

[0019] For each sample image, input the first sample description text and the first text prompt corresponding to the sample image into the first large language model, and extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object;

[0020] The extracted category name, the extracted attributes, and the third text prompt are input into the first language model to obtain multiple second sample description texts corresponding to the sample image; wherein the third text prompt is used to indicate: combine the input category name with at least one of the input attributes to obtain multiple description texts.

[0021] Optionally, for each object annotation box in the sample image, determining whether the object annotation box matches each attribute of a plurality of second sample description texts used for combining to obtain the corresponding sample image includes:

[0022] Using the image-text matching model and the fourth text prompt, it is respectively detected whether the object annotation box of each specified category in the sample image matches each description text to be matched; wherein, each description text to be matched is: each attribute of multiple second sample description texts corresponding to the sample image is combined with the extracted category name; the fourth text prompt is used to indicate: determine whether each object annotation box in the input image matches the input description text.

[0023] In a second aspect of the present application, a target detection method is further provided, the method comprising:

[0024] Acquire an image to be detected and a text prompt to be used containing a description text to be detected; wherein the text prompt to be used is used to indicate: determine the position of the image area occupied by the object that meets the description text to be detected in the image to be detected;

[0025] The image to be detected and the multimodal large model to be pre-trained using the text prompt are input to obtain a detection result in the form of a thought chain; wherein the multimodal large model is trained based on any of the multimodal large model training methods described above; the detection result includes: a second reasoning process text for describing the reasoning process of the multimodal large model; the second reasoning process text includes: each reasoning step and a second execution order between each reasoning step; according to the second execution order, each reasoning step is respectively: extracting the category name of the object described by the input text prompt and the attributes of the object, detecting the positions of the image areas occupied by all objects belonging to the categories represented by the extracted category names in the input image, respectively determining the matching results of each image area with each extracted attribute, and determining the positions of the image areas that match each extracted attribute.

[0026] In a third aspect of the present application, a multimodal large model training device is also provided, the device comprising:

[0027] A sample acquisition module, used to acquire a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image;

[0028] The description text generation module is used to extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object by using the first large language model and the first text prompt for each sample image, and combine the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the first text prompt is used to indicate: extract the category name of the object described by the input description text and the attributes of the object;

[0029] a matching result determination module, for determining, for each object annotation box in the sample image, whether the object annotation box matches each attribute of a plurality of second sample description texts corresponding to the sample image;

[0030] A question-answer sample pair construction module is used to construct a sample question containing each second sample description text, and to use the matching result between each attribute contained in the second sample description text and each object annotation box to construct a sample answer in the form of a thinking chain corresponding to the sample question, so as to obtain a question-answer sample pair; wherein, the sample question is used to indicate the position of the image area in the input image that matches the second sample description text; the sample answer contains a first reasoning process text for describing the multimodal large model reasoning process; the first reasoning process text includes: each reasoning step and a first execution order between each reasoning step; according to the first execution order, each reasoning step is respectively: extracting the category name and the attribute of the object described by the description text contained in the input question, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute;

[0031] A predicted answer determination module is used to input the sample image and the sample question in each question-answer sample pair into the multimodal macro model of the initial structure to obtain a predicted answer;

[0032] The model parameter adjustment module is used to adjust the parameters of the initial structure multimodal large model based on the difference between the obtained predicted answer and the sample answer in the corresponding question and answer sample pair until the preset convergence condition is reached to obtain the trained multimodal large model.

[0033] Optionally, the device further comprises:

[0034] a preset category name acquisition module, for extracting the category name of the object described in the first sample description text corresponding to the sample image and a plurality of attributes of the object by using the first large language model and the first text prompt for each sample image, and combining the obtained category name with at least one of the obtained attributes to obtain a plurality of second sample description texts corresponding to the sample image, and obtaining, for each sample image, the category name of the object in the object annotation box of the specified category in the sample image as the preset category name;

[0035] The description text generation module is specifically used to input, for each sample image, a first text prompt, a first sample description text corresponding to the sample image, and the preset category name into a first large language model, extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object, and combine the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the obtained category name is consistent with the category represented by the preset category name.

[0036] Optionally, the sample acquisition module is specifically used to obtain multiple sample images and the position of the object annotation box of the specified category in each sample image; for each sample image, the sample image and the second text prompt are input into the second largest language model to obtain a first sample description text of the object annotation box of the specified category in the sample image; wherein the second text prompt is used to indicate: generate a description text of the object annotation box in the input image.

[0037] Optionally, the description text generation module is specifically used to, for each sample image, input the first sample description text and the first text prompt corresponding to the sample image into the first large language model, extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object; input the extracted category name, the extracted attributes, and the third text prompt into the first large language model to obtain multiple second sample description texts corresponding to the sample image; wherein the third text prompt is used to indicate: combine the input category name with at least one of the input attributes to obtain multiple description texts.

[0038] Optionally, the matching result determination module is specifically used to use the image-text matching model and the fourth text prompt to respectively detect whether the object annotation box of each specified category in the sample image matches each description text to be matched; wherein, each description text to be matched is: each attribute of a plurality of second sample description texts corresponding to the sample image is combined with the extracted category name; the fourth text prompt is used to indicate: determine whether each object annotation box in the input image matches the input description text.

[0039] In a fourth aspect of the present application, there is also provided a target detection device, the device comprising:

[0040] A data acquisition module is used to acquire an image to be detected and a text prompt to be used containing a description text to be detected; wherein the text prompt to be used is used to indicate: determining the position of an image area occupied by an object in the image to be detected that matches the description text to be detected;

[0041] A detection result determination module is used to input the image to be detected and the text prompt to be used into a pre-trained multimodal large model to obtain a detection result in the form of a thought chain; wherein the multimodal large model is trained based on any of the multimodal large model training methods described above; the detection result includes: a second reasoning process text for describing the reasoning process of the multimodal large model; the second reasoning process text includes: each reasoning step and a second execution order between each reasoning step; according to the second execution order, each reasoning step is respectively: extracting the category name of the object described by the input text prompt and the attributes of the object, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute.

[0042] The present application also provides an electronic device, including:

[0043] Memory, used to store computer programs;

[0044] The processor is used to implement any of the multimodal large model training methods or target detection methods described above when executing the program stored in the memory.

[0045] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it implements any of the multimodal large model training methods or target detection methods described above.

[0046] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the multimodal large model training methods or target detection methods described above.

[0047] Beneficial effects of the embodiments of the present application:

[0048] A multimodal large model training method provided in an embodiment of the present application can obtain a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image; for each sample image, using a first large language model and a first text prompt, extract the category name of the object described in the first sample description text corresponding to the sample image and multiple attributes of the object, and combine the obtained category name with at least one of the obtained attributes to obtain a plurality of second sample description texts corresponding to the sample image; wherein the first text prompt is used to indicate: extract the category name of the object described in the input description text and the attributes of the object; for each object annotation box in the sample image, determine whether the object annotation box matches each extracted attribute; construct a sample question containing each second sample description text, and use the matching results of each attribute contained in the second sample description text and each object annotation box to construct a sample answer in the form of a thinking chain corresponding to the sample question to obtain a question-answer sample pair; wherein the sample question is used to The method is used to indicate the position of the image area in the input image that matches the second sample description text; the sample answer includes a first reasoning process text for describing the reasoning process of the multimodal large model; the first reasoning process text includes: each reasoning step and a first execution order between each reasoning step; according to the first execution order, each reasoning step is respectively: extracting the category name of the object described by the description text contained in the input question and the attributes of the object, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute; the sample image and the sample question in each question-answer sample pair are input into the multimodal large model of the initial structure to obtain a predicted answer; based on the difference between the obtained predicted answer and the sample answer in the corresponding question-answer sample pair, the parameters of the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached to obtain the trained multimodal large model.

[0049] Based on the above processing, the second sample description text can be generated using the first large language model according to the acquired sample image and the description text (i.e., the first sample description text) of the object annotation box used to describe the specified category in the sample image, so as to increase the number of the obtained second sample description texts and enrich the diversity of the second sample description texts. Furthermore, a sample question containing each second sample description text can be constructed to instruct the multimodal large model to determine the position of the image area in the input image that matches the second sample description text. And the matching results of each object annotation box in each sample image and each attribute of the multiple second sample description texts used to combine and obtain the sample image can be combined to construct a sample answer in the form of a thinking chain, so as to obtain a question-answer sample pair in the form of a thinking chain. The sample answer in the form of a thinking chain can include: the specific steps of inferring the position of the image area in the sample image that matches each attribute in the second sample description text.

[0050] Accordingly, training a large multimodal model based on question-answer samples in the form of thought chains can enable the large multimodal model to learn to reason in the form of thought chains, determine the location of the image area occupied by objects in the image that match the input description text, and generate answers in the form of thought chains. Subsequently, using the trained large multimodal model, end-to-end descriptive object detection can be achieved without the need to integrate multiple models, which can reduce the complexity of achieving descriptive object detection and improve detection efficiency.

[0051] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0053] Figure 1 A first flow chart of a multimodal large model training method provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of a sample image provided in an embodiment of the present application;

[0055] Figure 3 A second flow chart of the multimodal large model training method provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of a flow chart of obtaining a second sample description text by combining provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of a process for constructing a question-answer sample pair provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of a target detection method provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the structure of a multimodal large model training device provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0061] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0063] In order to reduce the complexity of achieving descriptive object detection and improve detection efficiency, the present application embodiment provides a multimodal large model training method, see Figure 1 , Figure 1 A first flow chart of a multimodal large model training method provided in an embodiment of the present application. The multimodal large model training method may include:

[0064] Step S101: obtaining a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image.

[0065] Step S102: For each sample image, using the first large language model and the first text prompt, extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object, and combine the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image.

[0066] The first text prompt is used to indicate: extract the category name of the object described by the input description text and the attributes of the object.

[0067] Step S103: for each object annotation box in the sample image, determining whether the object annotation box matches each attribute of a plurality of second sample description texts corresponding to the sample image that are combined.

[0068] Step S104: construct a sample question containing each second sample description text, and use the matching result of each attribute contained in the second sample description text and each object annotation box to construct a sample answer in the form of a thought chain corresponding to the sample question, so as to obtain a question-answer sample pair.

[0069] Among them, the sample question is used to indicate the position of the image area in the input image that matches the second sample description text; the sample answer contains a first reasoning process text used to describe the multimodal large model reasoning process; the first reasoning process text includes: each reasoning step and a first execution order between each reasoning step; according to the first execution order, each reasoning step is respectively: extracting the category name of the object described by the description text contained in the input question and the attributes of the object, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching results of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute.

[0070] Step S105: input the sample image and the sample question in each question-answer sample pair into the multimodal macro model of the initial structure to obtain a predicted answer;

[0071] Step S106: Based on the difference between the predicted answer and the sample answer in the corresponding question and answer sample pair, the parameters of the multimodal large model of the initial structure are adjusted until the preset convergence conditions are reached to obtain a trained multimodal large model.

[0072] Based on the above processing, the second sample description text can be generated using the first large language model according to the acquired sample image and the description text (i.e., the first sample description text) of the object annotation box used to describe the specified category in the sample image, so as to increase the number of the obtained second sample description texts and enrich the diversity of the second sample description texts. Furthermore, a sample question containing each second sample description text can be constructed to instruct the multimodal large model to determine the position of the image area in the input image that matches the second sample description text. And the matching results of each object annotation box in each sample image and each attribute of the multiple second sample description texts used to combine and obtain the sample image can be combined to construct a sample answer in the form of a thinking chain, so as to obtain a question-answer sample pair in the form of a thinking chain. The sample answer in the form of a thinking chain can include: the specific steps of inferring the position of the image area in the sample image that matches each attribute in the second sample description text.

[0073] Accordingly, training a large multimodal model based on question-answer samples in the form of thought chains can enable the large multimodal model to learn to reason in the form of thought chains, determine the location of the image area occupied by objects in the image that match the input description text, and generate answers in the form of thought chains. Subsequently, using the trained large multimodal model, end-to-end descriptive object detection can be achieved without the need to integrate multiple models, which can reduce the complexity of achieving descriptive object detection and improve detection efficiency.

[0074] For step S101, multiple sample images and sample description texts (i.e., first sample description texts) of object annotation boxes of a specified category in each sample image may be obtained. The specified category may be set according to the category of the target object that currently needs to be detected. For example, if the target object that currently needs to be detected is an athlete, the specified category may be an athlete; or, if the target object that currently needs to be detected is a vehicle, the specified category may be a vehicle. An object annotation box of a specified category in an image may represent a bounding box of an image area occupied by objects of the specified category in the image. The first sample description text of an object annotation box is used to describe the image content of the image area indicated by the object annotation box. The number of object annotation boxes of a specified category in a sample image is the same as the number of objects of the specified category in the sample image, which may be one or more.

[0075] Each sample image and the object annotation box of the specified category in each sample image can come from a public data set (for example, a public target detection data set). Alternatively, the sample images can be acquired using an image acquisition device, and the objects belonging to the specified category in each acquired sample image can be manually annotated to obtain the object annotation box of the specified category in each sample image. The first sample description text of the object annotation box of the specified category in each sample image can also come from a public data set, or the description text of the object annotation box of the specified category in each sample image can be manually determined to obtain the first sample description text. For example, the sample image can be a photo of a football field with multiple players on the field playing a football game. The specified category can be an athlete, and accordingly, the object annotation box of the specified category in the sample image can be a bounding box of the image area occupied by each athlete. Figure 2 As shown, Figure 2 A schematic diagram of a sample image provided by an embodiment of the present application. The sample image contains three football players, namely A, B and C. The object annotation frames of the specified category in the sample image include: an object annotation frame 21 of football player A, an object annotation frame 22 of football player B, and an object annotation frame 23 of football player C. Football players B and C are wearing green uniforms ( Figure 2 Football player A is wearing a yellow uniform ( Figure 2 ) and black stockings ( Figure 2 If the object is a football player wearing a yellow uniform and black socks, and is running, the description text of the object annotation frame 21 may be “in the framed area, there is a football player wearing a yellow uniform and black socks, and he is running”.

[0076] In one implementation, for each object annotation box of a specified category in each sample image, a model with the ability to generate region descriptions can also be used to generate description text for the object annotation box of the specified category in the sample image to obtain a first sample description text. Step S101 includes:

[0077] Step 1: Obtain multiple sample images and the location of the object annotation box of the specified category in each sample image.

[0078] Step 2: For each sample image, the sample image and the second text prompt are input into the second largest language model to obtain the first sample description text of the object annotation box of the specified category in the sample image.

[0079] The second text prompt is used to instruct: generating a description text of the object annotation box in the input image.

[0080] In this implementation, the second text prompt may instruct the second largest language model to generate description text for the object annotation box in the input image. In one implementation, the second text prompt may include information about the location of the object annotation box in the input image (e.g., the coordinates of the object annotation box). For example, the second text prompt may be "Please generate description text for the image area in the rectangular box with the upper left vertex coordinates (a, b) and the lower right vertex coordinates (c, d) in the input image." In another implementation, an object annotation box may be added to the input image, and the image with the object annotation box added may be input into the second largest language model for processing, and the second text prompt may instruct the second largest language model to generate description text for the image area in the object annotation box that has been added to the input image. For example, the second text prompt may be "Please generate description text for the image area in the object annotation box that has been added to the input image."

[0081] For each sample image, the sample image and the position of the object annotation box of the specified category in the sample image can be obtained, and then the sample image and the second text prompt can be input into the second largest language model to obtain the first sample description text of the object annotation box of the specified category in the sample image. The second largest language model can be a pre-trained multimodal large language model with regional description capabilities, which can be any multimodal large language model that can generate description text of any image area in the image, without specific limitation. For example, the second largest language model can be Qwen2.5-VL or ChatGPT.

[0082] Based on the above processing, for each sample image, the second largest language model can be used to generate the description text of the object annotation box in the sample image (i.e., the first sample description text). In this way, it is guaranteed that the first sample description text can be obtained. Subsequently, based on the sample image and the first sample image, a question-answer sample pair in the form of a thinking chain can be constructed, and then the multimodal large model can be trained. It is guaranteed that the trained multimodal large model can be used to achieve end-to-end descriptive object detection in the future, so there is no need to integrate multiple models, which can reduce the complexity of achieving descriptive object detection and improve detection efficiency.

[0083] For step S102, for each sample image, the first large language model can be used, combined with the first text prompt, to process the first sample description text corresponding to the sample image, and extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object. In order to ensure the accuracy of the obtained category name, the category name extracted by the first large language model is consistent with the category name of the object in the object annotation box of the specified category in the sample image. For details, please refer to the relevant instructions of step S107 and step S1021 in the subsequent embodiments. The first large language model can be any large language model capable of implementing step S102, without specific limitation. For example, the first large language model can be Qwen2.5-72B or ChatGPT.

[0084] The category name of the object described by a description text may indicate the category to which the object (also referred to as the subject) described by the description text belongs. For example, if the object described by a description text is a player on a football field, the category name of the object described by the description text may be a football player; or, if the object described by a description text is a little girl in a park, the category name of the object described by the description text may be a girl. The object described by a description text may be an objective entity, such as a person, an animal, a car, or a plant. The attributes of the object described by a description text may be attribute information that can be perceived visually, such as appearance, material, interaction relationship, or action state. For example, a first sample description text is "In the framed area, there is a football player, wearing a yellow uniform and black socks, and he is running." For the first sample description text, the extracted category name may be a football player, and the extracted attributes may be: wearing a yellow uniform, wearing black socks, and running.

[0085] The obtained category name can be combined with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image. For example, at least one attribute can be selected from the obtained attributes to obtain multiple attribute sets (also called non-empty attribute subsets), and a preset number of attribute sets can be selected from the obtained multiple attribute sets, and the selected attribute sets can be combined with the obtained category name to obtain multiple second sample description texts corresponding to the sample image. The preset number is greater than 1 and not greater than the total number of attribute sets. If there are N attributes obtained, the total number of attribute sets is . For example, the extracted category name is football player, and the extracted attributes are: wearing yellow uniform, wearing black socks, and running. You can select any one, any two, or three of the three extracted attributes and combine them with the extracted category name. For example, if you select "wearing yellow uniform" and "wearing black socks", the second sample description text obtained by combining them can be "football player wearing yellow uniform and black socks"; if you select "running", the second sample description text obtained by combining them can be "football player running".

[0086] In this application, the first language model can be instructed to extract the category name and the attribute, and the obtained category name and the obtained attribute can be combined in the following two ways:

[0087] In the first method, the first text prompt and the first language model can be used to extract the category name and the attribute. The third text prompt and the first language model can be used to combine the obtained category name and the obtained attribute to obtain multiple second sample description texts. Step S102 includes:

[0088] Step 1: For each sample image, the first sample description text and the first text prompt corresponding to the sample image are input into the first large language model to extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object.

[0089] Step 2: Input the extracted category name, the extracted attributes, and the third text prompt into the first language model to obtain a plurality of second sample description texts corresponding to the sample image.

[0090] The third text prompt is used to indicate: combining the input category name with at least one of the input attributes to obtain multiple description texts.

[0091] In an embodiment of the present application, for each sample image, the first large language model can extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object according to the indication of the first text prompt, and can combine the extracted category name with at least one of the extracted attributes according to the indication of the third text prompt to obtain multiple second sample description texts corresponding to the sample image. In this way, the first sample description text can be disassembled by the first large language model, and multiple new description texts (i.e., second sample description texts) can be combined. The processing ability of the first large language model for text can be used to conveniently and efficiently expand the description text. Furthermore, it is ensured that question-answer sample pairs in the form of a thinking chain can be constructed in the future to realize the training of the multimodal large model. Accordingly, using the trained multimodal large model, end-to-end descriptive object detection can be achieved without the need to integrate multiple models, which can reduce the complexity of achieving descriptive object detection and improve detection efficiency.

[0092] The first text prompt is used to indicate: extract the category name of the object described by the input description text and the attributes of the object. The first text prompt may include: outputting the format of the extracted category name and the extracted attributes, extracting the category name of the object described by the input description text and the extraction rules that the attributes of the object need to meet, and extracting samples of category names and attributes based on the input description text. In this way, it can be guaranteed that the category names and attributes that meet the requirements can be extracted. The following is an example of a first text prompt:

[0093] "You are a helpful assistant. I will provide you with a description based on a target box and its corresponding original target detection category name. Please help me extract visual information based on it. The format is [subject, first attribute, second attribute, ..., Nth attribute], where "subject" represents the target subject, "first attribute", "second attribute", ..., "Nth attribute" represents the attribute information related to the subject. Visual information must be extracted as strictly, diversely and concisely as possible. Here are some rules: 1. The extracted subject must be consistent with the given original target detection category name, and it must be a synonym or near-synonym. If it is inconsistent, the output should be abandoned; 2. The subject refers to entities such as people, animals, cars, objects, plants, etc., rather than non-entities such as environment and atmosphere. 3. Extract objective descriptions, rather than subjective ones, such as reasoning or guessing or emotional descriptions. 4. If the "subject" or "attribute" cannot be accurately extracted, the output should be abandoned. 5. Attributes refer to appearance, material, action relationship, state, etc. They cannot be functional words, reasoning, guessing or emotions.

[0094] For example:

[0095] Original test category name: Boys

[0096] Area Description: The young boy is wearing a blue helmet with a pattern of white and black stripes. The helmet appears to be securely fastened under his chin, and he is smiling, showing his teeth. The boy has short, light brown hair, and is wearing a dark blue T-shirt. The boy is riding a bicycle, and his smile shows that he enjoys the activity. He is in the middle of a group of children and bicycles, suggesting a group activity or family outing.

[0097] Output:

[0098] [Little boy, wearing a blue helmet, smiling, with short light brown hair, and a dark blue T-shirt, riding a bicycle].

[0099] Please extract visual information for the following region descriptions and category names:

[0100] Original detection category name: {detection category name of the box to be processed}

[0101] Area description: {area description of the box to be processed}".

[0102] In the above example, the extracted subject can represent the extracted category name, the original detection category name can represent the category name of the object in the object annotation box of the specified category in the sample image to be processed, and the region description is the description text. For each sample image, the detection category name of the box to be processed is the category name of the object in the object annotation box of the specified category in the sample image, and the region description of the box to be processed is the first sample description text corresponding to the sample image.

[0103] The third text prompt is used to indicate: combine the input category name with at least one of the input attributes to obtain multiple description texts. The third text prompt may include: combine the input category name with at least one of the input attributes to obtain multiple description text samples. In this way, it can be ensured that the description text that meets the requirements can be combined. The following is an example of the third text prompt:

[0104] "You are a helpful assistant. I will give you a set of visual information in the format of [subject, attribute 1, attribute 2, ..., attribute N], where "subject" represents the target entity, and "attribute 1, attribute 2, ..., attribute N" and so on represent the N attributes of the target. Please help me complete the following task: combine the N attributes into any non-empty subsets, and combine them with the subject to merge them into the target description. Ensure that the final expression should be fluent and accurate. For example:

[0105] Visual Relationship:

[0106] ['Boy, wearing a blue helmet, smiling, riding a bicycle']

[0107] result:

[0108] ['Boy, wearing a blue helmet']: A boy wearing a blue helmet

[0109] ['Boy, smile']: A boy with a smile

[0110] ['Boy, riding a bicycle']: A boy riding a bicycle

[0111] ['Boy, wearing a blue helmet, smiling']: A boy wearing a blue helmet and smiling

[0112] ['Boy, wearing a blue helmet, riding a bicycle']: A boy wearing a blue helmet riding a bicycle

[0113] ['Boy, smiling, riding a bicycle']: A smiling boy riding a bicycle

[0114] ['Boy, wearing a blue helmet, smiling, riding a bicycle']: A boy wearing a blue helmet, riding a bicycle and smiling.

[0115] Please synthesize a target description for the following visual relationship:

[0116] Visual relationship: {Visual relationship to be processed}".

[0117] In the above example, the visual relationship to be processed is the extracted category name and the extracted attributes, and the target description is the second sample description text.

[0118] In the second method, the first text prompt is used to indicate: extract the category name of the object described by the input description text and the attributes of the object, and combine the extracted category name with at least one of the extracted attributes to obtain a description text. For example, the first text prompt can be "You are a helpful assistant. I will provide you with a description based on a target box and its corresponding original target detection category name. Please help me extract visual information based on it. The format is [subject, first attribute, second attribute,..., Nth attribute], where "subject" represents the target subject, "first attribute", "second attribute",..., "Nth attribute" represents attribute information related to the subject. Visual information must be extracted as strictly, diversely and concisely as possible. Here are some rules: 1. The extracted subject must be consistent with the given original target detection category name and be a synonym or near-synonym. If not 1. If the "subject" or "attribute" cannot be accurately extracted, the output should be abandoned; 2. The subject refers to entities such as people, animals, cars, objects, plants, etc., rather than non-entities such as environment and atmosphere. 3. Extract objective descriptions, rather than subjective ones, such as reasoning or guessing or emotional descriptions. 4. If the "subject" or "attribute" cannot be accurately extracted, the output should be abandoned. 5. Attributes refer to appearance, material, action relationship, state, etc. They cannot be functional words, reasoning, guessing or emotions. And combine the N extracted attributes into any non-empty subset, and combine them with the extracted subject, and merge them into the target description respectively. Ensure that the final expression should be fluent and accurate. Please process based on the following area descriptions and category names:

[0119] Original detection category name: {detection category name of the box to be processed}

[0120] Area description: {area description of the box to be processed}".

[0121] For step S103, for each object annotation frame in the sample image, determine whether the object annotation frame matches each attribute of the plurality of second sample description texts used to combine to obtain the sample image. Matching an object annotation frame with an attribute means that the image content in the object annotation frame conforms to the attribute.

[0122] In one implementation, it may be manually determined whether the object annotation box matches each attribute of the plurality of second sample description texts used for combining to obtain the corresponding sample image.

[0123] In another implementation, step S103 includes: using the image-text matching model and the fourth text prompt to respectively detect whether the object annotation box of each specified category in the sample image matches each description text to be matched.

[0124] Among them, each description text to be matched is: each attribute of multiple second sample description texts corresponding to the sample image is combined with the extracted category name; the fourth text prompt is used to indicate: determine whether each object annotation box in the input image and the input description text match.

[0125] In an embodiment of the present application, for each object annotation box in the sample image, a model (which may be referred to as a graph-text matching model) that can determine whether the image content in the object annotation box in the image matches the description text can be used to determine whether the object annotation box matches each attribute of the multiple second sample description texts corresponding to the sample image. That is, a graph-text matching model that has the ability to determine the referential expression matching result can be used to determine whether the object annotation box matches each attribute of the multiple second sample description texts corresponding to the sample image. The graph-text matching model can be any multimodal large language model that has the ability to determine the referential expression matching result, without specific limitation. For example, the graph-text matching model can be Qwen2.5-VL. The matching of an object annotation box with a description text can indicate that the image content in the object annotation box conforms to the description text.

[0126] Each attribute used to combine to obtain the plurality of second sample description texts corresponding to the sample image may be combined with the extracted category name in advance to obtain each description text to be matched. For example, the extracted category name may be a football player, and the attributes used to combine to obtain the plurality of second sample description texts corresponding to the sample image may include: wearing a yellow uniform, wearing black socks, and running. Accordingly, each description text to be matched may be: a football player wearing a yellow uniform, a football player wearing black socks, and a football player running.

[0127] By using the image-text matching model, according to the instruction of the fourth text prompt, it is possible to respectively detect whether the object annotation box of each specified category in the sample image matches each description text to be matched. For example, the description text to be matched, the sample image and the fourth text prompt can be input into the image-text matching model to obtain a matching result output by the image-text matching model that characterizes whether the object annotation box of each specified category in the sample image matches. All description texts to be matched can be input into the image-text matching model at one time. Alternatively, all description texts to be matched can be input into the image-text matching model in multiple times until all description texts to be matched are detected. The number of description texts to be matched input into the image-text matching model each time can be set as needed without specific limitation.

[0128] In one implementation, the fourth text prompt may include information about the location of each object annotation box in the input image (e.g., the coordinates of the object annotation box). For example, the fourth text prompt may be "Please determine whether the rectangular box with the upper left vertex coordinates (a1, b1) and the lower right vertex coordinates (c1, d1) in the input image, and the rectangular box with the upper left vertex coordinates (a2, b2) and the lower right vertex coordinates (c2, d2), respectively, match each input description text to be matched". In another implementation, an object annotation box may be added to the input image, and the image with the object annotation box added may be input into the image-text matching model for processing, and the fourth text prompt may instruct the image-text matching model to determine whether the object annotation box added in the input image matches each input description text to be matched. For example, the fourth text prompt may be "Please determine whether each object annotation box added in the input image matches each input description text to be matched".

[0129] For example, the matching results obtained can be displayed in the form of an attribute table, and the attribute table is shown in Table 1. The object annotation boxes in the sample image may include: box No. 1, box No. 2, and box No. 3. The description texts to be matched may be: a football player in a yellow uniform, a football player in black socks, and a football player running. "√" indicates a match, and "×" indicates a mismatch. Taking the matching result of box No. 1 and a football player in a yellow uniform as an example, box No. 1 matches the football player in a yellow uniform, and accordingly, it means that box No. 1 matches the attribute "in a yellow uniform".

[0130] Table 1

[0131]

[0132] Based on the above processing, the matching result of each object annotation box in each sample image and each attribute of the multiple second sample description texts corresponding to the sample image can be obtained, that is, the matching result of each attribute contained in each second sample description text and each object annotation box in each sample image can be obtained. In this way, it is ensured that the question-answer sample pairs can be constructed based on the obtained matching results in the future to realize the training of the multimodal large model. It is ensured that the trained multimodal large model can be used in the future to realize end-to-end descriptive object detection, so there is no need to integrate multiple models, which can reduce the complexity of realizing descriptive object detection and improve detection efficiency.

[0133] For step S104, for each second sample description text, a sample question containing the second sample description text can be constructed. The constructed sample question can be used to indicate: determine the position of the image area in the input image that matches the second sample description text. For example, a second sample description text may be "an athlete wearing non-white stockings on a green field", and a sample question containing the second sample description text may be "Please provide all target box coordinates corresponding to this description in the figure: 'an athlete wearing non-white stockings on a green field'. Please use a chain of thought to reason to get the final answer".

[0134] And the matching result of each attribute contained in the second sample description text and each object annotation box can be used to construct a sample answer in the form of a thinking chain corresponding to the sample question. The sample answer contains a first reasoning process text for describing the multimodal large model reasoning process. The first reasoning process text may include: each reasoning step and the first execution order between each reasoning step. According to the first execution order, each reasoning step is: extracting the category name of the object described by the description text contained in the input question and the attribute of the object (which can be called description disassembly), detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image (which can be called open set target detection), respectively determining the matching result of each image area and each extracted attribute (which can be called frame-by-frame and attribute-by-attribute matching), and determining the position of the image area that matches each extracted attribute (which can be called summary).

[0135] In the process of description decomposition, the extracted category name is the category name used when the second sample description text is combined in step S102, and the extracted attribute is the attribute used when the second sample description text is combined in step S102. In the process of open set target detection, the position of the detected image area is the position of the object annotation box of the specified category in the sample image corresponding to the second sample description text, and the position of the object annotation box can be represented by the coordinates of the upper left corner vertex and the lower right corner vertex. In the process of frame-by-frame and attribute-by-attribute matching, the matching result of each image area and each extracted attribute is the matching result of each object annotation box in the sample image corresponding to the second sample description text and each attribute contained in the second sample description text determined in step S103. In the process of summarizing, the position of the image area that matches each extracted attribute is the position of the object annotation box that matches each attribute contained in the second sample description text. In the matching results determined in step S103, the object annotation box that matches each attribute contained in the second sample description text can be found, and the position of the found object annotation box can be used as the position determined in the summary. For example, an object annotation box that matches each attribute included in the second sample description text may be searched in the attribute table. For example, if the second sample description text is "a football player wearing a yellow uniform and black socks", an object annotation box that matches the attributes "wearing a yellow uniform" and "wearing black socks" may be searched in Table 1, that is, the first box. Accordingly, the position of the first box may be used as the position determined in the summary.

[0136] For example, for the second sample description text "Athletes wearing non-white socks on the green field", the constructed sample answers are as follows:

[0137] "Step 1: (Describe the disassembly)

[0138] First, split the description's main body and sub-attributes, which can be extracted as:

[0139] Subject: 'athlete', Attributes: 'athlete wearing non-white socks; athlete playing on a green field'.

[0140] Step 2: (Open set target detection)

[0141] All bounding boxes of the subject athlete in the image are: [008,080,242,494]; [260,081,460,511]; [521,085,772,476].

[0142] Step 3: (Matching by frame and attribute)

[0143] For all bounding boxes and all attributes in the description, the matches are as follows:

[0144] [008,080,242,494]: Athletes wearing non-white stockings: No; Athletes playing on a green field: Yes;

[0145] [260,081,460,511]: Athletes wearing non-white stockings: No; Athletes playing on a green field: Yes;

[0146] [521,085,772,476]: Athletes wearing non-white stockings: yes; Athletes playing on a green field: yes;

[0147] Step 4: (Summary)

[0148] In summary, the bounding box coordinates of the region described by this sentence are: [521,085,772,476]. ”

[0149] In this way, the descriptive object detection task can be divided into a paradigm of step-by-step reasoning of four reasoning steps: description disassembly, open set object detection, frame-by-frame and attribute-by-attribute matching, and summary. Subsequently, the multimodal large model is trained using the question-answer sample pairs constructed in the form of a thought chain, which enables the multimodal large model to learn to reason step by step according to the above four reasoning steps, determine the position of the image area occupied by the object in the image that matches the input description text, and generate answers in the form of a thought chain.

[0150] For step S105 and step S106, the sample image and the sample question in each question-answer sample pair are input into the multimodal large model of the initial structure to obtain a predicted answer. The question-answer sample pair corresponds to the obtained predicted answer. Furthermore, the difference between the obtained predicted answer and the sample answer in the corresponding question-answer sample pair can be calculated. For example, the similarity between the obtained predicted answer and the sample answer in the corresponding question-answer sample pair can be calculated to represent the difference between the two. According to the difference between the obtained predicted answer and the sample answer in the corresponding question-answer sample pair, the parameters of the multimodal large model of the initial structure can be adjusted until the preset convergence condition is reached to obtain the trained multimodal large model. For example, the preset convergence condition can be that the number of times the parameters of the multimodal large model are adjusted reaches a preset number, or it can also be that the difference between the obtained predicted answer and the sample answer in the corresponding question-answer sample pair is less than the preset difference.

[0151] In this way, using the trained multimodal large model, end-to-end descriptive object detection can be achieved without integrating multiple models, which can reduce the complexity of implementing descriptive object detection and improve detection efficiency. In addition, the multimodal large model can be reasoned step by step according to the reasoning steps of description decomposition, open set object detection, frame-by-frame and attribute-by-attribute matching, and summary, which can accurately and logically achieve complex descriptive object detection.

[0152] It can be seen that in the solution provided by this application, a descriptive target detection paradigm based on thought chains is proposed, and a corresponding descriptive target detection data set in the form of thought chains is constructed, so as to make full use of the generative characteristics and logical reasoning ability of the multimodal large model, and can make full use of the complementarity of image and text information to achieve accurate understanding of targets and their attributes in complex scenes, and can improve the performance of descriptive target detection tasks. The multimodal large model trained based on the multimodal large model training method provided by this application can realize the detection of open set targets with natural language description restrictions, providing more flexible and powerful target detection capabilities for intelligent visual systems.

[0153] In one embodiment, see Figure 3 , Figure 3 A second flow chart of the multimodal large model training method provided in the embodiment of the present application. Before step S102, the multimodal large model training method further includes:

[0154] Step S107: For each sample image, obtain the category name of the object in the object annotation box of the specified category in the sample image as the preset category name.

[0155] Step S102 includes:

[0156] Step S1021: For each sample image, the first text prompt, the first sample description text corresponding to the sample image and the preset category name are input into the first large language model to extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object, and the obtained category name is combined with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image.

[0157] The obtained category name is consistent with the category represented by the preset category name.

[0158] In the embodiment of the present application, affected by the accuracy of the first language model, the category name of the object described by the input description text identified by the first language model may not be accurate. In order to ensure the accuracy of the category name output by the first language model, for each sample image, the category name of the object in the object annotation box of the specified category in the sample image can be obtained as the preset category name. The preset category name can also be used as an accurate category name. And the first text prompt, the first sample description text corresponding to the sample image and the preset category name can be input into the first language model. The first language model can extract the category name consistent with the category represented by the preset category name. If the first language model extracts the category name that is consistent with the preset category name during the process of processing the first sample description text corresponding to a sample image, the output is abandoned. For example, the first language model can determine whether the extracted category name belongs to a synonym or a synonym of the preset category name. If so, it is determined that the extracted category name is consistent with the preset category name and can be output; otherwise, it is determined that the extracted category name is inconsistent with the preset category name and the output is abandoned.

[0159] Based on the above processing, the category name of the object in the object annotation box of the specified category in the sample image (i.e., the preset category name) can be used as a benchmark to ensure the accuracy of the category name extracted by the first large language model, and then ensure the accuracy of the combined second sample description text. Correspondingly, the accuracy of the constructed question and answer sample pairs can be guaranteed, and then, the training of the multimodal large model can be guaranteed. It is guaranteed that the trained multimodal large model can be used to achieve end-to-end descriptive target detection in the future, so there is no need to integrate multiple models, which can reduce the complexity of achieving descriptive target detection and improve detection efficiency.

[0160] In one embodiment, see Figure 4 , Figure 4 A flowchart of a combination to obtain a second sample description text provided in an embodiment of the present application. For each sample image, the first text prompt, the box detection category name (i.e., the preset category name in the above embodiment) and the box area description (i.e., the first sample description text corresponding to the sample image in the above embodiment) can be input into a large language model (i.e., the first large language model in the above embodiment) to extract visual information. That is, the category name (also referred to as the subject) of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object (such as Figure 4 The first attribute, the second attribute, ..., the Nth attribute in the sample image). The large language model can be used to combine the extracted category name with at least one of the obtained attributes to obtain a description text containing all attributes or any sub-attribute set (i.e., multiple second sample description texts corresponding to the sample image). The sub-attribute set is the other attribute set in the attribute set in the above embodiment except the attribute set containing all attributes. In this way, the main body and N sub-attributes can be decomposed, and any sub-attribute set and the main body can be combined, so as to generate the second sample description text more efficiently and fine-grained.

[0161] In one embodiment, see Figure 5 , Figure 5 A schematic diagram of a process for constructing a question-answer sample pair provided in an embodiment of the present application. The process includes the following steps:

[0162] Step S501: Obtain a traditional target detection dataset. For a sample image in the target detection dataset, for a certain category (ie, the specified category in the above embodiment), the sample image may have M boxes (ie, object annotation boxes of the specified category in the above embodiment).

[0163] Step S502: for any one or several frames in the sample image, a multimodal model with region description capability is used to generate a detailed region description, that is, step 1 and step 2 in the above embodiment.

[0164] Step S503: For each region description, use the large language model to extract the subject and N attributes, and arbitrarily combine them into A non-empty attribute subset is synthesized with the subject, that is, step S102 in the above embodiment.

[0165] Step S504: Perform consistency check of M boxes and N attributes through the designation expression matching model to generate an attribute table. The designation expression matching model is the image-text matching model in the above embodiment. That is, the image-text matching model and the fourth text prompt are used to detect whether each object annotation box of each specified category in the sample image matches each description text to be matched.

[0166] Step S505: Format conversion to generate descriptive object detection training data in the form of thought chain, that is, step S104 in the above embodiment.

[0167] Based on the same inventive concept, the present application also provides a target detection method, see Figure 6 , Figure 6 A schematic diagram of a target detection method provided in an embodiment of the present application, the target detection method includes:

[0168] Step S601: Acquire an image to be detected and a text prompt to be used including a description text to be detected.

[0169] The text prompt to be used is used to indicate: determining the position of the image area occupied by the object that matches the description text to be detected in the image to be detected.

[0170] Step S602: Input the image to be detected and the text prompt to be used into a pre-trained multimodal large model to obtain the detection result in the form of a thought chain.

[0171] Among them, the multimodal large model is trained based on any of the above-mentioned multimodal large model training methods; the detection result includes: a second reasoning process text for describing the multimodal large model reasoning process; the second reasoning process text includes: each reasoning step and a second execution order between each reasoning step; according to the second execution order, each reasoning step is respectively: extracting the category name of the object described by the input text prompt and the attributes of the object, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute.

[0172] In an embodiment of the present application, an image that currently needs to perform target detection (i.e., an image to be detected) and a text prompt to be used that contains a description text that matches the target that currently needs to be detected (i.e., a description text to be detected) can be obtained. The text prompt to be used can be used to indicate: determine the position of the image area occupied by the object that matches the description text to be detected in the image to be detected. For example, a description text to be detected can be "an athlete wearing black stockings on a green field", and the text prompt to be used that contains the description text to be detected can be "Please provide all the target box coordinates corresponding to this description in the figure: 'Athletes wearing black stockings on a green field'. Please use the form of thought chain reasoning to get the final answer."

[0173] Furthermore, the image to be detected and the text prompt to be used are input into the pre-trained multimodal large model. The multimodal large model can extract the category name of the object described in the description text to be detected in the text prompt to be used and the attributes of the object (i.e., decomposing the description), and detect the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the image to be detected (i.e., performing open set target detection), respectively determine the matching result of each image area with each extracted attribute (i.e., performing frame-by-frame and attribute-by-attribute matching), and determine the position of the image area that matches each extracted attribute (i.e., performing summary). That is, the multimodal large model can perform reasoning step by step according to the reasoning process of description decomposition, open set target detection, frame-by-frame and attribute-by-attribute matching, and summary to determine the position of the image area occupied by the object in the image to be detected that matches the description text to be detected, and output an answer in the form of a thought chain.

[0174] Based on the above processing, the multimodal large model can learn to reason in the form of thought chains during training, determine the location of the image area occupied by the object that matches the input description text, and generate answers in the form of thought chains. Using the trained multimodal large model, end-to-end descriptive object detection can be achieved without the need to integrate multiple models, which can reduce the complexity of implementing descriptive object detection and improve detection efficiency.

[0175] The present application also provides a multi-modal large model training device, see Figure 7 , Figure 7 A schematic diagram of the structure of a multimodal large model training device provided in an embodiment of the present application, the device comprising:

[0176] The sample acquisition module 701 is used to acquire a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image;

[0177] The description text generation module 702 is used to extract, for each sample image, the category name of the object described in the first sample description text corresponding to the sample image and multiple attributes of the object by using the first large language model and the first text prompt, and combine the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the first text prompt is used to indicate: extract the category name of the object described in the input description text and the attributes of the object;

[0178] A matching result determination module 703 is used to determine, for each object annotation box in the sample image, whether the object annotation box matches each attribute of the plurality of second sample description texts used for combining to obtain the corresponding sample image;

[0179] The question-answer sample pair construction module 704 is used to construct a sample question containing each second sample description text, and to construct a sample answer in the form of a thinking chain corresponding to the sample question using the matching result between each attribute contained in the second sample description text and each object annotation box, so as to obtain a question-answer sample pair; wherein the sample question is used to indicate the position of the image area in the input image that matches the second sample description text; the sample answer contains a first reasoning process text for describing the multimodal large model reasoning process; the first reasoning process text includes: each reasoning step and a first execution order between each reasoning step; according to the first execution order, each reasoning step is respectively: extracting the category name and the attribute of the object described by the description text contained in the input question, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute;

[0180] The predicted answer determination module 705 is used to input the sample image and the sample question in each question-answer sample pair into the multimodal macro model of the initial structure to obtain the predicted answer;

[0181] The model parameter adjustment module 706 is used to adjust the parameters of the initial structure multimodal large model based on the difference between the obtained predicted answer and the sample answer in the corresponding question and answer sample pair until the preset convergence condition is reached to obtain the trained multimodal large model.

[0182] Based on the multimodal large model training device provided in the embodiment of the present application, the second sample description text can be generated using the first large language model according to the acquired sample image and the description text (i.e., the first sample description text) of the object annotation box used to describe the specified category in the sample image, so as to increase the number of the obtained second sample description texts and enrich the diversity of the second sample description texts. Furthermore, a sample question containing each second sample description text can be constructed to instruct the multimodal large model to determine the position of the image area in the input image that matches the second sample description text. And the matching results of each object annotation box in each sample image and each attribute of the multiple second sample description texts used to combine and obtain the sample image can be combined to construct a sample answer in the form of a thinking chain, so as to obtain a question-answer sample pair in the form of a thinking chain. The sample answer in the form of a thinking chain can include: the specific steps of inferring the position of the image area in the sample image that matches each attribute in the second sample description text.

[0183] Accordingly, training a large multimodal model based on question-answer samples in the form of thought chains can enable the large multimodal model to learn to reason in the form of thought chains, determine the location of the image area occupied by objects in the image that match the input description text, and generate answers in the form of thought chains. Subsequently, using the trained large multimodal model, end-to-end descriptive object detection can be achieved without the need to integrate multiple models, which can reduce the complexity of achieving descriptive object detection and improve detection efficiency.

[0184] In one embodiment, the device further comprises: a preset category name acquisition module, which is used to, before extracting the category name of the object described in the first sample description text corresponding to the sample image and multiple attributes of the object using the first large language model and the first text prompt for each sample image, and combining the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image, obtain, for each sample image, the category name of the object in the object annotation box of the specified category in the sample image as the preset category name;

[0185] The description text generation module 702 is specifically used to input the first text prompt, the first sample description text corresponding to the sample image and the preset category name into the first large language model for each sample image, extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object, and combine the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the obtained category name is consistent with the category represented by the preset category name.

[0186] In one embodiment, the sample acquisition module 701 is specifically used to obtain multiple sample images and the position of the object annotation box of the specified category in each sample image; for each sample image, the sample image and the second text prompt are input into the second largest language model to obtain the first sample description text of the object annotation box of the specified category in the sample image; wherein the second text prompt is used to indicate: generate a description text of the object annotation box in the input image.

[0187] In one embodiment, the description text generation module 702 is specifically used to input, for each sample image, the first sample description text and the first text prompt corresponding to the sample image into the first large language model, and extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object; input the extracted category name, the extracted attributes, and the third text prompt into the first large language model to obtain multiple second sample description texts corresponding to the sample image; wherein the third text prompt is used to indicate: combine the input category name with at least one of the input attributes to obtain multiple description texts.

[0188] In one embodiment, the matching result determination module 703 is specifically used to use the image-text matching model and the fourth text prompt to respectively detect whether the object annotation box of each specified category in the sample image matches each description text to be matched; wherein, each description text to be matched is: each attribute of multiple second sample description texts corresponding to the sample image is combined with the extracted category name; the fourth text prompt is used to indicate: determine whether each object annotation box in the input image matches the input description text.

[0189] The present application also provides a target detection device. Figure 8 , Figure 8 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application, the device comprising:

[0190] The data acquisition module 801 is used to acquire an image to be detected and a text prompt to be used that includes a description text to be detected; wherein the text prompt to be used is used to indicate: determining the position of the image area occupied by the object that matches the description text to be detected in the image to be detected;

[0191] The detection result determination module 802 is used to input the image to be detected and the text prompt to be used into a pre-trained multimodal large model to obtain a detection result in the form of a thought chain; wherein the multimodal large model is trained based on any of the multimodal large model training methods described above; the detection result includes: a second reasoning process text for describing the reasoning process of the multimodal large model; the second reasoning process text includes: each reasoning step and a second execution order between each reasoning step; according to the second execution order, each reasoning step is respectively: extracting the category name of the object described by the input text prompt and the attributes of the object, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute.

[0192] Based on the target detection device provided by the embodiment of the present application, the multimodal large model can learn to reason in the form of thought chains during the training process, determine the position of the image area occupied by the object in the image that matches the input description text, and generate answers in the form of thought chains. Using the trained multimodal large model, end-to-end descriptive target detection can be achieved without the need to integrate multiple models, which can reduce the complexity of achieving descriptive target detection and improve detection efficiency.

[0193] The present application also provides an electronic device, such as Fig. 9 As shown, including:

[0194] Memory 901, used for storing computer programs;

[0195] The processor 902 is used to implement any of the above-mentioned multimodal large model training methods or target detection methods when executing the program stored in the memory 901.

[0196] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 902, the communication interface, and the memory 901 communicate with each other via the communication bus.

[0197] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0198] The communication interface is used for communication between the above electronic device and other devices.

[0199] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0200] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0201] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned multimodal large model training methods or target detection methods are implemented.

[0202] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any multimodal large model training method or target detection method in the above-mentioned embodiments.

[0203] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0204] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0205] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the target detection method, device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the multimodal large model training method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0206] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A multimodal large model training method, characterized in that: The method comprises: Acquire a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image; For each sample image, the first large language model and the first text prompt are used to extract the category name of the object described in the first sample description text corresponding to the sample image and multiple attributes of the object, and the obtained category name is combined with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the first text prompt is used to indicate: extract the category name of the object described in the input description text and the attributes of the object; For each object annotation box in the sample image, determining whether the object annotation box matches each attribute of a plurality of second sample description texts corresponding to the sample image that are combined; Construct a sample question containing each second sample description text, and use the matching result of each attribute contained in the second sample description text and each object annotation box to construct a sample answer in the form of a thinking chain corresponding to the sample question, so as to obtain a question-answer sample pair; wherein, the sample question is used to indicate the position of the image area in the input image that matches the second sample description text; the sample answer contains a first reasoning process text for describing the multimodal large model reasoning process; the first reasoning process text includes: each reasoning step and a first execution order between each reasoning step; according to the first execution order, each reasoning step is respectively: extracting the category name and the attribute of the object described by the description text contained in the input question, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area and each extracted attribute, and determining the position of the image area that matches each extracted attribute; Input the sample image and the sample question in each question-answer sample pair into the multimodal large model of the initial structure to obtain the predicted answer; Based on the difference between the predicted answer and the sample answer in the corresponding question and answer sample pair, the parameters of the multimodal large model of the initial structure are adjusted until the preset convergence conditions are reached, thereby obtaining a trained multimodal large model.

2. The method according to claim 1, characterized in that Before extracting, for each sample image, the category name of the object described in the first sample description text corresponding to the sample image and a plurality of attributes of the object using the first large language model and the first text prompt, and combining the obtained category name with at least one of the obtained attributes to obtain a plurality of second sample description texts corresponding to the sample image, the method further includes: For each sample image, obtaining the category name of the object in the object annotation box of the specified category in the sample image as the preset category name; The method of extracting the category name of the object described in the first sample description text corresponding to the sample image and multiple attributes of the object by using the first large language model and the first text prompt for each sample image, and combining the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image includes: For each sample image, the first text prompt, the first sample description text corresponding to the sample image and the preset category name are input into the first large language model to extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object, and the obtained category name is combined with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the obtained category name is consistent with the category represented by the preset category name.

3. The method according to claim 1, characterized in that: The step of obtaining a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image comprises: Obtain multiple sample images and the position of the object annotation box of the specified category in each sample image; For each sample image, the sample image and the second text prompt are input into the second largest language model to obtain a first sample description text of an object annotation box of a specified category in the sample image; wherein the second text prompt is used to indicate: generate a description text of the object annotation box in the input image.

4. The method according to claim 1, characterized in that: The method of extracting the category name of the object described in the first sample description text corresponding to the sample image and multiple attributes of the object by using the first large language model and the first text prompt for each sample image, and combining the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image includes: For each sample image, input the first sample description text and the first text prompt corresponding to the sample image into the first large language model, and extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object; The extracted category name, the extracted attributes, and the third text prompt are input into the first language model to obtain multiple second sample description texts corresponding to the sample image; wherein the third text prompt is used to indicate: combine the input category name with at least one of the input attributes to obtain multiple description texts.

5. The method according to claim 1, characterized in that The step of determining, for each object annotation frame in the sample image, whether the object annotation frame matches each attribute of a plurality of second sample description texts used for combining to obtain the corresponding sample image, includes: Using the image-text matching model and the fourth text prompt, it is respectively detected whether the object annotation box of each specified category in the sample image matches each description text to be matched; wherein, each description text to be matched is: each attribute of multiple second sample description texts corresponding to the sample image is combined with the extracted category name; the fourth text prompt is used to indicate: determine whether each object annotation box in the input image matches the input description text.

6. A target detection method, characterized in that: The method comprises: Acquire an image to be detected and a text prompt to be used containing a description text to be detected; wherein the text prompt to be used is used to indicate: determine the position of the image area occupied by the object that meets the description text to be detected in the image to be detected; The image to be detected and the multimodal large model to be pre-trained using the text prompt are input to obtain a detection result in the form of a thought chain; wherein the multimodal large model is trained based on the method described in any one of claims 1-5; the detection result includes: a second reasoning process text for describing the reasoning process of the multimodal large model; the second reasoning process text includes: each reasoning step and a second execution order between each reasoning step; according to the second execution order, each reasoning step is respectively: extracting the category name of the object described by the input text prompt and the attributes of the object, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute.

7. A multimodal large model training device, characterized in that: The device comprises: A sample acquisition module, used to acquire a plurality of sample images and a first sample description text of an object annotation box of a specified category in each sample image; The description text generation module is used to extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object by using the first large language model and the first text prompt for each sample image, and combine the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the first text prompt is used to indicate: extract the category name of the object described by the input description text and the attributes of the object; a matching result determination module, for determining, for each object annotation box in the sample image, whether the object annotation box matches each attribute of a plurality of second sample description texts corresponding to the sample image; A question-answer sample pair construction module is used to construct a sample question containing each second sample description text, and to use the matching result between each attribute contained in the second sample description text and each object annotation box to construct a sample answer in the form of a thinking chain corresponding to the sample question, so as to obtain a question-answer sample pair; wherein, the sample question is used to indicate the position of the image area in the input image that matches the second sample description text; the sample answer contains a first reasoning process text for describing the multimodal large model reasoning process; the first reasoning process text includes: each reasoning step and a first execution order between each reasoning step; according to the first execution order, each reasoning step is respectively: extracting the category name and the attribute of the object described by the description text contained in the input question, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute; A predicted answer determination module is used to input the sample image and the sample question in each question-answer sample pair into the multimodal macro model of the initial structure to obtain a predicted answer; The model parameter adjustment module is used to adjust the parameters of the initial structure multimodal large model based on the difference between the obtained predicted answer and the sample answer in the corresponding question and answer sample pair until the preset convergence condition is reached to obtain the trained multimodal large model.

8. The device according to claim 7, characterized in that The device also includes: a preset category name acquisition module, for extracting the category name of the object described in the first sample description text corresponding to the sample image and a plurality of attributes of the object by using the first large language model and the first text prompt for each sample image, and combining the obtained category name with at least one of the obtained attributes to obtain a plurality of second sample description texts corresponding to the sample image, and obtaining, for each sample image, the category name of the object in the object annotation box of the specified category in the sample image as the preset category name; The description text generation module is specifically used to input the first text prompt, the first sample description text corresponding to the sample image and the preset category name into the first large language model for each sample image, extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object, and combine the obtained category name with at least one of the obtained attributes to obtain multiple second sample description texts corresponding to the sample image; wherein the obtained category name is consistent with the category represented by the preset category name; and / or, The sample acquisition module is specifically used to obtain multiple sample images and the position of the object annotation box of the specified category in each sample image; for each sample image, the sample image and the second text prompt are input into the second largest language model to obtain the first sample description text of the object annotation box of the specified category in the sample image; wherein the second text prompt is used to indicate: generate the description text of the object annotation box in the input image; and / or, The description text generation module is specifically used to input, for each sample image, a first sample description text and a first text prompt corresponding to the sample image into a first large language model, extract the category name of the object described by the first sample description text corresponding to the sample image and multiple attributes of the object; input the extracted category name, the extracted attributes, and a third text prompt into the first large language model to obtain multiple second sample description texts corresponding to the sample image; wherein the third text prompt is used to indicate: combining the input category name with at least one of the input attributes to obtain multiple description texts; and / or, The matching result determination module is specifically used to use the image-text matching model and the fourth text prompt to respectively detect whether the object annotation box of each specified category in the sample image matches each description text to be matched; wherein the description text to be matched is: each attribute of the multiple second sample description texts corresponding to the sample image is combined with the extracted category name; the fourth text prompt is used to indicate: determine whether each object annotation box in the input image matches the input description text.

9. A target detection device, characterized in that: The device comprises: A data acquisition module is used to acquire an image to be detected and a text prompt to be used containing a description text to be detected; wherein the text prompt to be used is used to indicate: determining the position of an image area occupied by an object in the image to be detected that matches the description text to be detected; A detection result determination module is used to input the image to be detected and the text prompt to be used into a pre-trained multimodal large model to obtain a detection result in the form of a thought chain; wherein the multimodal large model is trained based on the method described in any one of claims 1-5; the detection result includes: a second reasoning process text for describing the reasoning process of the multimodal large model; the second reasoning process text includes: each reasoning step and a second execution order between each reasoning step; according to the second execution order, each reasoning step is respectively: extracting the category name of the object described by the input text prompt and the attributes of the object, detecting the position of the image area occupied by all objects belonging to the category represented by the extracted category name in the input image, respectively determining the matching result of each image area with each extracted attribute, and determining the position of the image area that matches each extracted attribute.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method described in any one of claims 1 to 5 or the method described in claim 6 when executing a program stored in a memory.

Images