Object detection and multi-modal large model training method and device

By combining a multimodal large model with a visual encoder, word segmenter, large language model and visual detector, the problems of low accuracy and false output in traditional object detection methods are solved, and efficient and accurate object detection and coordinate recognition are achieved.

CN119006885APending Publication Date: 2024-11-22BEIJING BAIDU NETCOM SCI & TECH CO LTD

Patent Information

Application Number
CN202410961459.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Traditional object detection methods have low accuracy in detecting objects in images and cannot effectively reject outputs of non-existent objects, leading to the illusion problem.

Method used

A multimodal large model is used for object detection. By combining a visual encoder, a word segmenter, a large language model and a visual detector, the output results include text information and object detection results. The model is optimized by training samples to improve accuracy.

Benefits of technology

It improves the accuracy and efficiency of object detection, accurately identifies objects in images and provides coordinate information, and avoids false output.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006885A_ABST
    Figure CN119006885A_ABST
Patent Text Reader

Abstract

The invention provides an object detection and multi-modal large model training method and device, and relates to the artificial intelligence fields of computer vision, deep learning, large models and the like. The object detection method comprises the following steps: acquiring a to-be-processed target image and a first problem corresponding to the target image; taking the target image and the first problem as input of a multi-modal large model to obtain an output result of the multi-modal large model; the output result comprises a first output result composed of a first answer and a second answer, the first answer comprises first text information used for indicating that the target image comprises a first target object and object detection result information for the target image, and the first target object is an object involved in the first question; the second answer comprises coordinate information, determined according to the object detection result information, of the first target object in the target image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to object detection and multimodal large model training methods and devices in the fields of computer vision, deep learning, and large models. Background Art

[0002] In practical applications, users have object detection requirements in many scenarios. For example, a user gives an image and asks a corresponding question, "Is there a ship in the image?" Accordingly, object detection will be performed in the image based on the user's question to determine whether there is a ship. However, the accuracy of traditional implementation methods is usually low. Summary of the invention

[0003] The present disclosure provides a method and device for object detection and multimodal large model training.

[0004] An object detection method, comprising:

[0005] Acquire a target image to be processed and a first question corresponding to the target image;

[0006] The target image and the first question are used as inputs of a multimodal large model to obtain an output result of the multimodal large model; the output result includes: a first output result consisting of a first answer and a second answer, the first answer includes: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is an object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined according to the object detection result information.

[0007] A multimodal large model training method, comprising:

[0008] Acquire a training sample, wherein the training sample includes: a sample image, a second question corresponding to the sample image, and a label corresponding to the sample image, wherein the label includes: third text information for indicating that the sample image includes a second target object, object detection result information for the sample image, and coordinate information of the second target object in the sample image, and the second target object is an object involved in the second question;

[0009] The multimodal large model is trained using the training samples, and the multimodal large model is used to generate an output result based on a target image and a first question corresponding to the target image, the output result including: a first output result consisting of a first answer and a second answer, the first answer including: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is an object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

[0010] An object detection device comprises: an information acquisition module and an object detection module;

[0011] The information acquisition module is used to acquire a target image to be processed and a first question corresponding to the target image;

[0012] The object detection module is used to use the target image and the first question as inputs of the multimodal large model to obtain the output result of the multimodal large model; the output result includes: a first output result consisting of a first answer and a second answer, the first answer includes: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is the object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined according to the object detection result information.

[0013] A multimodal large model training device, comprising: a sample acquisition module and a model training module;

[0014] The sample acquisition module is used to acquire a training sample, wherein the training sample includes: a sample image, a second question corresponding to the sample image, and a label corresponding to the sample image, wherein the label includes: third text information for indicating that the sample image includes a second target object, object detection result information for the sample image, and coordinate information of the second target object in the sample image, and the second target object is an object involved in the second question;

[0015] The model training module is used to train the multimodal large model using the training samples, and the multimodal large model is used to generate an output result based on a target image and a first question corresponding to the target image, and the output result includes: a first output result consisting of a first answer and a second answer, the first answer includes: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is an object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

[0016] An electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0020] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0021] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the method described above when executed by a processor.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a flow chart of an embodiment of the object detection method disclosed in the present invention;

[0025] Figure 2 A schematic diagram of a third answer given for a target image according to the present disclosure;

[0026] Figure 3 Schematic diagram of the overall implementation process of the object detection method disclosed in the present invention;

[0027] Figure 4 This is a flowchart of an embodiment of the multimodal large model training method disclosed in the present invention;

[0028] Figure 5 Schematic diagram of the structure of an object detection device embodiment 500 of the present disclosure;

[0029] Figure 6 Schematic diagram of the composition structure of the multimodal large model training device embodiment 600 of the present disclosure;

[0030] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0033] Figure 1 FIG. 1 is a flow chart of an embodiment of the object detection method disclosed in the present invention. Figure 1 As shown, the following specific implementation methods are included.

[0034] In step 101, a target image to be processed and a first question corresponding to the target image are obtained.

[0035] In step 102, the target image and the first question are used as inputs of the multimodal large model to obtain an output result of the multimodal large model.

[0036] Among them, the output result may include: a first output result consisting of a first answer and a second answer, the first answer including: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is the object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

[0037] By adopting the scheme described in the above method embodiment, the powerful reasoning ability of the multimodal large language model can be used to perform object detection based on the target image and the first question corresponding to the target image, and generate a corresponding answer, thereby improving the accuracy of the answer result, that is, improving the accuracy of the detection result. Moreover, the answer may include not only text information, but also object detection result information and coordinate information of the target object in the target image determined according to the object detection result information, thereby enriching the output content and improving the efficiency and comprehensiveness of user information acquisition.

[0038] The target image can be any image input by the user, and when the target image is input, the corresponding first question can also be input. For example, the first question can be "Is there a giraffe in the picture?".

[0039] The target image and the first question can be used as inputs of the multimodal large model to obtain an output result of the multimodal large model, and the output result may include: a first output result consisting of a first answer and a second answer, wherein the first answer may include first text information for explaining that the target image includes a first target object, such as two giraffes on the grass in the figure, and the first answer may also include object detection result information. In addition, the second answer may include coordinate information of the first target object in the target image determined according to the object detection result information, such as coordinate information of two giraffes in the target image, and the coordinate information may include the upper left vertex coordinate information and the lower right vertex coordinate information of the corresponding detection box (rectangular box), or include the upper left vertex coordinate information of the corresponding detection box and the width and height information of the detection box, etc.

[0040] In some embodiments of the present disclosure, the output result may also include: a second output result composed of a third answer, the third answer may include: second text information for explaining that the first target object is not included in the target image, wherein the multimodal large language outputs the first output result or the second output result.

[0041] Figure 2 FIG. 4 is a schematic diagram of a third answer given to a target image according to the present disclosure. Figure 2 As shown in , assuming that the first question is still "Is there a giraffe in the picture?", then for Figure 2 For the target image shown, the following third answer can be given: There is no giraffe in the picture.

[0042] In the traditional method, even if the first target object does not exist in the target image, the output will not be rejected, which may cause the output hallucination problem. After adopting the above processing method, a negative answer can be directly output, thereby further improving the accuracy of the answer result.

[0043] In some embodiments of the present disclosure, the multimodal large model may include: a visual encoder, a tokenizer, a large language model and a visual detector, wherein the visual encoder is used to generate visual tokens corresponding to the target image, the tokenizer is used to generate text tokens corresponding to the first question, the large language model is used to generate a first answer or a third answer based on the visual tokens and the text tokens, and the visual detector is used to generate a second answer based on the object detection result information.

[0044] That is to say, the target image can be used as the input of the visual encoder to obtain the output visual word unit. The visual encoder can be a text-image pre-training (CLIP) model using contrastive learning or a derivative model of CLIP, such as Vision Transformer-Large (ViT-Large) or Vision Transformer-Huge (ViT-Huge).

[0045] In addition, a word segmenter can be used to convert the first question into text words. This process can be understood as a preprocessing process of the text by a large language model.

[0046] Afterwards, the visual word unit and the text word unit can be concatenated and sent to the large language model together. Accordingly, the large language model can decode the string in a generative manner and output the generated first answer or third answer. The first answer may include: first text information for indicating that the target image includes the first target object and object detection result information. The large language model is a deep learning model trained based on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text, handle various natural language tasks, etc.

[0047] Further, the visual detector can determine the coordinate information of the first target object according to the object detection result information. The visual detector can select a model based on a detection transformer (DETR) structure, such as a DETR model or a DETR with Improved deNoising anchOrboxes (DINO) model.

[0048] Accordingly, with the help of the multimodal large language model with the above structure, the required output results can be obtained efficiently and accurately.

[0049] In addition, in some embodiments of the present disclosure, the object detection result information may include: a detection tag word and M detection content words, where M is a positive integer greater than 1, and each detection content word may include: the detection frame coordinate information and the detection frame score information of the corresponding detection object, and the detection tag word may include a start tag.<det_start> and the end tag<det_end> , the M detection content words may be located between the start tag and the end tag, and the start tag and the end tag are used to mark the start and end of the M detection content words. Accordingly, the visual detector may be used to screen out the detection objects whose corresponding detection frame scores are greater than a predetermined threshold from the M detection objects as the first target object, and the detection frame coordinate information corresponding to the screened detection objects may be used as the coordinate information of the first target object. In addition, the second answer output by the visual detector may include not only the coordinate information of the first target object, but also the score information corresponding to the coordinate information, that is, the corresponding detection frame score information.

[0050] M can be a pre-set hyperparameter, and its specific value can be determined according to actual needs, such as 5.

[0051] It can be seen that two types of detection words are designed in the solution described in the present disclosure, namely, detection tag words and detection content words. For example, the detection content word can be <d1> 、 <d2> 、 <d3> 、 <d4>and <d5>By setting multiple detection content words, the detection output of multiple target objects can be realized, thereby improving the dense multi-target detection capability, etc.

[0052] If the object involved in the first question, i.e., the first target object, exists in the target image, then the answer (first answer) of the large language model may include the first text information used to explain that the target image includes the first target object, the detection mark word, and M detection content words. The M detection content words will be inserted between the start mark and the end mark in the detection mark word, and each detection content word includes the detection frame coordinate information and the detection frame score information of the corresponding detection object. In other words, the detection content words corresponding to the M detection objects can be obtained, and the M detection objects include the first target object. Accordingly, the visual detector can determine the first target object and the corresponding coordinate information from the M detection objects.

[0053] There is no limitation on the specific forms of the detection mark word-unit and the detection content word-unit, for example, they may be in a predetermined character string form.

[0054] In some embodiments of the present disclosure, a detection frame corresponding to the first target object may be added to the target image according to the second answer to obtain a result image and display it.

[0055] That is, the first target object in the target image can be marked in the form of a detection frame, so that the user can view the detected first target object more intuitively and conveniently.

[0056] Combined with the above introduction, Figure 3 FIG. 1 is a schematic diagram of the overall implementation process of the object detection method disclosed in the present invention. Figure 3 As shown, the user inputs a target image and the corresponding first question "Are there giraffes in the picture?" Accordingly, the visual decoder can be used to generate visual word-grams corresponding to the target image, and the word segmenter (not shown to simplify the figure) can be used to generate text word-grams corresponding to the first question. After that, the visual word-gram and the text word-gram can be used as the input of the large language model together, so as to obtain the first answer output by the large language model, that is, there are 2 giraffes in the picture<det_start> <d1> <d2> d3> <d4> <d5><det_end>On the grass, the visual detector can be used based on <d1> <d2> d3> <d4> <d5>The coordinate information of the first target object is determined, that is, a second answer is generated (not shown to simplify the drawing). In addition, a result image can be generated according to the second answer, and the result image can be displayed to the user.

[0057] In addition, if Figure 3 As shown,<det_start> <d1> <d2> d3> <d4> <d5><det_end>Usually next to the first target object in the first answer, which is the giraffe.

[0058] The above-mentioned multimodal large model can be obtained by pre-training. The training method of the multimodal large model is described below.

[0059] Figure 4 Flow chart of an embodiment of the multimodal large model training method described in the present disclosure. Figure 4 As shown, the following specific implementation methods are included.

[0060] In step 401, a training sample is obtained, wherein the training sample includes: a sample image, a second question corresponding to the sample image, and a label corresponding to the sample image.

[0061] In step 402, the multimodal large model is trained using training samples, and the multimodal large model is used to generate an output result according to the target image and the first question corresponding to the target image.

[0062] The label includes: third text information for indicating that the sample image includes a second target object, object detection result information for the sample image, and coordinate information of the second target object in the sample image, and the second target object is the object involved in the second question.

[0063] The output result includes: a first output result composed of a first answer and a second answer, the first answer includes: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is the object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

[0064] By adopting the scheme described in the above method embodiment, a multimodal large model can be trained, and then with the help of the powerful reasoning ability of the multimodal large language model, object detection can be performed based on the target image and the first question corresponding to the target image, and a corresponding answer can be generated, thereby improving the accuracy of the answer result, that is, improving the accuracy of the detection result. Moreover, the answer may include not only text information, but also object detection result information and coordinate information of the target object in the target image determined according to the object detection result information, thereby enriching the output content and improving the efficiency and comprehensiveness of user information acquisition.

[0065] In some embodiments of the present disclosure, the output result may further include: a second output result composed of a third answer, and the third answer may include: second text information for explaining that the first target object is not included in the target image.

[0066] In the traditional method, even if the first target object involved does not exist in the target image, the output will not be rejected, which may cause the output hallucination problem. After adopting the above processing method, a negative answer can be directly output, thereby further improving the accuracy of the answer result.

[0067] In some embodiments of the present disclosure, the multimodal large model may include: a visual encoder, a word segmenter, a large language model and a visual detector, wherein the visual encoder is used to generate visual word elements corresponding to the target image, the word segmenter is used to generate text word elements corresponding to the first question, the large language model is used to generate a first answer or a third answer based on the visual word elements and the text word elements, and the visual detector is used to generate a second answer based on the object detection result information.

[0068] Accordingly, with the help of the multimodal large language model with the above structure, the required output results can be obtained efficiently and accurately.

[0069] In addition, in some embodiments of the present disclosure, the object detection result information may include: a detection mark word and M detection content words, M is a positive integer greater than 1, and each detection content word includes: detection frame coordinate information and detection frame score information of the corresponding detection object, the detection mark word includes a start mark and an end mark, the M detection content words are located between the start mark and the end mark, and the start mark and the end mark are used to mark the start and end of the M detection content words. Accordingly, the visual detector can be used to filter out the detection objects whose corresponding detection frame scores are greater than a predetermined threshold from the M detection objects as the first target object, and the detection frame coordinate information corresponding to the filtered detection objects can be used as the coordinate information of the first target object.

[0070] It can be seen that two types of detection words are designed in the scheme described in the present disclosure, namely detection tag words and detection content words. By setting multiple detection content words, the detection output of multiple target objects can be achieved, thereby improving the dense multi-target detection capability, etc.

[0071] There is no restriction on how to obtain training samples. For example, a sufficient number of training samples can be constructed by combining manual labeling with automatic labeling.

[0072] Afterwards, the constructed training samples can be used to perform end-to-end training on the multimodal large model. In some embodiments of the present disclosure, when the multimodal large model is trained using training samples, for any training sample, the sample image and the second question therein can be used as inputs of the multimodal large model, respectively, and the language model loss (loss) and the box prediction loss can be determined based on the output results of the multimodal large model and the labels in the training sample, and then the comprehensive loss can be determined by combining the language model loss and the box prediction loss, and the parameters of the multimodal large model can be optimized based on the comprehensive loss.

[0073] That is, the language model loss and the box prediction loss can be combined to optimize the parameters of the multimodal large model, thereby improving the optimization efficiency and optimization effect.

[0074] Among them, after obtaining the language model loss and the box prediction loss respectively, the two can be added together, and the sum is used as the required comprehensive loss.

[0075] In some embodiments of the present disclosure, a method for determining a language model loss may include: determining a text loss based on first text information output by a multimodal large model and third text information in a label, and determining a detection tag loss based on detection tag words in an object detection result output by the multimodal large model and detection tag words in object detection result information in the label, and then determining a language model loss in combination with the text loss and the detection tag loss.

[0076] That is, the third text information in the label can be taken as the true value, and the text loss can be determined by comparing the first text information output by the multimodal large model with the third text information as the true value. In addition, the detection tag word in the object detection result information in the label can be taken as the true value, and the detection tag loss can be determined by comparing the detection tag word in the object detection result output by the multimodal large model with the detection tag word as the true value. Then, the text loss and the detection tag loss can be added, and the sum can be used as the language model loss.

[0077] By adopting the above processing method, the required language model loss can be determined by combining the text loss and the detection mark loss, thereby improving the accuracy of the obtained language model loss.

[0078] In addition, in some embodiments of the present disclosure, the method of determining the out-of-frame prediction loss may include: for the detection objects corresponding to the M detection content words in the object detection result information output by the multimodal large model, select the detection object that matches the second target object corresponding to the training sample, determine the coordinate loss based on the detection frame coordinate information of the selected detection object and the coordinate information of the second target object in the label, and determine the out-of-frame prediction loss based on the coordinate loss.

[0079] Assuming that the value of M is 5, there are 5 detection objects in total. The Hungarian matching algorithm can be used to select the detection object that matches the second target object from the 5 detection objects, and then the coordinate loss can be determined based on the detection frame coordinate information of the selected detection object and the coordinate information of the second target object in the label.

[0080] In some embodiments of the present disclosure, the following three coordinate losses can be determined: mean absolute error (L1) loss of coordinates, generalized intersection over union (GIOU) loss of coordinates, and objectiveness classification loss of coordinates, and the box prediction loss can be determined by combining these three coordinate losses.

[0081] For example, these three coordinate losses can be added together and the sum can be used as the required box prediction loss.

[0082] It should be noted that, assuming that the number of second target objects is one, then a detection object that matches (is the most matched) the second target object can be selected from each detection object, and the coordinate loss can be determined based on the detection frame coordinate information of the selected detection object and the coordinate information of the second target object in the label. If the number of second target objects is multiple, such as two, assuming that they are target object a and target object b, then a detection object that matches target object a (for ease of expression, referred to as detection object a) and a detection object that matches target object b (for ease of expression, referred to as detection object b) can be selected respectively, and then the coordinate loss can be determined based on the detection object. The coordinate loss corresponding to target object a can be determined based on the detection frame coordinate information of object a and the coordinate information of target object a in the label, and the coordinate loss corresponding to target object b can be determined based on the detection frame coordinate information of object b and the coordinate information of target object b in the label. In this case, before adding the above three coordinate losses, the L1 loss corresponding to target object a and target object b can be added, the GIOU loss corresponding to target object a and target object b can be added, and the objectiveness classification loss corresponding to target object a and target object b can be added, and then the three addition results can be added to obtain the required frame prediction loss.

[0083] Through the above processing, the coordinate loss is evaluated from different dimensions, thereby improving the accuracy of the determined box prediction loss, etc. Among them, there is no restriction on the acquisition methods of the three coordinate losses, such as the existing acquisition methods can be used.

[0084] Through the above method, the parameters of the multimodal large model can be continuously optimized until the convergence conditions are met, and the trained multimodal large model can be applied to actual object detection, such as Figure 1 In the illustrated method embodiment, an output result is generated based on an input target image and a first question corresponding to the target image.

[0085] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure. In addition, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0086] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0087] Figure 5 FIG. 5 is a schematic diagram of the structure of an object detection device embodiment 500 of the present disclosure. Figure 5 As shown, it includes: an information acquisition module 501 and an object detection module 502.

[0088] The information acquisition module 501 is used to acquire a target image to be processed and a first question corresponding to the target image.

[0089] The object detection module 502 is used to use the target image and the first question as inputs of the multimodal large model to obtain the output results of the multimodal large model; the output results include: a first output result consisting of a first answer and a second answer, the first answer includes: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is the object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined according to the object detection result information.

[0090] By adopting the scheme described in the above-mentioned device embodiment, the powerful reasoning ability of the multimodal large language model can be used to perform object detection based on the target image and the first question corresponding to the target image, and generate a corresponding answer, thereby improving the accuracy of the answer result, that is, improving the accuracy of the detection result. Moreover, the answer may include not only text information, but also object detection result information and coordinate information of the target object in the target image determined according to the object detection result information, thereby enriching the output content and improving the efficiency and comprehensiveness of user information acquisition.

[0091] In some embodiments of the present disclosure, the output result may also include: a second output result composed of a third answer, the third answer may include: second text information for explaining that the first target object is not included in the target image, wherein the multimodal large language outputs the first output result or the second output result.

[0092] In some embodiments of the present disclosure, the multimodal large model may include: a visual encoder, a word segmenter, a large language model and a visual detector, wherein the visual encoder is used to generate visual word elements corresponding to the target image, the word segmenter is used to generate text word elements corresponding to the first question, the large language model is used to generate a first answer or a third answer based on the visual word elements and the text word elements, and the visual detector is used to generate a second answer based on the object detection result information.

[0093] In addition, in some embodiments of the present disclosure, the object detection result information may include: a detection mark word and M detection content words, M is a positive integer greater than 1, and each detection content word may respectively include: detection frame coordinate information and detection frame score information of the corresponding detection object, the detection mark word may include a start mark and an end mark, the M detection content words may be located between the start mark and the end mark, and the start mark and the end mark are used to mark the beginning and the end of the M detection content words. Accordingly, the visual detector can be used to filter out the detection objects whose corresponding detection frame scores are greater than a predetermined threshold from the M detection objects as the first target object, and the detection frame coordinate information corresponding to the filtered detection objects can be used as the coordinate information of the first target object.

[0094] In addition, in some embodiments of the present disclosure, the object detection module 502 may also add a detection frame corresponding to the first target object to the target image according to the second answer, obtain a result image, and display the result image.

[0095] Figure 6 FIG. 6 is a schematic diagram of the structure of the multimodal large model training device embodiment 600 of the present disclosure. Figure 6 As shown, it includes: a sample acquisition module 601 and a model training module 602.

[0096] The sample acquisition module 601 is used to acquire training samples, wherein the training samples include: a sample image, a second question corresponding to the sample image, and a label corresponding to the sample image, the label includes: third text information used to indicate that the sample image includes a second target object, object detection result information for the sample image, and coordinate information of the second target object in the sample image, and the second target object is the object involved in the second question.

[0097] Model training module 602 is used to train the multimodal large model using training samples. The multimodal large model is used to generate an output result based on the target image and the first question corresponding to the target image. The output result includes: a first output result consisting of a first answer and a second answer. The first answer includes: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image. The first target object is the object involved in the first question. The second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

[0098] By adopting the scheme described in the above-mentioned device embodiment, a multimodal large model can be trained, and then with the help of the powerful reasoning ability of the multimodal large language model, object detection can be performed based on the target image and the first question corresponding to the target image, and a corresponding answer can be generated, thereby improving the accuracy of the answer result, that is, improving the accuracy of the detection result. Moreover, the answer can include not only text information, but also object detection result information and coordinate information of the target object in the target image determined according to the object detection result information, thereby enriching the output content and improving the efficiency and comprehensiveness of user information acquisition.

[0099] In some embodiments of the present disclosure, the output result may further include: a second output result composed of a third answer, and the third answer may include: second text information for explaining that the first target object is not included in the target image.

[0100] In some embodiments of the present disclosure, the multimodal large model may include: a visual encoder, a word segmenter, a large language model and a visual detector, wherein the visual encoder can be used to generate visual word elements corresponding to the target image, the word segmenter can be used to generate text word elements corresponding to the first question, the large language model can be used to generate a first answer or a third answer based on the visual word elements and the text word elements, and the visual detector can be used to generate a second answer based on the object detection result information.

[0101] In addition, in some embodiments of the present disclosure, the object detection result information may include: a detection mark word and M detection content words, M is a positive integer greater than 1, and each detection content word includes: detection frame coordinate information and detection frame score information of the corresponding detection object, the detection mark word includes a start mark and an end mark, the M detection content words are located between the start mark and the end mark, and the start mark and the end mark are used to mark the start and end of the M detection content words. Accordingly, the visual detector can be used to filter out the detection objects whose corresponding detection frame scores are greater than a predetermined threshold from the M detection objects as the first target object, and the detection frame coordinate information corresponding to the filtered detection objects can be used as the coordinate information of the first target object.

[0102] There is no restriction on how the sample acquisition module 601 acquires the training samples. The model training module 602 can use the training samples to perform end-to-end training on the multimodal large model. In some embodiments of the present disclosure, when the model training module 602 uses the training samples to train the multimodal large model, for any training sample, the sample image and the second question therein can be used as the input of the multimodal large model, respectively, and the language model loss and the box prediction loss can be determined based on the output results of the multimodal large model and the labels in the training sample, and then the comprehensive loss can be determined by combining the language model loss and the box prediction loss, and the parameters of the multimodal large model can be optimized based on the comprehensive loss.

[0103] Among them, after obtaining the language model loss and the box prediction loss respectively, the two can be added together, and the sum is used as the required comprehensive loss.

[0104] In some embodiments of the present disclosure, the model training module 602 may determine the language model loss in a manner that may include: determining the text loss based on the first text information output by the multimodal large model and the third text information in the label, and determining the detection tag loss based on the detection tag words in the object detection result output by the multimodal large model and the detection tag words in the object detection result information in the label, and then determining the language model loss in combination with the text loss and the detection tag loss.

[0105] That is, the third text information in the label can be taken as the true value, and the text loss can be determined by comparing the first text information output by the multimodal large model with the third text information as the true value. In addition, the detection tag word in the object detection result information in the label can be taken as the true value, and the detection tag loss can be determined by comparing the detection tag word in the object detection result output by the multimodal large model with the detection tag word as the true value. Then, the text loss and the detection tag loss can be added, and the sum can be used as the language model loss.

[0106] In addition, in some embodiments of the present disclosure, the way in which the model training module 602 determines the out-of-frame prediction loss may include: for the detection objects corresponding to the M detection content words in the object detection result information output by the multimodal large model, selecting the detection object that matches the second target object corresponding to the training sample, determining the coordinate loss based on the detection frame coordinate information of the selected detection object and the coordinate information of the second target object in the label, and determining the out-of-frame prediction loss based on the coordinate loss.

[0107] Assuming that the value of M is 5, there are 5 detection objects in total. The Hungarian matching algorithm can be used to select the detection object that matches the second target object from the 5 detection objects, and then the coordinate loss can be determined based on the detection frame coordinate information of the selected detection object and the coordinate information of the second target object in the label.

[0108] In some embodiments of the present disclosure, the following three coordinate losses may be determined: L1 loss of coordinates, GIOU loss of coordinates, and objectiveness classification loss of coordinates, and the box prediction loss may be determined by combining the three coordinate losses. For example, the three coordinate losses may be added together, and the sum of the additions may be used as the required box prediction loss.

[0109] Figure 5 and Figure 6 The specific working process of the illustrated device embodiment can refer to the relevant description in the aforementioned method embodiment and will not be described in detail.

[0110] The scheme disclosed in the present invention can be applied to the field of artificial intelligence, especially computer vision, deep learning, and large models. Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0111] In addition, the images and questions in the embodiments described in this disclosure are not targeted at a specific user and do not reflect the personal information of a specific user. In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0112] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0113] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0114] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0115] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the methods described in the present disclosure may be executed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).

[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general programmable processor, which can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure. < / d4> < / d2> < / d1> < / d4> < / d2> < / d1> < / d4> < / d2> < / d1> < / d3> < / d2> < / d1>

Claims

1. An object detection method, comprising: Acquire a target image to be processed and a first question corresponding to the target image; Using the target image and the first question as inputs of a multimodal large model, and obtaining an output result of the multimodal large model; The output result includes: a first output result consisting of a first answer and a second answer, the first answer including: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is the object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

2. The method according to claim 1, wherein: The output result also includes: a second output result consisting of a third answer; The third answer includes: second text information for explaining that the first target object is not included in the target image.

3. The method according to claim 2, wherein: The multimodal large model includes: a visual encoder, a word segmenter, a large language model and a visual detector; The visual encoder is used to generate visual word units corresponding to the target image; The word segmenter is used to generate text words corresponding to the first question; The large language model is used to generate the first answer or the third answer according to the visual word-gram and the text word-gram; The visual detector is used to generate the second answer according to the object detection result information.

4. The method according to claim 3, wherein: The object detection result information includes: a detection mark word and M detection content words, where M is a positive integer greater than 1, and each detection content word includes: detection frame coordinate information and detection frame score information of the corresponding detection object, the detection mark word includes a start mark and an end mark, and the M detection content words are located between the start mark and the end mark, and the start mark and the end mark are used to mark the start and end of the M detection content words; The visual detector is used to filter out the detection objects whose corresponding detection frame scores are greater than a predetermined threshold from M detection objects as the first target objects, and use the detection frame coordinate information corresponding to the filtered detection objects as the coordinate information of the first target object.

5. The method according to any one of claims 1 to 4, further comprising: A detection frame corresponding to the first target object is added to the target image according to the second answer to obtain a result image, which is then displayed.

6. A multimodal large model training method, comprising: Acquire a training sample, wherein the training sample includes: a sample image, a second question corresponding to the sample image, and a label corresponding to the sample image, wherein the label includes: third text information for indicating that the sample image includes a second target object, object detection result information for the sample image, and coordinate information of the second target object in the sample image, and the second target object is an object involved in the second question; The multimodal large model is trained using the training samples, and the multimodal large model is used to generate an output result based on a target image and a first question corresponding to the target image, the output result including: a first output result consisting of a first answer and a second answer, the first answer including: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is an object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

7. The method according to claim 6, wherein: The output result also includes: a second output result consisting of a third answer; The third answer includes: second text information for explaining that the first target object is not included in the target image.

8. The method according to claim 7, wherein: The multimodal large model includes: a visual encoder, a word segmenter, a large language model and a visual detector; The visual encoder is used to generate visual word units corresponding to the target image; The word segmenter is used to generate text words corresponding to the first question; The large language model is used to generate the first answer or the third answer according to the visual word unit and the text word unit; The visual detector is used to generate the second answer according to the object detection result information.

9. The method according to claim 8, wherein: The object detection result information includes: a detection mark word and M detection content words, where M is a positive integer greater than 1, and each detection content word includes: detection frame coordinate information and detection frame score information of the corresponding detection object, the detection mark word includes a start mark and an end mark, and the M detection content words are located between the start mark and the end mark, and the start mark and the end mark are used to mark the start and end of the M detection content words; The visual detector is used to filter out the detection objects whose corresponding detection frame scores are greater than a predetermined threshold from M detection objects as the first target objects, and use the detection frame coordinate information corresponding to the filtered detection objects as the coordinate information of the first target object.

10. The method according to claim 9, wherein: The step of training the multimodal large model using the training samples includes: For any training sample, the sample image and the second question are used as inputs of the multimodal large model, and the language model loss and the box prediction loss are determined according to the output results of the multimodal large model and the labels in the training sample. The comprehensive loss is determined by combining the language model loss and the box prediction loss, and the parameters of the multimodal large model are optimized according to the comprehensive loss.

11. The method according to claim 10, wherein: Determining the language model loss includes: Determine text loss based on the first text information output by the multimodal large model and the third text information in the label; Determining a detection tag loss according to a detection tag word in the object detection result output by the multimodal large model and a detection tag word in the object detection result information in the label; The language model loss is determined by combining the text loss and the detection token loss.

12. The method according to claim 10, wherein: Determining the box prediction loss includes: For the detection objects respectively corresponding to the M detection content words in the object detection result information output by the multimodal large model, select the detection object that matches the second target object corresponding to the training sample; The coordinate loss is determined based on the detection frame coordinate information of the selected detection object and the coordinate information of the second target object in the label, and the frame prediction loss is determined based on the coordinate loss.

13. The method according to claim 12, wherein: The determining of the coordinate loss and determining the frame prediction loss according to the coordinate loss includes: The following three coordinate losses are determined: mean absolute error loss of coordinates, generalized intersection-over-union loss of coordinates, and object classification loss of coordinates. The box prediction loss is determined by combining the three coordinate losses.

14. An object detection device, comprising: Information acquisition module and object detection module; The information acquisition module is used to acquire a target image to be processed and a first question corresponding to the target image; The object detection module is used to take the target image and the first question as inputs of a multimodal large model to obtain an output result of the multimodal large model; The output result includes: a first output result consisting of a first answer and a second answer, the first answer including: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is the object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

15. The device according to claim 14, wherein: The output result also includes: a second output result consisting of a third answer; The third answer includes: second text information for explaining that the first target object is not included in the target image.

16. The device according to claim 15, wherein: The multimodal large model includes: a visual encoder, a word segmenter, a large language model and a visual detector; The visual encoder is used to generate visual word units corresponding to the target image; The word segmenter is used to generate text words corresponding to the first question; The large language model is used to generate the first answer or the third answer according to the visual word unit and the text word unit; The visual detector is used to generate the second answer according to the object detection result information.

17. The device according to claim 16, wherein: The object detection result information includes: a detection mark word and M detection content words, where M is a positive integer greater than 1, and each detection content word includes: detection frame coordinate information and detection frame score information of the corresponding detection object, the detection mark word includes a start mark and an end mark, and the M detection content words are located between the start mark and the end mark, and the start mark and the end mark are used to mark the start and end of the M detection content words; The visual detector is used to filter out the detection objects whose corresponding detection frame scores are greater than a predetermined threshold from M detection objects as the first target objects, and use the detection frame coordinate information corresponding to the filtered detection objects as the coordinate information of the first target object.

18. The device according to any one of claims 14 to 17, wherein: The object detection module is further used to add a detection frame corresponding to the first target object to the target image according to the second answer, obtain a result image, and display it.

19. A multimodal large model training device, comprising: Sample acquisition module and model training module; The sample acquisition module is used to acquire a training sample, wherein the training sample includes: a sample image, a second question corresponding to the sample image, and a label corresponding to the sample image, wherein the label includes: third text information for indicating that the sample image includes a second target object, object detection result information for the sample image, and coordinate information of the second target object in the sample image, and the second target object is an object involved in the second question; The model training module is used to train the multimodal large model using the training samples, and the multimodal large model is used to generate an output result based on a target image and a first question corresponding to the target image, and the output result includes: a first output result consisting of a first answer and a second answer, the first answer includes: first text information used to illustrate that the target image includes a first target object and object detection result information for the target image, the first target object is an object involved in the first question, and the second answer includes: coordinate information of the first target object in the target image determined based on the object detection result information.

20. The device according to claim 19, wherein The output result also includes: a second output result consisting of a third answer; The third answer includes: second text information for explaining that the first target object is not included in the target image.

21. The device according to claim 20, wherein: The multimodal large model includes: a visual encoder, a word segmenter, a large language model and a visual detector; The visual encoder is used to generate visual word units corresponding to the target image; The word segmenter is used to generate text words corresponding to the first question; The large language model is used to generate the first answer or the third answer according to the visual word-gram and the text word-gram; The visual detector is used to generate the second answer according to the object detection result information.

22. The device according to claim 21, wherein The object detection result information includes: a detection mark word and M detection content words, where M is a positive integer greater than 1, and each detection content word includes: detection frame coordinate information and detection frame score information of the corresponding detection object, the detection mark word includes a start mark and an end mark, and the M detection content words are located between the start mark and the end mark, and the start mark and the end mark are used to mark the start and end of the M detection content words; The visual detector is used to filter out the detection objects whose corresponding detection frame scores are greater than a predetermined threshold from M detection objects as the first target objects, and use the detection frame coordinate information corresponding to the filtered detection objects as the coordinate information of the first target object.

23. The device according to claim 22, wherein: The model training module uses the sample image and the second question in any training sample as inputs of the multimodal large model, determines the language model loss and the box prediction loss according to the output results of the multimodal large model and the labels in the training sample, determines the comprehensive loss by combining the language model loss and the box prediction loss, and optimizes the parameters of the multimodal large model according to the comprehensive loss.

24. The device according to claim 23, wherein: The model training module determines the text loss based on the first text information output by the multimodal large model and the third text information in the label, determines the detection tag loss based on the detection tag word in the object detection result output by the multimodal large model and the detection tag word in the object detection result information in the label, and determines the language model loss by combining the text loss and the detection tag loss.

25. The device according to claim 23, wherein: The model training module selects a detection object that matches the second target object corresponding to the training sample from the detection objects corresponding to the M detection content words in the object detection result information output by the multimodal large model, determines the coordinate loss based on the detection frame coordinate information of the selected detection object and the coordinate information of the second target object in the label, and determines the frame prediction loss based on the coordinate loss.

26. The device according to claim 25, wherein The model training module determines the following three coordinate losses: mean absolute error loss of coordinates, generalized intersection-over-union loss of coordinates, and object classification loss of coordinates, and determines the box prediction loss by combining the three coordinate losses.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 13.

29. A computer program product, comprising a computer program / instruction, which implements the method according to any one of claims 1 to 13 when executed by a processor.

Citation Information

Patent Citations

  • Target identification and detection method based on multi-modal learning and related components

    CN114913487A

  • Cboth generation method and device, equipment, storage medium and product

    CN116702737A

  • Conversational target positioning method and device based on arbitrary granularity text input

    CN118350464A

  • Processing method and apparatus for visual question answering task, and device and non-volatile readable storage medium

    WO2024045444A1

Cited By

  • Multi-modal large model retrieval method and device for security check field

    CN119719397A

  • Multi-modal large model retrieval method and device for security field

    CN119719397B

  • Underground pipeline parameter detection method, device and equipment based on multi-modal algorithm and ground penetrating radar

    CN120472463A

  • Structured target detection method, device and equipment based on multi-modal language model

    CN120580514A

  • Structured object detection method, device and equipment based on multi-modal language model

    CN120580514B