Representation understanding method and device based on large language model and medium
Through the reference understanding method based on the large language model, combined with the recognition and reasoning of the visual model, a list of positionable steps is generated, which solves the data dependence and scenario adaptability problems in the prior art, and improves the accuracy and generalization of reference understanding. Metamorphosis.
Patent Information
- Application Number
- CN202411142477.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-27
AI Technical Summary
The existing refers to the understanding technology that highly relies on a large number of training data for specific tasks, which makes the data acquisition and training process time-consuming and labor-intensive, and difficult to generalize to new scenarios in which different data distributions are limited, limiting the development and application of technology.
Using a reference understanding method based on a large language model, by obtaining the text and images of the reference object, using the large language model to generate a positionable step list, and combining the visual model for visual recognition and reasoning, outputting the predicted reference object bounding box coordinates. , solves the problems of data dependence and scenario adaptability.
Improve the accuracy and generalization of reference understanding, reduce dependence on specific training data, and enhance the application scope and adaptability of the technology.
Smart Images
Figure CN119206166A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of human-computer interaction, multimodal learning, natural language processing and computer vision technology, and in particular to a method, device and medium for understanding referential expressions based on a large language model. Background Art
[0002] Referential expression understanding, also known as visual localization. This technology aims to understand the referential intentions implied by humans through natural language expressions, and accurately locate the visual objects referred to by humans in the image. Among them, this technology provides a basis for natural interaction between humans and intelligent agents (such as robots and embodied systems), so it belongs to the category of human-computer interaction; this technology involves the understanding of referential expression texts presented in the form of human natural language, so it belongs to the category of natural language processing; this technology requires analysis of image content, so it belongs to the category of computer vision; this technology conducts comprehensive reasoning on text and image content, so it belongs to the category of multimodal learning. As a basic technology for scenarios such as human-computer interaction and multimodality, referential expression understanding has a wide range of application scenarios and can play an auxiliary role in applications such as visual question answering, robot operation, and visual language navigation.
[0003] Existing referential expression understanding technologies mainly use model training. However, model training is highly dependent on a large amount of training data for specific tasks. When obtaining such data, a lot of manual time and effort is required, and many links such as task-specific data acquisition, labeling, and cleaning are required, which limits the development speed of the technology. In addition, referential expression understanding technologies obtained through fixed data training are often limited to scenarios that conform to specific data distributions, and are therefore difficult to promote and apply to new scenarios with different data distributions, which restricts the scope of technology development. Summary of the invention
[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a method, device and medium for understanding referential expressions based on a large language model.
[0005] The first technical solution adopted by the present invention is:
[0006] A method for understanding referential expressions based on a large language model comprises the following steps:
[0007] Acquire a referential expression text describing a referent object and an image containing the referent object;
[0008] Inputting the referential expression text into a large language model for text understanding and text generation, and outputting a step list that can locate the referential object; the step list includes multiple sub-steps;
[0009] Execute each sub-step in the step list in sequence, input the image and the operation object of the sub-step into the corresponding visual model in the sub-step for visual recognition, positioning and reasoning, screen out candidate objects that meet the constraints in the sub-step according to the results output by the visual model, obtain visual information of the candidate objects, use the candidate objects as the operation objects of the next sub-step, and pass the visual information of the candidate objects to the next sub-step;
[0010] When the last sub-step in the step list is executed, the candidate object output by the last sub-step is used as the predicted referent object, and the bounding box coordinates of the predicted referent object are output.
[0011] Furthermore, the large language model includes GPT series, Gemini series, Llama series and variants of the Llama series; wherein the large language model adopts a full-parameter training method, a fine-tuning training method or a non-training method.
[0012] Furthermore, the step list S is obtained by predicting the probability p of text generation by the large language model M under the condition of referring to the expression text E, which is specifically expressed as:
[0013] p(S|E;M)
[0014] S= <s1,…,s T >
[0015] Where s is the sub-step and T is the number of sub-steps.
[0016] Furthermore, the form of the step list S includes text, code and instruction; the structure of the step list S includes chain, and, or, not and combination.
[0017] Furthermore, the sub-step s is a constraint satisfaction problem; wherein the sub-step s includes a set of operation objects O in , a visual model V and a constraint k.
[0018] Furthermore, the execution of the sub-step s is specifically as follows:
[0019] The image I and the operation object O in Input to the visual model V, the visual model V performs corresponding visual recognition, positioning and reasoning operations, from the operation object O in Select the candidate objects O that meet the constraints cand , specifically expressed as:
[0020] O cand =V(O in ,I,k).
[0022] Furthermore, the visual model V is used to identify, locate and reason about images or visual objects; wherein the visual model V includes an open visual positioning model, an attribute recognition model and a relationship reasoning model;
[0023] The open visual positioning models include ResNet series, DarkNet series, DETR series, Transformer variants, and multimodal large models; wherein the multimodal large models include GPT series, Gemini series, Llama series, and variants of the Llama series;
[0024] The attribute recognition model includes the CLIP series, variants of the CLIP series, and a multimodal large model; wherein the multimodal large model includes the GPT series, the Gemini series, the Llama series, and variants of the Llama series;
[0025] The relational reasoning models include the CLIP series, variants of the CLIP series, and multimodal large models; wherein the multimodal large models include the GPT series, the Gemini series, the Llama series, and variants of the Llama series.
[0026] Furthermore, the visual information is a candidate object O cand The visual information of the object includes the category name, visual attributes, bounding box coordinates, and the relationship between objects.
[0027] The second technical solution adopted by the present invention is:
[0028] A device for understanding referential expressions based on a large language model, comprising:
[0029] at least one processor;
[0030] at least one memory for storing at least one program;
[0031] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0032] The third technical solution adopted by the present invention is:
[0033] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.
[0034] The beneficial effects of the present invention are as follows: the present invention utilizes the ability of a large language model to understand and generate complex texts, understands the intention of referential expression, and generates a list of steps that can locate the referential object, thereby solving the problem that existing referential expression technology is highly dependent on a large amount of training data for specific tasks, and solving the problem that existing referential expression technology is difficult to generalize to new scenarios that are different from the distribution of training data, thereby improving the accuracy and generalization of referential expression understanding and referential object positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0036] Figure 1 It is a flowchart of a method for understanding referential expressions based on a large language model in an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0038] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0039] In the description of the present invention, the meaning of "several" is one or more, the meaning of "more" is two or more, and the meanings of "greater than", "less than", "exceed" and the like are understood as not including the number itself, and the meanings of "above", "below", "within" and the like are understood as including the number itself. If there is a description of the first and the second, it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, "and / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship.
[0040] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0041] Terminology explanation:
[0042] GPT series: The GPT series includes models such as GPT-2, GPT-3, GPT-3.5, GPT-4, and GPT-4V. It is a language model based on generative pre-trained Transformer developed by OpenAI. It learns the statistical laws and semantic representation of language by pre-training on a large amount of unlabeled text, and then fine-tunes it on various downstream tasks, such as text generation, knowledge question answering, and reasoning calculation;
[0043] Gemini series: The Gemini series includes models such as Gemini 1.0, Gemini Pro, and Gemini Ultra. It is a series of language models released by DeepMind. It is based on the Transformer architecture and is multimodally trained using data such as videos, texts, images, and codes.
[0044] Llama series: The Llama series includes models such as Llama 1, Llama 2, and Llama 3. It is a series of open source large-scale language models released by Meta. They generally have multiple parameter scales such as 7B, 13B, and 33B, which are suitable for applications of different scales and support pre-training and fine-tuning of various natural language processing tasks.
[0045] Variants of the Llama series: models that are improved based on the Llama series models, including LLaVA1.5, LLaVA1.6 and other models, which can understand and generate multimodal data such as images and videos.
[0046] ResNet series: The ResNet series includes models such as ResNet-34, ResNet-50, ResNet-101, and FasterRCNN. It is a series of deep convolutional neural network architectures, which are characterized by residual connection operations. It solves the problem of gradient vanishing or exploding during training of deep neural networks, and has excellent performance in classification and detection applications.
[0047] DarkNet series: The DarkNet series includes models such as DarkNet-53, YOLOv7, and YOLOv6. It is a series of neural networks with a multi-scale prediction framework. It is characterized by an efficient and lightweight framework and has excellent performance in real-time target detection applications.
[0048] DETR series: DETR is a target detection model provided by Facebook. It combines Transformer and convolutional neural networks to directly predict the category and location of the target in an end-to-end manner, simplifying the complex pipeline in traditional target detection. It is efficient and simple.
[0049] Transformer variants: various improved and extended models based on the original Transformer architecture. These variants are usually optimized or enhanced in specific application scenarios, including BERT, T5, ViT and other models.
[0050] CLIP series: CLIP is a model that maps images and text into the same feature space through contrastive learning, so that cross-modal tasks such as image-text retrieval and image description generation can be performed.
[0051] Variants of the CLIP series: models that are improved based on the CLIP framework, adding downstream task modules and frameworks to optimize in specific application scenarios, including GLIP, BLIP and other models.
[0052] like Figure 1 As shown, this embodiment provides a method for understanding referential expressions based on a large language model, and the specific steps include:
[0053] S1. Obtain a referential expression text E describing a referential object and an image I containing the referential object.
[0054] S2. Input the reference expression text E into the large language model M for text understanding and text generation, and output a step list S that can locate the reference object.
[0055] The large language model M is used to understand the referential expression text E and generate a step list S accordingly. The large language model M includes the GPT series, Gemini series, Llama series and variants of the Llama series. The large language model M adopts a full parameter training method, a fine-tuning training method or a non-training method.
[0056] In an optional embodiment, the large language model M is GPT-4, which is trained without any training.
[0057] The step list S is a list of T sub-steps s, which is specifically expressed as:
[0058] S= <s1,…,s T >
[0059] The step list S is obtained by predicting the probability p of text generation by the large language model M under the condition of referring to the expression text E, which is specifically expressed as:
[0060] p(S|E;M)
[0061] The form of the step list S includes text, code and instruction, and the structure of the step list S includes chain, and, or, negate and combination.
[0062] In an optional embodiment, the step list S is in the form of code, and the structure of the step list S is chained.
[0063] S3, execute each sub-step s in the step list S in sequence, and convert the image I and the operation object O of the sub-step into in Input into the corresponding visual model V in the sub-step s for visual recognition, positioning and reasoning, and screen out candidate objects O that meet the constraint k in the sub-step s according to the results of the visual model cand , obtain the candidate object O cand The visual information of the candidate object O cand As the operation object of the next substep O in , and the candidate object O cand The visual information is passed to the next sub-step.
[0064] Sub-step s is a constraint satisfaction problem. Sub-step s includes a set of operation objects O in , a visual model V and a constraint k. The execution of sub-step s is as follows: the image I and the operation object O in Input to the visual model V, the visual model V performs corresponding visual recognition, positioning and reasoning operations, from the operation object O in Select the candidate objects O that meet the constraints cand , specifically expressed as:
[0065] O cand =V(O in ,I,k)
[0066] A visual model V is used to identify, locate and reason about images or visual objects. The visual model V includes an open visual positioning model, an attribute recognition model and a relational reasoning model. The open visual positioning model includes the ResNet series, the DarkNet series, the DETR series, a variant of the Transformer, and a multimodal large model; wherein the multimodal large model includes the GPT series, the Gemini series, the Llama series and a variant of the Llama series. The attribute recognition model includes the CLIP series, a variant of the CLIP series, and a multimodal large model; wherein the multimodal large model includes the GPT series, the Gemini series, the Llama series and a variant of the Llama series. The relational reasoning model includes the CLIP series, a variant of the CLIP series, and a multimodal large model; wherein the multimodal large model includes the GPT series, the Gemini series, the Llama series and a variant of the Llama series.
[0067] In an optional embodiment, the visual model V includes an open visual localization model of the DETR series, a variant attribute recognition model of the CLIP series, and a relational reasoning model of the GPT series.
[0068] Visual information is the candidate object O cand The visual information of the object includes the category name, visual attributes, bounding box coordinates, and the relationship between objects.
[0069] In an optional embodiment, the visual information is the category name and bounding box coordinates.
[0070] S4, iterate step S3 until the last sub-step s in the step list S is executed, and the candidate object O output by the last sub-step s is cand As the predicted referent object, the bounding box coordinates of the predicted referent object are output.
[0071] This embodiment also provides a device for understanding referential expressions based on a large language model, including:
[0072] at least one processor;
[0073] at least one memory for storing at least one program;
[0074] When the at least one program is executed by the at least one processor, the at least one processor implements the following Figure 1 The method shown.
[0075] A referential expression understanding device based on a large language model in this embodiment can execute a referential expression understanding method based on a large language model provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0076] The present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 1 The method shown.
[0077] This embodiment also provides a storage medium storing instructions or programs that can execute a method for understanding referential expressions based on a large language model provided by an embodiment of the method of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.
[0078] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0079] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0080] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0081] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0082] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0083] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0084] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0085] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0086] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A method for understanding referential expressions based on a large language model, characterized in that: The following steps are involved: Acquire a referential expression text describing a referent object and an image containing the referent object; Inputting the referential expression text into a large language model for text understanding and text generation, and outputting a step list that can locate the referential object; the step list includes multiple sub-steps; Execute each sub-step in the step list in sequence, input the image and the operation object of the sub-step into the corresponding visual model in the sub-step for visual recognition, positioning and reasoning, screen out candidate objects that meet the constraints in the sub-step according to the results output by the visual model, obtain visual information of the candidate objects, use the candidate objects as the operation objects of the next sub-step, and pass the visual information of the candidate objects to the next sub-step; When the last sub-step in the step list is executed, the candidate object output by the last sub-step is used as the predicted referent object, and the bounding box coordinates of the predicted referent object are output.
2. A method for understanding referential expressions based on a large language model according to claim 1, characterized in that: The large language models include the GPT series, the Gemini series, the Llama series, and variants of the Llama series.
3. A method for understanding referential expressions based on a large language model according to claim 1, characterized in that: The step list S is obtained by predicting the probability p of text generation by the large language model M under the condition of referring to the expression text E, which is specifically expressed as: p(S|E;M) S=<s1,…,s T > Where s is the sub-step and T is the number of sub-steps.
4. The method for understanding referential expressions based on a large language model according to claim 1, characterized in that: The form of the step list S includes text, code and instruction; the structure of the step list S includes chain, and, or, negate and combination.
5. The method for understanding referential expressions based on a large language model according to claim 1, characterized in that: The sub-step s is a constraint satisfaction problem; wherein the sub-step s includes a set of operation objects O in , a visual model V and a constraint k.
6. A method for understanding referential expressions based on a large language model according to claim 1 or 5, characterized in that: The execution of the sub-step s is as follows: The image I and the operation object O in Input to the visual model V, the visual model V performs corresponding visual recognition, positioning and reasoning operations, from the operation object O in Select the candidate objects O that meet the constraints cand , specifically expressed as: O cand =V(O in ,I,k)。 7. The method for understanding referential expressions based on a large language model according to claim 1, characterized in that: The visual model V is used to identify, locate and reason about images or visual objects; wherein the visual model V includes an open visual positioning model, an attribute recognition model and a relationship reasoning model; The open visual positioning models include ResNet series, DarkNet series, DETR series, Transformer variants, and multi-modal large models; The attribute recognition model includes the CLIP series, variants of the CLIP series, and multimodal large models; The relational reasoning models include the CLIP series, variants of the CLIP series, and multimodal large models.
8. The method for understanding referential expressions based on a large language model according to claim 1, characterized in that: The visual information is a candidate object O cand The visual information of the object includes the category name, visual attributes, bounding box coordinates, and the relationship between objects.
9. A device for understanding referential expression based on a large language model, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 8.
10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 8 when executed by the processor.