A system and method for quickly implementing image grounding capabilities for large language models
Through systematic data fusion and network structure design, the accuracy of large language models in image target position prediction is improved, the problem of insufficient accuracy and stability in the existing technology is solved, and the same object detection capability as professional models is achieved.
Patent Information
- Application Number
- CN202411871500.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-04-04
AI Technical Summary
现有的大语言模型在图像目标位置的精准描述能力不足,无法达到专业图像目标检测模型的精度和稳定性。
A system is designed, including visual coding layer, text coding layer, projection layer, transformers decoder module, bottleneck hidden layer, text output head and grounded output head module. Data fusion is achieved through self-attention network, text cross attention network and image cross attention network to achieve accurate prediction of target positions.
On the basis of ensuring the original text understanding and reasoning ability, the target position prediction ability of large language models is achieved with the same accuracy and stability as that of professional models such as yolo, and only a small amount of data is required to train.
Smart Images
Figure CN119762938A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image data processing, and in particular relates to a system and method for rapidly realizing image grounding capability for a large language model. Background Art
[0002] With the rise of big model technology and embodied intelligent robots, it has become possible to assist or even replace humans in completing many things in daily life and work. To achieve this, multimodal interaction capabilities are indispensable. Among them, various Internet searches and the use of mobile phones are inseparable from the operation of specific objects in various page images.
[0003] Although the current large language models and cross-modal large models have rich text and even image understanding capabilities, as well as certain reasoning capabilities, their ability to accurately describe the location of targets in images (grounding capabilities) is still inferior to professional image target detection models such as yolo, dert, and DINO. Summary of the invention
[0004] The present invention provides a system and method for quickly realizing image grounding capability for large language models. To help cross-modal large language models achieve the same accuracy and stability of target position prediction capability as YOLO while ensuring the original text understanding and reasoning capabilities, the technical solution of the present invention is as follows: a system for quickly realizing image grounding capability for large language models, the system includes the system comprising a visual encoding layer, a text encoding layer, a first projection layer, a transformers decoder module, a bottleneck hidden layer, a second projection layer, a text output head and a grounding output head module. The second projection layer module consists of four sub-modules: self-attention network, text cross-attention network, image cross-attention network and fully connected network.
[0005] As an improvement of the present invention, the grounded output head module is composed of the output head of any image target detection neural network, including the head network of the YOLO model or the Fast RCNN model.
[0006] As an improvement of the present invention, the visual encoding layer is a vit visual encoding network, and the text encoding layer is an embedding network.
[0007] As an improvement of the present invention, the first projection layer is composed of a concat layer and a position encoding network, wherein the transformers decoder module is composed of the decoder part of the transformers model connection structure, which is the backbone network part of any open source large model, and the bottleneck hidden layer is composed of a single or multiple fully connected networks.
[0008] As an improvement of the present invention, a method for quickly realizing image grounding capability for a large language model using the system includes the following steps: In the first step, the image is sent to the visual encoding layer and the text is sent to the text encoding layer to obtain their respective outputs, which are recorded as f_v and f_text respectively; In the second step, the outputs of the first step are sent to the first projection layer, and after the data fusion processing of the first projection layer, the output of the second step is obtained; In the third step, the output of the second step is sent to the transformer decoder module for forward reasoning calculation to obtain the output of the third step; The fourth step is to receive the output of the third step, perform calculations on the bottleneck hidden layer, and obtain the final feature encoding of the fused image and text information, denoted as f; Step 5: The text output head receives f and outputs the content generated by the text reasoning of the entire system; Step 6: Input f and f_v into the second projection layer at the same time, and after the mapping calculation of the second projection layer, output the feature layer, which is recorded as ff_v; In the seventh step, the ground output head module performs mapping calculation on ff_v and outputs the ground coordinate information of the image to the system, which is recorded as Y_box.
[0009] As an improvement of the present invention, during the training phase of the entire network, for a training sample: When the text output header outputs a termination token that is not equal to the termination token identifier, the true value label corresponding to Y_box is empty, denoted as []. When the text output header is output, the output token is the termination token identifier, and Y_box is the target coordinates [x1, y1, x2, y2], where x1, y1 are the normalized coordinate values of the upper left corner of the target position, and x2, y2 are the normalized coordinate values of the lower right corner of the target position.
[0010] As an improvement of the present invention, the network training is a two-stage training, including: In the first stage, when calculating the loss function, the Y_box information in the data set is masked, and the back propagation calculation is not performed on the ground output head and the second projection layer; In the second stage of training, the loss function of the text output and the loss function of the grounded output are calculated simultaneously. During back-propagation, the text output head is not back-propagated, and the bottleneck hidden layer, transformer decoder module, first projection layer, and text encoding layer are selectively frozen.
[0011] As an improvement of the present invention, the mapping calculation of the second projection layer in step 6 includes: The first step is to perform self-attention network mapping calculation on the input f_v, and use the back propagation of the network to calculate the three mapping matrices Q, K, and V of the self-attention network. In the forward reasoning stage, the output of the self-attention network is calculated, which is recorded as ff1; In the second step, in the text cross-attention network submodule, f is connected to the Q and K matrices, ff1 is connected to the Q matrix, and cross-attention calculation is performed on ff1 and f to obtain the output ff2; The third step is to perform another cross-attention calculation on ff2 and f_v in the image cross-attention network module to obtain the output ff3. The fourth step is to calculate ff3 through a fully connected network to obtain the output ff_v.
[0012] Compared with the prior art, the beneficial effects of the present invention are: Different from the acceptable input of the GroudingDINO model, the input of GroudingDINO is "the description text / prompt text of the detection target" and the image to be detected. The input text of this method is complex, and the text description information of the target to be detected is inferred by a large model, and the hidden layer of the bottleneck layer of the model is used as input; Different from the network structure of the reasoning output of the GroudingDINO model, the location information of the target in the present invention is output by the visual head, and the category information is output by the text head. The text head supports any open category. The structural difference compared with the Large VM is that the system has two output heads, and the target position information is output by a separate prediction network branch instead of being directly predicted by the decoder network and the text; The present invention provides a system and method for quickly realizing image grounding capability for a large language model, wherein the introduction of a first projection layer and a grounding output head, as well as the definition of a training method thereof, realizes multi-head output of a cross-modal large model, and helps the cross-modal large language model to achieve the target position prediction capability with the same accuracy and stability as target detection models such as YOLO while ensuring the original text understanding and reasoning capabilities, with only a small amount of data training required. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 This is a flow chart of the system and method for quickly implementing image grounding capabilities for a large language model in this embodiment; Figure 2 This is a structural diagram of the projection layer 2 of this embodiment. DETAILED DESCRIPTION
[0014] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0015] Example: Figure 1-2 As shown, this embodiment provides an application in a GUI agent, including: The system consists of eight parts: visual encoding layer, text encoding layer (text embedding layer), projection layer 1, transformers decoder module, bottleneck hidden layer, projection layer 2, text output head and ground output head.
[0016] In the first step, the image is sent to the visual encoding layer and the text is sent to the text encoding layer to obtain their respective outputs, which are recorded as f_v and f_text respectively; In the second step, the outputs of the first step are sent to the projection layer 1, and after the data fusion processing of the projection layer, the output of the second step is obtained; In the third step, the output of the second step is sent to the transformer decoder module for forward reasoning calculation to obtain the output of the third step. In the fourth step, the output of the third step is received and the bottleneck hidden layer is calculated to obtain the final feature encoding of the fused image and text information, which is recorded as f. Step 5: The text output head receives f and outputs the content generated by the text reasoning of the entire system; Step 6: Input f and f_v into projection layer 2 at the same time, and after the mapping calculation of the projection layer, output the feature layer, which is recorded as ff_v; In the seventh step, the ground output head module performs mapping calculation on ff_v and outputs the ground coordinate information of the image to the system, which is recorded as Y_box.
[0017] During the training phase of the entire network, for a training sample, when the text output head outputs a termination token that is not equal to the termination token identifier, the true value label corresponding to Y_box is empty, recorded as []. When the text output head outputs a termination token identifier, Y_box is the target coordinates [x1, y1, x2, y2], where x1, y1 are the normalized coordinate values of the upper left corner of the target position, and x2, y2 are the normalized coordinate values of the lower right corner of the target position.
[0018] The network training is a two-stage training. In the first stage, when calculating the loss function, the Y_box information in the dataset is masked, and the ground output head and projection layer 2 are not back-propagated. In the second stage of training, the loss function of the text output and the ground output loss function are calculated at the same time. During back-propagation, the text output head is not back-propagated. The bottleneck hidden layer, transformer decoder module, projection layer 1, and text embedding layer can be selectively frozen.
[0019] The projection layer 2 module consists of four sub-modules: self attenion (self-attention network), text cross attention (text cross attention network), image cross attention (network) and FFN (fully connected network).
[0020] Specific process: The first step is to perform self-attention mapping calculation on the input f_v, and use the back propagation of the network to calculate the three mapping matrices Q, K, and V of self-attention. In the forward reasoning stage, the output of self-attention is calculated, recorded as ff1; In the second step, in the text cross attention submodule, f is connected to the Q and K matrices, ff1 is connected to the Q matrix, and cross attention calculation is performed on ff1 and f to obtain the output ff2; In the third step, in the image cross attention module, ff2 and f_v perform another cross attention calculation to obtain the output ff3; in the fourth step, ff3 is subjected to a fully connected network calculation (FFN) to obtain the output ff_v.
[0021] For 1, the grounded output head module can be composed of the output head of any image object detection neural network, such as the head network of the YOLO model or the Fast RCNN model.
[0022] For 1, the visual encoding layer is a vit visual encoding network; the text encoding layer (text embedding layer) is an embedding network; the projection layer 1 can be composed of a concat layer and a position encoding network; the transformers decoder module is composed of the decoder part of the transformers model connection structure, which can be the backbone network part of any open source large model; the bottleneck hidden layer can be composed of a single or multiple fully connected networks.
[0023] It should be noted that the above content only illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.
Claims
1. A system for quickly implementing image grounding capabilities for large language models, characterized in that: The system comprises eight parts: a visual encoding layer, a text encoding layer, a first projection layer, a transformer decoder module, a bottleneck hidden layer, a second projection layer, a text output head and a ground output head module; The second projection layer module consists of four sub-modules: self-attention network, text cross-attention network, image cross-attention network and fully connected network.
2. The system for rapidly implementing image grounding capability for a large language model according to claim 1, characterized in that: The grounded output head module consists of the output head of any image object detection neural network, including the head network of the YOLO model or the Fast RCNN model.
3. The system for rapidly implementing image grounding capability for a large language model according to claim 1, characterized in that: The visual encoding layer is a vit visual encoding network, and the text encoding layer is an embedding network.
4. The system for rapidly implementing image grounding capability for a large language model according to claim 1, characterized in that: The first projection layer is composed of a concat layer and a position encoding network. The transformers decoder module is composed of the decoder part of the transformer model connection structure, which is the backbone network part of any open source large model. The bottleneck hidden layer is composed of a single or multiple fully connected networks.
5. A method for quickly realizing image grounding capability for a large language model using the system according to any one of claims 1 to 4, characterized in that: The steps include: In the first step, the image is sent to the visual encoding layer and the text is sent to the text encoding layer to obtain their respective outputs, which are recorded as f_v and f_text respectively; In the second step, the outputs of the first step are sent to the first projection layer, and after the data fusion processing of the first projection layer, the output of the second step is obtained; In the third step, the output of the second step is sent to the transformer decoder module for forward reasoning calculation to obtain the output of the third step; The fourth step is to receive the output of the third step, perform calculations on the bottleneck hidden layer, and obtain the final feature encoding of the fused image and text information, denoted as f; Step 5: The text output head receives f and outputs the content generated by the text reasoning of the entire system; Step 6: Input f and f_v into the second projection layer at the same time, and after the mapping calculation of the second projection layer, output the feature layer, which is recorded as ff_v; In the seventh step, the ground output head module performs mapping calculation on ff_v and outputs the ground coordinate information of the image to the system, which is recorded as Y_box.
6. The system and method for quickly implementing image grounding capability for a large language model according to claim 5, characterized in that: During the training phase of the entire network, for a training sample: When the text output header outputs a termination token that is not equal to the termination token identifier, the true value label corresponding to Y_box is empty, denoted as []. When the text output header is output, the output token is the termination token identifier, and Y_box is the target coordinates [x1, y1, x2, y2], where x1, y1 are the normalized coordinate values of the upper left corner of the target position, and x2, y2 are the normalized coordinate values of the lower right corner of the target position.
7. The system and method for quickly implementing image grounding capability for a large language model according to claim 5, characterized in that: The network training is a two-stage training, including: In the first stage, when calculating the loss function, the Y_box information in the data set is masked, and the back propagation calculation is not performed on the ground output head and the second projection layer; In the second stage of training, the loss function of the text output and the loss function of the grounded output are calculated simultaneously. During back-propagation, the text output head is not back-propagated, and the bottleneck hidden layer, transformer decoder module, first projection layer, and text encoding layer are selectively frozen.
8. The system and method for quickly implementing image grounding capability for a large language model according to claim 5, characterized in that: The mapping calculation of the second projection layer in step 6 includes: The first step is to perform self-attention network mapping calculation on the input f_v, and use the back propagation of the network to calculate the three mapping matrices Q, K, and V of the self-attention network. In the forward reasoning stage, the output of the self-attention network is calculated, which is recorded as ff1; In the second step, in the text cross-attention network submodule, f is connected to the Q and K matrices, ff1 is connected to the Q matrix, and cross-attention calculation is performed on ff1 and f to obtain the output ff2; The third step is to perform another cross-attention calculation on ff2 and f_v in the image cross-attention network module to obtain the output ff3. The fourth step is to calculate ff3 through a fully connected network to obtain the output ff_v.