A method for generating image description text based on large language model

Abstract

The present invention discloses a method for generating image description text based on a large language model. By constructing an image description text generation model, the entity information and entity relationship scene graph of the image are obtained; then the entity information and entity relationship scene graph are introduced into the large language model, and the initial description of the image is obtained for further polishing to obtain the final polished text. The present invention can accurately identify and describe each object in the image, and can also clearly express the complex interactive relationship between them. This thinking chain description generation method not only improves the accuracy of the model-generated description, but also makes the description more in line with human cognitive habits. Through step-by-step guidance, the description finally generated can better convey the overall information and intention of the image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image text processing, and in particular relates to a method for generating image description text based on a large language model. Background Art

[0002] Image description generation is a complex task that combines vision and natural language processing technologies. Its core goal is to enable computers to accurately recognize and understand the content in an image and generate descriptions in natural language. The advancement of this technology is not only of great significance in academic research, but also has broad prospects in practical applications, such as automatically annotating images, assisting the visually impaired, and improving the performance of image search engines. Through continuous technological innovation and optimization, image description generation will play a more important role in future development.

[0003] The first challenge that needs to be addressed in image description generation is the problem of identifying objects in the image. This involves the development of computer vision techniques, such as convolutional neural networks (CNNs), which are able to gradually refine and understand the various elements in the image through layer-by-layer feature extraction, but there are still limitations. Although the current datasets used for Image Caption tasks contain a large number of images and descriptions, they still have limitations. For example, some items may appear less frequently in the dataset, resulting in the model being unable to fully learn the characteristics and descriptions of these items during training. In addition, in some images, objects may be occluded or overlapped by other objects, making it more difficult to identify these objects.

[0004] In addition, image description generation requires not only that the computer can recognize the objects in the image, but also that it can understand how these objects interact. For example, in a picture, there is a cat sitting on a table. The computer not only needs to recognize "cat" and "table", but also needs to understand the relationship between "cat on table". If the interaction relationship between objects cannot be accurately identified, the model cannot generate accurate image descriptions.

[0005] Based on this, recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) play an important role in this regard. They are able to generate fluent and grammatical text descriptions based on the information in the image. In addition, recent research has also introduced Transformers models such as BERT and GPT, which have shown higher effectiveness and flexibility in generating natural language text.

[0006] However, the text descriptions generated by current image captioning are rigid, straightforward, and less vivid. However, human image descriptions are often not limited to the information of the image itself, but are enriched through imagination and association. In the image description generation task, it is necessary not only to generate accurate descriptions, but also to make the descriptions more vivid and specific. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a method for generating image description text based on a large language model, comprising the following steps:

[0008] Step S1: Get several images and corresponding labels, and use them to build a memory library;

[0009] Step S2: construct an image description text generation model, which includes an object extraction module and a relationship extraction module, an initial description text generation module and a text polishing module; import an image I that needs text description into the object extraction module, and obtain entity information of the image based on the memory library in step S1;

[0010] Step S3: importing the image I and the entity information of the image that need text description in step S2 into the relationship extraction module to obtain the corresponding entity relationship scene graph;

[0011] Step S4: importing the image I that needs to be described in text in step S2, the entity information in step S2, and the entity relationship scene graph in step S3 into the initial description text generation module to obtain the corresponding initial description;

[0012] Step S5: Import the image I that needs text description in step S2 and the initial description text of the image imported in step S4 into the text polishing module to obtain the final polished text.

[0013] Furthermore, step S1 is specifically as follows:

[0014] Use the image encoder to obtain several images Encode to get image embedding; expressed as:

[0015] ;

[0016] ;

[0017] ;

[0018] in, represents the image encoding operation, Representing images The Sample, Representing image embedding The samples, represents the total number of images in the memory bank, represents image embedding;

[0019] The image embedding and the corresponding label are stored in the form of key-value pairs. Several pairs of key-value pairs constitute the memory library, where the image embedding is the key and the corresponding label is the value, expressed as:

[0020] ;

[0021] ;

[0022] in, Represents the memory bank, represents the i-th key, represents the i-th value, Representing image embedding The Image embedding, Indicates The images are embedded with corresponding labels.

[0023] Further, the object extraction module in step S2 includes an image segmentation module and an image encoder;

[0024] Step S2 specifically includes: inputting the image I that needs text description into the image segmentation module, and obtaining the bounding boxes and masks of all objects in the image I that needs text description;

[0025] According to the pixel values ​​in the mask, the pixel values ​​of the part with objects are set to 1, and the pixel values ​​of the part without objects are set to 0. The mask is multiplied with the image I that needs to be described in text to obtain the cutout of the object in the image I, which is expressed as:

[0026] ;

[0027] ;

[0028] in, represents the image segmentation operation, represents the mask of the i-th object, is the cutout of the i-th object, n is the number of all objects in the image, Represents the cutout of the object in image I;

[0029] The cutout of each object is passed through the image encoder to obtain the corresponding image embedding , to get the total image embedding , expressed as:

[0030] ;

[0031] Embed the corresponding image of each object The memory bank in step S1 All images in the corresponding key are embedded in The cosine similarity calculation is expressed as:

[0032] = ;

[0033] in, , represents the cosine similarity function;

[0034] Get the key of the most similar object , and then obtain Corresponding value ; In this way, the image in the memory bank that is most similar to the input image I that needs text description is obtained, and then the corresponding label is obtained, and this label is assigned to the image I that needs text description as the result of its object recognition, that is, the entity information ; expressed as:

[0035] ;

[0036] ;

[0037] ;

[0038] Among them, j is the subscript index value of the key of the object with the highest similarity, represents the maximum value function, Represents the label corresponding to the nth object in image I.

[0039] Furthermore, step S3 is specifically as follows: construct relevant prompts based on the identified entity information Input a large language model to obtain the interaction relationship between various objects in the image I that needs to be described in text; the interaction relationship includes the relative position, action and state between the objects; construct the objects and the interaction relationship in the form of a graph structure to obtain a scene graph, wherein the nodes of the graph structure represent objects and the edges of the graph structure represent interaction relationships;

[0040] Among them, relevant tips for large language model design for:

[0041] ;

[0042] in, Indicates interactive relationship condition related prompts, used to set generation conditions. Represents entity information. Indicates interactive task-related prompts, used to set generation requirements;

[0043] Images and related tips Input into the large language model to generate an object relationship scene graph ; expressed as:

[0044] ;

[0045] in For large language models, is the visual encoder, is a text encoder;

[0046] Furthermore, in the process of constructing the object relationship scene graph, the objects are sorted by importance, which is expressed as:

[0047] Sort each object by importance according to the score to determine which objects are the main body of the image and which objects are in a secondary position;

[0048] The constraints for sorting are as follows: the score of the object in the foreground is greater than that of the object in the background, and / or; the closer the object is to the center of the image I, the higher the score, and / or; the larger the area occupied by the object in the image I, the higher the score, and / or; the more times the object interacts with other objects, the higher the score;

[0049] According to the above constraints, calculate the importance score of each object , expressed as:

[0050] ;

[0051] in, To obtain the calculation function of the Euclidean distance between the center of the object position and the center of the entire image I, To obtain the calculation function of the area of ​​the mask corresponding to the object, and is a hyperparameter, To obtain the calculation function of the degree of the object in the object relationship scene graph, Represents an object;

[0052] According to importance score , sort the objects from high to low according to their scores; update by sorting to get the updated objects ;

[0053] The calculation method of the object position center is specifically as follows:

[0054] According to the bounding box obtained in step S2, it is assumed that the object The position coordinates of the four vertices of the corresponding bounding box are:

[0055] ;

[0056] The coordinates of its center position are:

[0057] ;

[0058] Assume that the position coordinates of the geometric center of the entire image I are ;

[0059] but:

[0060] .

[0061] Furthermore, step S4 is specifically as follows:

[0062] Constructing a second related prompt ; Through the second relevant prompt An object relationship scene graph will be generated , updated object Input the initial description text generation module based on the large language model and combine it with the image I that needs text description to generate the initial description , expressed as:

[0063] ;

[0064] ;

[0065] in, Indicates the initial description condition related prompts, Indicates initial description task related prompts.

[0066] Furthermore, step S5 is specifically as follows:

[0067] Constructing the third relevant hint to guide the text polishing module based on the large language model to the initial description Further reasoning and expansion are performed, and the final polished text is obtained by combining the image I that needs text description. , expressed as:

[0068] ;

[0069] ;

[0070] in, Indicates tips related to polishing text conditions. Indicates tips related to the text polishing task.

[0071] The positive and progressive effects of the present invention are:

[0072] 1) The present invention can accurately identify and describe each object in the image, and can also clearly express the complex interactive relationship between them. This chain-like description generation method not only improves the accuracy of the model-generated description, but also makes the description more in line with human cognitive habits. Through step-by-step guidance, the description finally generated can better convey the overall information and intention of the image.

[0073] 2) Furthermore, the present invention divides the image into multiple objects and sorts them by importance to determine which objects are the main body of the image and which objects are in a secondary position. After determining the description priority and importance sorting, when describing the image, first describe the main object in detail to clarify their characteristics, position, action and other information. For example, when describing an image containing a person and a landscape, first describe the person's appearance, posture and activities in detail, and then describe the main features of the landscape. For secondary objects, briefly describe them or only mention their existence without elaborating on them. This description method can make the image content clearly layered and highlight the key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 The present invention is a flowchart of the steps of a method for generating image description text based on a large language model. DETAILED DESCRIPTION

[0075] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0076] refer to Figure 1 , a method for generating image description text based on a large language model, in one example, comprises the following steps:

[0077] Step S1: Get several images and corresponding labels, and use them to build a memory library;

[0078] Step S2: construct an image description text generation model, which includes an object extraction module and a relationship extraction module, an initial description text generation module and a text polishing module; import an image I that needs text description into the object extraction module, and obtain entity information of the image based on the memory library in step S1;

[0079] Step S3: importing the image I and the entity information of the image that need text description in step S2 into the relationship extraction module to obtain the corresponding entity relationship scene graph;

[0080] Step S4: importing the image I that needs to be described in text in step S2, the entity information in step S2, and the entity relationship scene graph in step S3 into the initial description text generation module to obtain the corresponding initial description;

[0081] Step S5: Import the image I that needs text description in step S2 and the initial description text of the image imported in step S4 into the text polishing module to obtain the final polished text.

[0082] Furthermore, in one example, step S1 is specifically as follows:

[0083] Use the image encoder to obtain several images Encode to get image embedding; expressed as:

[0084] ;

[0085] ;

[0086] ;

[0087] in, represents the image encoding operation, Representing images The Sample, Representing image embedding The samples, represents the total number of images in the memory bank, represents image embedding;

[0088] The image embedding and the corresponding label are stored in the form of key-value pairs. Several pairs of key-value pairs constitute the memory library, where the image embedding is the key and the corresponding label is the value, expressed as:

[0089] ;

[0090] ;

[0091] in, Represents the memory bank, represents the i-th key, represents the i-th value, Representing image embedding The Image embedding, Indicates The images are embedded with corresponding labels.

[0092] Further, the object extraction module in step S2 includes an image segmentation module and an image encoder;

[0093] Step S2 specifically includes: inputting the image I that needs text description into the image segmentation module, and obtaining the bounding boxes and masks of all objects in the image I that needs text description;

[0094] According to the pixel values ​​in the mask, the pixel values ​​of the part with objects are set to 1, and the pixel values ​​of the part without objects are set to 0. The mask is multiplied with the image I that needs to be described in text to obtain the cutout of the object in the image I, which is expressed as:

[0095] ;

[0096] ;

[0097] in, represents the image segmentation operation, represents the mask of the i-th object, is the cutout of the i-th object, n is the number of all objects in the image, Represents the cutout of the object in image I;

[0098] The cutout of each object is passed through the image encoder to obtain the corresponding image embedding , to get the total image embedding , expressed as:

[0099] ;

[0100] Embed the corresponding image of each object The memory bank in step S1 All images in the corresponding key are embedded in The cosine similarity calculation is expressed as:

[0101] = ;

[0102] in, , represents the cosine similarity function;

[0103] Get the key of the most similar object , and then obtain Corresponding value ; In this way, the image in the memory bank that is most similar to the input image I that needs text description is obtained, and then the corresponding label is obtained, and this label is assigned to the image I that needs text description as the result of its object recognition, that is, the entity information ; expressed as:

[0104] ;

[0105] ;

[0106] ;

[0107] Among them, j is the subscript index value of the key of the object with the highest similarity, represents the maximum value function, Represents the label corresponding to the nth object in image I.

[0108] Furthermore, in one example, step S3 specifically includes: constructing relevant prompts based on the identified entity information Input a large language model to obtain the interaction relationship between various objects in the image I that needs to be described in text; the interaction relationship includes the relative position, action and state between the objects; construct the objects and the interaction relationship in the form of a graph structure to obtain a scene graph, wherein the nodes of the graph structure represent objects and the edges of the graph structure represent interaction relationships;

[0109] Among them, relevant tips for large language model design for:

[0110] ;

[0111] in, Indicates interactive relationship condition related prompts, used to set generation conditions. Represents entity information. Indicates interactive task-related prompts, used to set generation requirements;

[0112] Images and related tips Input into the large language model to generate an object relationship scene graph ; expressed as:

[0113] ;

[0114] in For large language models, is the visual encoder, is a text encoder;

[0115] Furthermore, in one example, in the process of constructing the object relationship scene graph, the objects are sorted by importance, which is expressed as:

[0116] Sort each object by importance according to the score to determine which objects are the main body of the image and which objects are in a secondary position;

[0117] The constraints for sorting are as follows: the score of the object in the foreground is greater than that of the object in the background, and / or; the closer the object is to the center of the image I, the higher the score, and / or; the larger the area occupied by the object in the image I, the higher the score, and / or; the more times the object interacts with other objects, the higher the score;

[0118] According to the above constraints, calculate the importance score of each object , expressed as:

[0119] ;

[0120] in, To obtain the calculation function of the Euclidean distance between the center of the object position and the center of the entire image I, To obtain the calculation function of the area of ​​the mask corresponding to the object, and is a hyperparameter, To obtain the calculation function of the degree of the object in the object relationship scene graph, Represents an object;

[0121] According to importance score , sort the objects from high to low according to their scores; update by sorting to get the updated objects ;

[0122] The calculation method of the object position center is specifically as follows:

[0123] According to the bounding box obtained in step S2, it is assumed that the object The position coordinates of the four vertices of the corresponding bounding box are:

[0124] ;

[0125] The coordinates of its center position are:

[0126] ;

[0127] Assume that the position coordinates of the geometric center of the entire image I are ;

[0128] but:

[0129] .

[0130] Furthermore, in one example, step S4 is specifically as follows:

[0131] Constructing a second related prompt ; Through the second relevant prompt An object relationship scene graph will be generated , updated object Input the initial description text generation module based on the large language model and combine it with the image I that needs text description to generate the initial description , expressed as:

[0132] ;

[0133] ;

[0134] in, Indicates the initial description condition related prompts, Indicates initial description task related prompts.

[0135] Furthermore, in one example, step S5 is specifically as follows:

[0136] Constructing the third relevant hint to guide the text polishing module based on the large language model to the initial description Further reasoning and expansion are performed, and the final polished text is obtained by combining the image I that needs text description. , expressed as:

[0137] ;

[0138] ;

[0139] in, Indicates tips related to polishing text conditions. Indicates tips related to the text polishing task.

[0140] The present invention is described in detail above in conjunction with the embodiments of the accompanying drawings. A person skilled in the art can make various variations of the present invention according to the above description. Therefore, some details in the embodiments should not be construed as limiting the present invention, and the present invention shall be protected by the scope defined by the attached claims.

Claims

1. A method for generating image description text based on a large language model, characterized in that: The following steps are involved: Step S1: Get several images and corresponding labels, and use them to build a memory library; Step S2: construct an image description text generation model, which includes an object extraction module and a relationship extraction module, an initial description text generation module and a text polishing module; import an image I that needs text description into the object extraction module, and obtain entity information based on the memory library in step S1; Step S3: importing the image I and the entity information of the image that need text description in step S2 into the relationship extraction module to obtain the object relationship scene graph; Step S4: importing the image I that needs to be described in text in step S2, the entity information in step S2, and the entity relationship scene graph in step S3 into the initial description text generation module to obtain the initial description; Step S5: importing the image I that needs to be described in text in step S2 and the initial description text of the image imported in step S4 into a text polishing module to obtain a final polished text; Step S3 is specifically as follows: Build relevant hints based on recognized entity information Input a large language model to obtain the interaction relationship between various objects in the image I that needs to be described in text; the interaction relationship includes the relative position, action and state between the objects; construct the objects and the interaction relationship in the form of a graph structure to obtain a scene graph, wherein the nodes of the graph structure represent objects and the edges of the graph structure represent interaction relationships; Among them, relevant tips for large language model design for: ； in, Indicates interactive relationship condition related prompts, used to set generation conditions. Represents entity information. Indicates interactive task-related prompts, used to set generation requirements; Images and related tips Input into the large language model to generate an object relationship scene graph ; expressed as: ； in For large language models, is the visual encoder, A text encoder.

2. The method for generating image description text based on a large language model as claimed in claim 1, characterized in that: Step S1 is specifically as follows: Use the image encoder to obtain several images Encode to get image embedding; expressed as: ； ； ； in, represents the image encoding operation, Representing images The Sample, Representing image embedding The samples, represents the total number of images in the memory bank, represents image embedding; The image embedding and the corresponding label are stored in the form of key-value pairs. Several pairs of key-value pairs constitute the memory library, where the image embedding is the key and the corresponding label is the value, expressed as: ； ； in, Represents the memory bank, represents the i-th key, represents the i-th value, Representing image embedding The Image embedding, Indicates The images are embedded with corresponding labels.

3. The method for generating image description text based on a large language model as claimed in claim 2, characterized in that: The object extraction module in step S2 includes an image segmentation module and an image encoder; Step S2 specifically includes: inputting the image I that needs text description into the image segmentation module, and obtaining the bounding boxes and masks of all objects in the image I that needs text description; According to the pixel values ​​in the mask, the pixel values ​​of the part with objects are set to 1, and the pixel values ​​of the part without objects are set to 0. The mask is multiplied with the image I that needs to be described in text to obtain the cutout of the object in the image I, which is expressed as: ； ； in, represents the image segmentation operation, represents the mask of the i-th object, is the cutout of the i-th object, n is the number of all objects in the image, Represents the cutout of the object in image I; The cutout of each object is passed through the image encoder to obtain the corresponding image embedding , to get the total image embedding , expressed as: ； Embed the corresponding image of each object The memory bank in step S1 All images in the corresponding key are embedded in The cosine similarity calculation is expressed as: = ； in, , represents the cosine similarity function; Get the key of the most similar object , and then obtain Corresponding value ; In this way, the image in the memory bank that is most similar to the input image I that needs text description is obtained, and then the corresponding label is obtained, and this label is assigned to the image I that needs text description as the result of its object recognition, that is, the entity information ; expressed as: ； ； ； Among them, j is the subscript index value of the key of the object with the highest similarity, represents the maximum value function, Represents the label corresponding to the nth object in image I.

4. The method for generating image description text based on a large language model as claimed in claim 3, characterized in that: In the process of building the object relationship scene graph, the objects are sorted by importance, which is expressed as: Sort each object by importance according to the score to determine which objects are the main body of the image and which objects are in a secondary position; The constraints for sorting are as follows: the score of the object in the foreground is greater than that of the object in the background, and / or; the closer the object is to the center of the image I, the higher the score, and / or; the larger the area occupied by the object in the image I, the higher the score, and / or; the more times the object interacts with other objects, the higher the score; According to the above constraints, calculate the importance score of each object , expressed as: ； in, To obtain the calculation function of the Euclidean distance between the center of the object position and the center of the entire image I, To obtain the calculation function of the area of ​​the mask corresponding to the object, and is a hyperparameter, To obtain the calculation function of the degree of the object in the object relationship scene graph, Represents an object; According to importance score , sort the objects from high to low according to their scores; update by sorting to get the updated objects ; The calculation method of the object position center is specifically as follows: According to the bounding box obtained in step S2, it is assumed that the object The position coordinates of the four vertices of the corresponding bounding box are: ； The coordinates of its center position are: ； Assume that the position coordinates of the geometric center of the entire image I are ; but: 。 5. The method for generating image description text based on a large language model as claimed in claim 4, characterized in that: Step S4 is specifically as follows: Constructing a second related prompt ; Through the second relevant prompt An object relationship scene graph will be generated , updated object Input the initial description text generation module based on the large language model and combine it with the image I that needs text description to generate the initial description , expressed as: ； ； in, Indicates the initial description condition related prompts, Indicates initial description task related prompts.

6. The method for generating image description text based on a large language model as claimed in claim 5, characterized in that: Step S5 is specifically as follows: Build third related tips Guide the text polishing module based on the large language model to the initial description Further reasoning and expansion are performed, and the final polished text is obtained by combining the image I that needs text description. , expressed as: ； ； in, Indicates tips related to polishing text conditions. Indicates tips related to the text polishing task.

Images