Text correction method and device, equipment and storage medium
By generating target description text, extracting target entities, forming target questions and answers, and finally generating target declaration text to correct the original description text, the resource consumption problem of MLLM illusion phenomenon is solved, and the accuracy of MLLM output is improved without consuming a large amount of resources.
Patent Information
- Application Number
- CN202411762818.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-04-25
AI Technical Summary
When the prior art reduces the multimodal large language model (MLLM) hallucination phenomenon, resource consumption is large, and it is difficult to improve hallucination phenomenon without consuming a large amount of resources.
By obtaining the original image and the description text output from the MLLM, generating the target description text, extracting the target entity, forming the target questions and answers, generating the target declaration text, and finally correcting the original description text to improve the illusion phenomenon of MLLM.
Without consuming a large amount of resources, visual verification and target declaration text generation can improve the accuracy of MLLM output image description text and reduce hallucinations.
Smart Images
Figure CN119886246A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text correction method, apparatus, device and storage medium. Background Art
[0002] Hallucinations in multimodal large language models (MLLMs) have recently attracted increasing attention due to their potential impact on model reliability. Current research on hallucinations in MLLMs focuses primarily on evaluation / detection and mitigation. Evaluation and detection methods include training classification models to identify hallucinations or comparing output text with the true answer. Mitigation strategies aim to improve hallucinations by optimizing data collection and training processes.
[0003] In the existing technology, in order to reduce the hallucination phenomenon of MLLM, it is proposed to use custom knowledge bases or external knowledge from the Internet to enhance LLM. However, this method consumes a lot of training resources.
[0004] Therefore, how to improve the hallucination phenomenon of MLLM without consuming a lot of resources has become an urgent problem to be solved. Summary of the Invention
[0005] The embodiment of the present application provides a text correction method to correct the image description text output by a multimodal language model, thereby solving the hallucination problem of large multimodal language models.
[0006] Correspondingly, the embodiments of the present application also provide a text correction device, an electronic device, and a storage medium to ensure the implementation and application of the above method.
[0007] In order to solve the above problems, the present invention discloses a text correction method, which includes:
[0008] Obtain the original image and the original description text corresponding to the original image output by the multimodal large language model;
[0009] Generate target description text according to original description text;
[0010] Extract target entities from target description text;
[0011] Generate target questions related to the target entity based on the target description text;
[0012] Generate the target answer corresponding to the target question;
[0013] Generate target statement text based on target questions and corresponding target answers;
[0014] The original description text is modified according to the target declaration text to obtain a modified description text.
[0015] Optionally, generate target description text based on the original description text, including:
[0016] The pronouns in the original description text are replaced with the entity words corresponding to the pronouns to generate the target description text corresponding to the original image.
[0017] Optionally, extract the target entity in the target description text, including:
[0018] Segment the target description text into several target sentences;
[0019] Extract the target entity in the target sentence.
[0020] Optionally, generate target questions related to the target entity based on the target description text, including:
[0021] Generate a global entity dictionary and a global entity list based on the target entity in the target description text; the global entity dictionary contains the number of target entities in the original image;
[0022] Get a target sentence in the target description text as the current target sentence;
[0023] Get the target entity corresponding to the current target sentence from the global entity list as the current target entity;
[0024] If the current target entity exists in the global entity dictionary and the number is greater than 1, the current target entity is considered an existing entity;
[0025] Generate target questions related to existing entities based on the current target sentence.
[0026] Optionally, a global entity dictionary and a global entity list are generated based on the target entity in the target sentence corresponding to the target description text, including:
[0027] According to the target entity in the target sentence corresponding to the target description text, the target coordinates corresponding to the target entity are extracted from the original image;
[0028] According to the target coordinates, the original image is cropped to generate a target cropped image corresponding to the target entity;
[0029] According to the target coordinates, the number of target entities in the original image is obtained;
[0030] The target entity in the target sentence corresponding to the target description text, the target cropped image corresponding to the target entity, the target coordinates, and the number of target entities in the original image are stored in the global entity dictionary;
[0031] Store the target entity corresponding to the target sentence into the global entity list.
[0032] Optionally, generate a target answer corresponding to the target question based on the target question, including:
[0033] The target answers are divided into single-entity target answers and multi-entity target answers according to the number of corresponding entities.
[0034] If the target question corresponds to multiple existing entities, a multi-entity target answer corresponding to the target question is generated according to the target question, the multiple existing entities corresponding to the target question, and the original image;
[0035] If the target question corresponds to a single existing entity, then based on the single existing entity corresponding to the target question, the existing cropped image corresponding to the single existing entity is extracted from the global entity dictionary, the corresponding cropped description text is generated based on the existing cropped image, and then the corresponding single-entity target answer is generated based on the target question and the cropped description text; the existing cropped image is the target cropped image of the target entity corresponding to the single existing entity.
[0036] Optionally, generate a target statement text based on the target question and target answer, including:
[0037] Convert a target question-answer pair consisting of a target question and a target answer into a single entity statement and / or an overall statement;
[0038] Single-entity declarations are generated by converting single-entity target question-answer pairs consisting of a single-entity target answer and the corresponding target question;
[0039] The overall statement is generated by converting a multi-entity target question-answer pair consisting of a multi-entity target answer and the corresponding target question;
[0040] According to the entity corresponding to the target question, the number of target entities corresponding to the entity in the original image is extracted from the global entity dictionary to generate a count statement of the entity.
[0041] The target declaration text includes a single entity declaration, an overall declaration, and a count declaration.
[0042] The present application also discloses a text correction device, which includes:
[0043] An acquisition module is used to obtain the original image and the original description text corresponding to the original image output by the multimodal large language model;
[0044] A target description text generation module is used to generate a target description text based on the original description text;
[0045] Extraction module, used to extract target entities from target description text;
[0046] Target question generation module, used to generate target questions related to the target entity based on the target description text;
[0047] A target answer generation module is used to generate a target answer corresponding to the target question;
[0048] A target statement text generation module is used to generate a target statement text according to the target question and the corresponding target answer;
[0049] The revised description text generation module is used to revise the original description text according to the target declaration text to obtain the revised description text.
[0050] An embodiment of the present application also discloses an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, one or more text correction methods in the embodiments of the present application are implemented.
[0051] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more text correction methods in the embodiments of the present application.
[0052] Compared with the prior art, the embodiments of the present application have the following advantages:
[0053] In an embodiment of the present application, an original image and an original description text of the corresponding original image output by a multimodal large language model are obtained; a target description text is generated based on the original description text; a target entity in the target description text is extracted; a target question related to the target entity is generated based on the target description text; a target answer corresponding to the target question is generated; a target declaration text is generated based on the target question and the corresponding target answer; and the original description text is corrected based on the target declaration text to obtain a corrected description text. In an embodiment of the present application, the original description text can be visually verified based on the target entity and the target question, and then a target declaration text is generated based on the target question and the target answer without consuming a large amount of resources, and the image description text output by the multimodal language model is corrected to improve the hallucination phenomenon of the multimodal language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 This is a flowchart of the steps of an embodiment of a text correction method of the present application;
[0055] Figure 2 is a schematic diagram of an embodiment of a text correction method of the present application;
[0056] Figure 3 This is a structural block diagram of an embodiment of a text correction device of the present application. DETAILED DESCRIPTION
[0057] To make the above-mentioned purposes, features, and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and specific embodiments. It is obvious that the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.
[0058] Reference Figure 1 , is a flowchart of a text correction method embodiment of the present application, comprising the following steps:
[0059] Step 101: obtaining an original image and an original description text corresponding to the original image output by a multimodal large language model;
[0060] The original image may be an input image of a Multimodal Large Language Model (MLLM), the original image may include one or more entities, and the number of entities in the image may also be one or more. The original description text may be a descriptive text generated by the MLLM after the original image is input into the MLLM.
[0061] Step 102: Generate a target description text based on the original description text;
[0062] The target description text may be a description text generated after performing reference resolution on the original description text.
[0063] Step 103: extracting the target entity from the target description text;
[0064] The target entity can be a key entity in the target description text, corresponding to a key object in the original image. These objects are more likely to cause visual hallucinations. Key entities in the target description text can be nouns or phrases with specific meaning and importance, such as names of people, places, or organizations. They can also be the names of other physical objects present in the original image, such as cats, dogs, horses, clouds, trees, tables, and cups.
[0065] Step 104: Generate a target question related to the target entity based on the target description text;
[0066] Step 105: Generate a target answer corresponding to the target question;
[0067] Step 106: Generate a target statement text based on the target question and the corresponding target answer;
[0068] The target declaration text is a visual declaration of the original image, which can be divided into the declaration of the target object and the declaration of the object attributes. The target object declaration can include the name and quantity of the object entity. The object attribute declaration can include the location, color, and action of the object entity.
[0069] Step 107: Modify the original description text according to the target declaration text to obtain a modified description text.
[0070] The embodiment of the present application is achieved through the above steps. After the original image is input into MLLM to obtain the original description text corresponding to the original image, a target description text is generated based on the original description text, and then the target entity in the target description text is extracted. A target question involving the target entity is raised based on the target description text, and then a target answer corresponding to the target question is generated. The original description text is visually verified, which helps to verify the authenticity of the original description text. A target declaration text is generated based on the answer to the target question, and then the original description text output by the multimodal language model is corrected based on the target declaration text. Any erroneous description caused by the MLLM hallucination phenomenon is corrected to obtain a more accurately expressed revised description text, thereby improving the hallucination phenomenon of the multimodal language model.
[0071] Optionally, in an embodiment of the present application, generating a target description text according to the original description text includes:
[0072] The pronouns in the original description text are replaced with entity words corresponding to the pronouns to generate a target description text corresponding to the original image.
[0073] In this embodiment, the original description text and the corresponding reference resolution prompt can be input into the large language model together. The reference resolution prompt can be "extracting entity words from the text, and replacing pronouns with the entity words they refer to based on the context, and the other content of the text does not need to be modified." The large language model can use the Qwen large model. The Qwen large model is a general large language model that can handle general text question and answer, and has good performance on multiple benchmark tests. The large language model will replace the pronouns in the original description text with the entity words corresponding to the pronouns based on the input original description text and prompt, and generate a target description text corresponding to the original image. For example, "he" in the description text can be replaced with "Tom" to which it refers. Through the above steps, the original description text is converted into the target description text, which can improve the text comprehension ability, enhance the accuracy of information extraction, and facilitate subsequent hallucination correction.
[0074] Optionally, in an embodiment of the present application, extracting the target entity from the target description text includes:
[0075] Segmenting the target description text into a plurality of target sentences;
[0076] A target entity in the target sentence is extracted.
[0077] In this embodiment, the spacy natural language processing library can be used to load 'en_core_web_lg' and segment the generated target description text into several separate target sentences. The 'en_core_web_lg' model is a pre-trained English language model provided by Spacy and can be used for various natural language processing tasks requiring high precision and performance, such as text classification and entity recognition. All target sentences obtained by segmenting the target description text can be stored in a target sentence list. Then, each target sentence in the target sentence list is traversed, and each target sentence is added with a prompt for extracting entities. The target entities in each target sentence can be one or more. If there are multiple target entities in a target sentence, the entities in the same target sentence are concatenated using the "." symbol. Then, the concatenated entities corresponding to each target sentence can be stored as elements in the concatenated entity list. The number of elements in the concatenated entity list is the same as the number of elements in the target sentence list. The prompt for extracting entities can be "Extract entity words from the sentence and concatenate the entity words using the "." symbol." For example, a target sentence extracted by a large language model might contain three target entities: woman, umbrella, and street. These three target entities are concatenated using the dot symbol (.). This process segments the target description text into several target sentences, improving the accuracy of subsequent entity extraction and enhancing semantic integrity. Extracting the target entities, also known as key entities, within the target sentence facilitates the subsequent diagnosis of hallucinations.
[0078] Optionally, in an embodiment of the present application, generating a target question related to the target entity according to the target description text includes:
[0079] Generate a global entity dictionary and a global entity list according to the target entity in the target description text; the global entity dictionary includes the number of the target entity in the original image;
[0080] The global entity dictionary is a dictionary accessible throughout the entire scope, storing entity data in key-value pairs and used to store and manage entity information. Entities can be used as keys in the dictionary, and the attributes corresponding to other entities can be stored as values in the dictionary. The global entity list is a list accessible and modifiable throughout the entire scope, representing a collection of multiple entities.
[0081] Obtain a target sentence in the target description text as the current target sentence;
[0082] Obtaining the target entity corresponding to the current target sentence from the global entity list as the current target entity;
[0083] Under the condition that the current target entity exists in the global entity dictionary and the number of entities is greater than 1, the current target entity is regarded as an existing entity;
[0084] A target question related to the existential entity is generated according to the current target sentence.
[0085] In this embodiment, a global entity dictionary and a global entity list are generated based on the target entities in the target description text. A target sentence in the target description text is obtained as the current target sentence, and then the global entity list is traversed to obtain the target entity corresponding to the current target sentence as the current target entity. It is then determined whether the current target entity exists in the global entity dictionary and the number of entities that exist is greater than 1. The entities that meet the conditions are stored in the existing entity list. To determine whether the current target entity exists in the global entity dictionary, the current target entity can be searched in the global entity dictionary. If the key value corresponding to the current target entity exists in the global entity dictionary, it can be considered that the current target entity exists in the global entity dictionary. Selecting target entities with a number greater than 1 in the original image as existing entities is more consistent with the main objects in the original image and the original description text. Existing entity words in the same target sentence can be spliced using the "." symbol, and the spliced existing entities are then stored in the existing entity list. According to the above steps, the existing entities in all target sentences are extracted and saved in sequence, and an existing entity list containing the existing entities corresponding to all target sentences can be obtained.
[0086] To generate target questions related to existing entities based on the current target sentence, for example, a list of existing entities containing all existing entities in the target sentence can be fed into the large language model along with the current target sentence. A prompt for entity-based questions is then provided to guide the large language model in generating target question-existing entity pairs corresponding to the current target sentence. The prompt for entity-based questions can be, "Given a sentence and some entities connected by periods, you are asked to ask questions about specific entities in the sentence, so that these questions can help verify the sentence's authenticity." A target sentence can correspond to one or more target questions. A target question can be about an existing entity or about its attributes. A target question can correspond to one or more existing entities. Each target question-existing entity pair is stored as two elements in a list. After all target questions and existing entities are stored in a set, duplicate questions are deleted. Furthermore, the system can determine whether the entities in the target question-existing entity pair are existing entities extracted from the current target sentence and remove any that do not meet the criteria. This avoids inconsistent output caused by the randomness of the large language model's generation. Through the above steps, a series of questions can be asked for the entities in the target description text, that is, the main objects in the original image and the original description text, to perform hallucination diagnosis, which helps to determine whether there is a hallucination problem in the original description text.
[0087] Optionally, in an embodiment of the present application, generating a global entity dictionary and a global entity list according to the target entity in the target sentence corresponding to the target description text includes:
[0088] Extracting target coordinates corresponding to the target entity from the original image according to the target entity in the target sentence corresponding to the target description text;
[0089] The target coordinates corresponding to the target entity may be the detection frame coordinates corresponding to the target entity, which are used to identify the specific position and size of the target entity in the image. The detection frame coordinates may include the bounding box coordinates of the target, i.e., the x and y coordinates of the upper left corner and the width and height of the box.
[0090] Cropping the original image according to the target coordinates to generate a target cropped image corresponding to the target entity;
[0091] A target entity can correspond to one or more target cropped images. The number of target cropped images is the same as the number of target entities in the original image.
[0092] Obtaining the number of the target entities in the original image according to the target coordinates;
[0093] The target entity in the target sentence corresponding to the target description text, the target cropped image corresponding to the target entity, the target coordinates, and the number of the target entity in the original image are stored in the global entity dictionary;
[0094] The target entity corresponding to the target sentence is stored in the global entity list.
[0095] In this embodiment, the target entities in each target sentence corresponding to the target description text are sequentially fed into the object detection model. The target coordinates corresponding to all target entities are extracted from the original image. The original image is then cropped based on the target coordinates to generate target cropped images corresponding to the target entities. For example, a target entity in a target sentence, such as 'woman.umbrella.street', can be fed into GroundingDINO. GroundingDINO will parse the input target entity 'woman.umbrella.street', obtain the target coordinates of the target entity in the original image based on the parsing results, and then further crop the target cropped image corresponding to the target entity. When GroundingDINO parses the input, the parsed entity may be inconsistent with the input target entity. In this case, spacy can be used to correct the inconsistency and find the entity that is most similar to the input entity. Since the number of target cropped images is the same as the number of corresponding target entities in the original image, and each target cropped image corresponds to a unique target coordinate, counting the target coordinates can determine the number of corresponding target entities in the original image.
[0096] The target entity in the target sentence corresponding to the target description text, as well as the target cropped image, target coordinates, and the number of target entities in the original image corresponding to the target entity, are stored in a global entity dictionary. The target entity can be saved as a key in the global entity dictionary, and the target cropped image, target coordinates, and the number of target entities in the original image corresponding to the target entity are saved as attributes of the target entity, that is, as corresponding values in the global entity dictionary. In the global entity dictionary, the corresponding value can be obtained by searching for the key. Before storing in the global entity dictionary, the target coordinates can also be normalized to make the data more generalizable. The target entity corresponding to the target sentence is stored in a global entity list. Multiple target entities corresponding to the same target sentence are concatenated using the "." symbol. These target entities can be first divided into target lists based on the "." symbol, and then the target list corresponding to each target sentence can be stored as an element in the global entity list. For example, if the target entity corresponding to a target sentence is 'woman.umbrella.street', it can be first divided into a list [woman,umbrella,street] using ., and then the target entity list corresponding to this target sentence is stored as an element in the global entity list.
[0097] Reference Figure 2 , showing a schematic diagram of an embodiment of a text correction method of the present application. Optionally, in an embodiment of the present application, generating a target answer corresponding to the target question includes:
[0098] The target answer is divided into a single-entity target answer and a multi-entity target answer according to the number of corresponding existing entities;
[0099] Among them, the single-entity target answer is the target answer corresponding to the target question containing only one existing entity, and the multi-entity target answer is the target answer corresponding to the target question containing multiple existing entities.
[0100] If the target question corresponds to multiple existing entities, generating a multi-entity target answer corresponding to the target question according to the target question, the multiple existing entities corresponding to the target question, and the original image;
[0101] If the target question corresponds to a single existing entity, then based on the single existing entity corresponding to the target question, the existing cropped image corresponding to the single existing entity is extracted from the global entity dictionary, the corresponding cropped description text is generated based on the existing cropped image, and then the corresponding single entity target answer is generated based on the target question and the cropped description text; the existing cropped image is the target cropped image of the target entity corresponding to the single existing entity.
[0102] The blip2_t5 model is a model that feeds an image into a visual encoder, uses a visual-text converter to align the visual feature vector and the text feature vector, and completes the image text description and image question answering through a large language model. The blip2_t5 model structure includes a multimodal visual-text large language model Blip2, a query transformer Q-former (QueryingTransformer) and a large language model (LLM), and the large language model completes the final question generation and text question answering. In this embodiment, the target question and entity pairs corresponding to each target sentence can be traversed. If the current target question corresponds to multiple entities, the current question and the corresponding entity list and the original image are fed into the blip2_t5 model, and the model generates a multi-entity target answer corresponding to the target question. The multi-entity target answer or the target answer where there is a target question can be stored in cur_answers{'overall'}, and overall is used as the key value of the multi-entity target answer.
[0103] If the current target question corresponds to a single existing entity, then the global entity dictionary should be traversed sequentially based on the single existing entity corresponding to the current target question, and the global entity dictionary should be queried to see if there is a target entity with the corresponding key, and the target cropped image of the target entity with the corresponding key should be extracted as the existing cropped image corresponding to the single existing entity. When sequentially traversing the global entity dictionary to extract existing cropped images, the corresponding id can be added to the extracted existing cropped images in the order of extraction. The existing cropped image and the description prompt are sent to the blip2_t5 model to obtain the corresponding existing cropped image description text. In the case where a single existing entity corresponds to multiple existing cropped images, it is necessary to loop through each existing cropped image and send them to the blip2_t5 model to obtain the cropped image description text corresponding to each existing cropped image. For example Figure 2As shown, after feeding a cropped image into the blip2_t5 model, the model's image encoder identifies image features in the cropped image, which can be corresponding entities. The query transformer (Q-Former) then selects the most meaningful information from the extracted image features and, based on the input description prompt, presents this information in a format understandable to the language model. Finally, the large language model generates and outputs a description of the cropped image based on these transformed prompts. The current target question and the corresponding cropped image description are fed into a pre-trained natural language encoder (BERT) and a downstream task model (SQuAD) (Stanford Question Answering Dataset) to extract the existing answer from the cropped image description and generate the corresponding single-entity target answer. The description prompt can be "Please describe this image." BERT is a natural language encoder pre-trained on a large amount of text, resulting in excellent text information extraction capabilities. BERT can be used for various downstream tasks. SQuAD, one of BERT's downstream tasks, can identify the range of answers in text. After generating the corresponding single-entity target answer, the target question and single-entity target answer tuple can be placed in the cur_answers[entity][idx] list, with the entity term of the single entity serving as the key for the single-entity target answer. If a corresponding answer does not exist, no key is assigned. "entity" represents the entity term for the single entity. "idx" represents the xth extracted image, which serves as both the ID of the cropped image and the ID in the list corresponding to the single entity term. The cropped image ID can be used as the parent list for the target question and single-entity target answer pair, and the entity term of the single entity as the parent list for the cropped image ID. The "entity" and "overall" keys belong to the same level. This hierarchy allows us to construct all question-answer pairs for a target sentence. If multiple target questions for a target sentence correspond to the same entity, the cropped image ID of the entity's next-level entity can correspond to multiple target questions and answer pairs. This means that the cropped image corresponding to the cropped image ID will be posed to multiple target questions and generate corresponding target answers. Based on the generated target answer, the original description text output by MLLM can be visually verified to confirm whether the content of the original description text generated by the model is consistent with the content of the original image input, thereby correcting the hallucination phenomenon generated by the MLLM model.
[0104] Optionally, in an embodiment of the present application, generating a target statement text according to the target question and the target answer includes:
[0105] Converting a target question-answer pair consisting of the target question and the target answer into a single entity statement and / or an overall statement;
[0106] The single entity statement is generated by converting a single entity target question-answer pair consisting of a single entity target answer and the corresponding target question;
[0107] The overall statement is generated by converting a multi-entity target question-answer pair consisting of a multi-entity target answer and a corresponding target question;
[0108] According to the existent entity corresponding to the target question, extracting the number of target entities corresponding to the existent entity in the original image from a global entity dictionary, and generating a count statement of the existent entity;
[0109] The target declaration text includes a single entity declaration, a general declaration, and a count declaration.
[0110] In this embodiment, all target question-answer pairs can be fed into the zerofec-qa2claim-t5-base model to convert the question-answer pairs into a single-sentence statement. Statements corresponding to multi-entity question-answer pairs can be grouped under the overall key, while statements corresponding to single-entity question-answer pairs can use the corresponding single entity as its key. zerofec-qa2claim-t5-base is a faithful and interpretable fact-correction framework based on the T5 model that can convert single question-answer pairs into declarative statements. Furthermore, based on the entity corresponding to the target question-answer pair, the global entity dictionary can be queried for the number of target entities with corresponding key values in the original image to generate a count statement for the entity. For example, a count statement could be "There are n entities in this image," where n refers to the number of entities in the original image. Single-entity statements and overall statements are statements at the target object level, while count statements are statements at the object attribute level. Furthermore, based on the entity corresponding to the target question-answer pair, the global entity dictionary can be queried for corresponding other entity attributes to generate other attribute statements.
[0111] Optionally, in an embodiment of the present application, the modifying the original description text according to the target declaration text to obtain the modified description text may include:
[0112] All target declaration text, including single-entity declarations, aggregate declarations, and count declarations, is fed into the large language model. A correction description prompt is added, and the large language model outputs the corrected description. The correction description prompt can provide examples, listing all declarations, providing image text descriptions, and corrected text descriptions. Through multiple examples, the large language model learns the patterns and automatically combines the declarations input into a corrected description, outputting the corrected description text. These steps can correct the image description text output by the multimodal language model, improving the hallucination phenomenon of the multimodal language model.
[0113] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0114] On the basis of the above embodiments, this embodiment further provides a text correction device, which is applied to electronic devices such as terminal devices and servers.
[0115] Reference Figure 3 , shows a structural block diagram of an embodiment of a text correction device of the present application, which may specifically include the following modules:
[0116] An acquisition module 301 is configured to acquire an original image and an original description text corresponding to the original image output by a multimodal large language model;
[0117] A target description text generating module 302 is configured to generate a target description text based on the original description text;
[0118] Extraction module 303, used to extract target entities from the target description text;
[0119] A target question generating module 304 is configured to generate a target question related to the target entity according to the target description text;
[0120] A target answer generating module 305 is configured to generate a target answer corresponding to the target question;
[0121] A target statement text generating module 306 is configured to generate a target statement text according to the target question and the corresponding target answer;
[0122] The revised description text generating module 307 is configured to revise the original description text according to the target declaration text to obtain a revised description text.
[0123] Optionally, the target description text generation module is further used to:
[0124] The pronouns in the original description text are replaced with entity words corresponding to the pronouns to generate a target description text corresponding to the original image.
[0125] Optionally, the extraction module includes:
[0126] A segmentation submodule, configured to segment the target description text into a plurality of target sentences;
[0127] The extraction submodule is used to extract the target entity in the target sentence.
[0128] Optionally, the target question generating module includes:
[0129] A first generating submodule is configured to generate a global entity dictionary and a global entity list according to the target entity in the target description text; the global entity dictionary includes the number of the target entity in the original image;
[0130] A current target sentence acquisition submodule, configured to acquire a target sentence in the target description text as the current target sentence;
[0131] A current target entity acquisition submodule is used to acquire the target entity corresponding to the current target sentence from the global entity list as the current target entity;
[0132] An existing entity acquisition submodule, configured to treat the current target entity as an existing entity if the current target entity exists in the global entity dictionary and the number of entities is greater than 1;
[0133] The target question generation submodule is used to generate a target question related to the existential entity based on the current target sentence.
[0134] Optionally, the first generation submodule includes:
[0135] A target coordinate extraction unit, configured to extract target coordinates corresponding to the target entity from the original image according to the target entity in the target sentence corresponding to the target description text;
[0136] a target cropped image generating unit, configured to crop the original image according to the target coordinates to generate a target cropped image corresponding to the target entity;
[0137] a target entity number counting unit, configured to obtain the number of the target entity in the original image according to the target coordinates;
[0138] A global entity dictionary generating unit, configured to store a target entity in a target sentence corresponding to the target description text, a target cropped image corresponding to the target entity, target coordinates, and the number of the target entity in the original image into the global entity dictionary;
[0139] The global entity list generating unit is configured to store the target entity corresponding to the target sentence into the global entity list.
[0140] Optionally, the target answer generation module is further configured to:
[0141] The target answer is divided into a single-entity target answer and a multi-entity target answer according to the number of corresponding existing entities;
[0142] If the target question corresponds to multiple existing entities, generating a multi-entity target answer corresponding to the target question according to the target question, the multiple existing entities corresponding to the target question, and the original image;
[0143] If the target question corresponds to a single existing entity, then based on the single existing entity corresponding to the target question, the existing cropped image corresponding to the single existing entity is extracted from the global entity dictionary, the corresponding cropped description text is generated based on the existing cropped image, and then the corresponding single entity target answer is generated based on the target question and the cropped description text; the existing cropped image is the target cropped image of the target entity corresponding to the single existing entity.
[0144] Optionally, the target declaration text generation module includes:
[0145] The second generation submodule is used to convert the target question-answer pair consisting of the target question and the target answer into a single entity statement and / or an overall statement; the single entity statement is generated by converting the single entity target question-answer pair consisting of the single entity target answer and the corresponding target question; the overall statement is generated by converting the multi-entity target question-answer pair consisting of the multi-entity target answer and the corresponding target question;
[0146] The third generation submodule is used to extract the number of target entities corresponding to the existing entity in the original image from the global entity dictionary according to the existing entity corresponding to the target problem, and generate a counting statement of the existing entity; the target statement text includes a single entity statement, an overall statement and a counting statement.
[0147] The embodiment of the present application provides a text correction device, which can input the original image into MLLM to obtain the original description text corresponding to the original image, generate a target description text based on the original description text, and then extract the target entity in the target description text, raise a target question involving the target entity based on the target description text, and then generate a target answer corresponding to the target question, performing visual knowledge verification on the original description text, which helps to verify the authenticity of the original description text. According to the answer to the target question, a target statement text is generated, and then the original description text output by the multimodal language model is corrected according to the target statement text, and any erroneous descriptions that may exist due to the MLLM hallucination phenomenon are corrected to obtain a more accurately expressed revised description text, thereby improving the hallucination phenomenon of the multimodal language model.
[0148] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0149] An embodiment of the present application also provides a text correction device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above data playback method is implemented.
[0150] An embodiment of the present application further provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.
[0151] The present application provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause an electronic device to perform one or more text correction methods as described in the above embodiments. In the present application, the electronic device includes various types of devices such as terminal devices and servers (clusters).
[0152] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters).
[0153] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0154] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable text correction terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable text correction terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0155] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable text correction terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, the instruction device being implemented in the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0156] These computer program instructions can also be loaded onto a computer or other programmable text correction terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0157] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0158] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.
[0159] The above is a detailed introduction to a text correction method and device, an electronic device and a storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A text correction method, characterized in that: include: Obtaining an original image and an original description text corresponding to the original image output by a multimodal large language model; Generate a target description text according to the original description text; Extracting target entities from the target description text; Generate a target question related to the target entity according to the target description text; Generate a target answer corresponding to the target question; Generate a target statement text according to the target question and the corresponding target answer; The original description text is modified according to the target declaration text to obtain a modified description text.
2. The method according to claim 1, characterized in that The step of generating a target description text according to the original description text comprises: The pronouns in the original description text are replaced with entity words corresponding to the pronouns to generate a target description text corresponding to the original image.
3. The method according to claim 1, characterized in that: The extracting the target entity in the target description text comprises: Segmenting the target description text into a plurality of target sentences; A target entity in the target sentence is extracted.
4. The method according to claim 1, characterized in that: The step of generating a target question related to the target entity according to the target description text includes: Generate a global entity dictionary and a global entity list according to the target entity in the target description text; the global entity dictionary contains the number of the target entity in the original image; Acquire a target sentence in the target description text as the current target sentence; Acquire a target entity corresponding to the current target sentence from the global entity list as the current target entity; Under the condition that the current target entity exists in the global entity dictionary and the number of entities is greater than 1, the current target entity is regarded as an existing entity; A target question related to the existential entity is generated according to the current target sentence.
5. The method according to claim 4, characterized in that The generating a global entity dictionary and a global entity list according to the target entity in the target sentence corresponding to the target description text includes: According to the target entity in the target sentence corresponding to the target description text, extracting the target coordinates corresponding to the target entity from the original image; According to the target coordinates, cropping the original image to generate a target cropped image corresponding to the target entity; According to the target coordinates, obtaining the number of the target entities in the original image; The target entity in the target sentence corresponding to the target description text, the target cropped image corresponding to the target entity, the target coordinates and the number of the target entity in the original image are stored in the global entity dictionary; The target entity corresponding to the target sentence is stored in the global entity list.
6. The method according to claim 5, characterized in that Generating a target answer corresponding to the target question includes: The target answer is divided into a single-entity target answer and a multi-entity target answer according to the number of corresponding existing entities; If the target question corresponds to a plurality of the existing entities, generating a multi-entity target answer corresponding to the target question according to the target question, the plurality of existing entities corresponding to the target question and the original image; If the target question corresponds to a single existing entity, then based on the single existing entity corresponding to the target question, an existing cropped image corresponding to the single existing entity is extracted from the global entity dictionary, a corresponding cropped description text is generated based on the existing cropped image, and then a corresponding single entity target answer is generated based on the target question and the cropped description text; the existing cropped image is a target cropped image of the target entity corresponding to the single existing entity.
7. The method according to claim 1, characterized in that The step of generating a target statement text according to the target question and the target answer includes: Converting a target question-answer pair consisting of the target question and the target answer into a single entity statement and / or an overall statement; The single entity statement is generated by converting a single entity target question-answer pair consisting of a single entity target answer and the corresponding target question; The overall statement is generated by converting a multi-entity target question-answer pair consisting of a multi-entity target answer and a corresponding target question; According to the existent entity corresponding to the target question, the number of the target entity corresponding to the existent entity in the original image is extracted from the global entity dictionary to generate a count statement of the existent entity; The target declaration text includes a single entity declaration, a total declaration, and a count declaration.
8. A text correction device, characterized in that: The device comprises: An acquisition module, used to acquire an original image and an original description text corresponding to the original image output by a multimodal large language model; A target description text generation module, used to generate a target description text according to the original description text; An extraction module, used for extracting target entities in the target description text; A target question generating module, used for generating a target question related to the target entity according to the target description text; A target answer generation module, used to generate a target answer corresponding to the target question; A target statement text generation module, used to generate a target statement text according to the target question and the corresponding target answer; The revised description text generating module is used to revise the original description text according to the target declaration text to obtain the revised description text.
9. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the text correction method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the text correction method according to any one of claims 1 to 7 is implemented.