Multi-mode large model automatic error correction method and system

Abstract

The invention discloses a multi-mode large model automatic error correction method and system. The method comprises the steps that information to be input is converted into feature vectors, after the feature vectors are spliced into placeholder feature vectors, the feature vectors are input into a multi-mode large model, and the multi-mode large model is configured to output corresponding reply vectors and placeholder vectors according to the input vectors; further decoding the placeholder vector, and if the result is correct, directly outputting a reply text corresponding to the reply vector; and if the result is wrong, replacing the placeholder feature vector with the placeholder vector, and inputting the placeholder vector into the multi-mode large model again until the decoding result is correct or reaches the preset error correction frequency. According to the automatic error correction method provided by the invention, multiple modal capabilities can be dynamically called and are matched with one another to carry out error correction, so that the overall performance and the output quality of the multi-modal large model are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically to a multi-modal large model automatic error correction method and system. Background Art

[0002] In modern technology applications, especially in the field of artificial intelligence, multimodal large models have become an important development trend. These models combine multiple data types such as text, images, and voice to provide more comprehensive and accurate service capabilities. However, although multimodal large models have made significant progress in data processing and cognitive functions, they still face some challenges in ensuring the accuracy and reliability of output results.

[0003] Currently, a major problem with large multimodal models is the lack of coordination between the modalities, such as insufficient information fusion, which leads to unsatisfactory processing results. For example, when the model processes text and image data at the same time, if the two types of information cannot be accurately fused, it may lead to misunderstandings or wrong conclusions. In addition, although existing large models can adopt certain error correction measures, these measures are often limited to a single modality and are difficult to play a role in cross-modal error identification and correction. For example, text error correction methods based on language models may not be able to correct text generation errors caused by errors in image data parsing.

[0004] The error correction process of most existing systems also lacks flexibility and adaptability, and cannot dynamically adjust the error correction strategy according to different types of errors. This limits the practicality and reliability of the model in complex application scenarios.

[0005] Therefore, how to overcome the above problems and enable the model to achieve cross-modal automatic error correction is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides a multi-modal large model automatic error correction method and system, which aims to integrate the data processing capabilities of different modalities and significantly improve the accuracy and reliability of the system output through precise information fusion and efficient error correction mechanism.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] On the one hand, the present application discloses a multi-modal large model automatic error correction method, comprising:

[0009] The information to be input is converted into a feature vector, a placeholder feature vector is concatenated, and then the feature vector is input into the multimodal large model, wherein the multimodal large model is configured to output a corresponding response vector and a placeholder vector according to the input vector; the placeholder vector is a fusion vector including the input information and the response information of the multimodal large model;

[0010] The placeholder vector is decoded. If the result is correct, the reply text corresponding to the reply vector is directly output; if the result is wrong, the placeholder vector is replaced by the placeholder feature vector and re-input into the multimodal large model until the decoding result is correct or the preset number of error corrections is reached.

[0011] Furthermore, the input information includes text or pictures;

[0012] The text is converted into a text feature vector by a word segmenter;

[0013] The picture is converted into an image feature vector through an image encoder.

[0014] Preferably, the input information also includes audio data, which is converted into text by an ASR model and then further converted into a text feature vector by a word segmenter.

[0015] Furthermore, the word segmenter includes a character table corresponding to the placeholder, which is used to map the placeholder into a learnable feature vector.

[0016] Furthermore, the picture is converted into an image feature vector by an image encoder, further comprising:

[0017] The conversion result of the image encoder is mapped to a common embedding space through a projection layer, so that the feature vectors corresponding to the image and the descriptive text are combined in the same dimensional space to obtain the image feature vector.

[0018] Furthermore, the reply vector is converted into a probability vector through a softmax function, and the probability vector is used as the final reply vector, and the placeholder vector remains unchanged.

[0019] Further, the placeholder vector is decoded by a decoder, and the output result is correct or wrong.

[0020] Furthermore, the decoder is a Transformer network model or a diffusion model.

[0021] On the other hand, the present application discloses a multimodal large model automatic error correction system, which applies the multimodal large model automatic error correction method as described above, and the system includes an encoding module, a multimodal large model, a decoding module and a judgment module; wherein,

[0022] The encoding module is used to convert the input information and placeholders into feature vectors, concatenate them, and input them into the multimodal large model;

[0023] A multimodal large model, used for outputting a corresponding reply vector and a placeholder vector according to an input vector;

[0024] A decoding module, used for decoding the reply vector to obtain the reply text, and decoding the placeholder vector to obtain a correct or incorrect judgment result;

[0025] The judgment module is used to directly output the reply text when the judgment result is correct; when the judgment result is wrong, it replaces the placeholder feature vector obtained by the encoding module with the placeholder vector and re-inputs it into the multimodal large model until the decoding result is correct.

[0026] Furthermore, the encoding module includes a first word segmenter and an image encoder, the first word segmenter is used to convert the text and information placeholders into corresponding feature vectors, and the image encoder is used to convert the image into an image feature vector;

[0027] The decoding module includes a second word segmenter and a decoder, the second analyzer is used to convert the reply vector into a reply text, and the decoder is used to decode the placeholder vector into a judgment result.

[0028] Furthermore, training data is constructed to train the multimodal large model automatic error correction system, and the training steps include:

[0029] Construct question-answer pairs with deterministic answers and conduct preliminary training of the multimodal large model automatic error correction system.

[0030] Construct question-answer pairs without deterministic answers, and retrain the preliminarily trained multimodal large model automatic error correction system based on question-answer pairs with deterministic answers and question-answer pairs without deterministic answers.

[0031] It can be seen from the above technical solutions that the present invention discloses a method and system for automatic error correction of a multimodal large model. The main difference between the present invention and the prior art is that it can dynamically call multiple modal capabilities and cooperate with each other for error correction, thereby effectively improving the overall performance and output quality of the multimodal large model.

[0032] In general, the invention not only solves the deficiencies in the prior art, but also provides an efficient multimodal large model method that can dynamically adjust the error correction strategy, significantly improving the accuracy and reliability of multimodal data processing. This is of great significance for promoting the advancement and application of multimodal large model technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0034] Figure 1 A schematic diagram of the decoding process provided by the present invention;

[0035] Figure 2 A schematic diagram of the multimodal large model workflow provided by the present invention;

[0036] Figure 3 A schematic diagram showing the output of a multi-modal large model provided by the present invention;

[0037] Figure 4 A schematic diagram of the decoding process provided by the present invention;

[0038] Figure 5 This is a schematic diagram of the structure of the multi-modal large model automatic error correction system provided by the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In traditional error correction systems, the error correction mechanism mainly relies on external input text. Specifically, this system receives a piece of text and then uses a large model to determine whether the text contains errors. The key here is that the input text is not generated by the large model itself, so the large model cannot directly evaluate whether the text it generates is correct.

[0041] To solve this problem, this application proposes a new method of automatic error correction. In the process of generating text by the large model, not only the text content is generated, but also a special "character" or vector is generated at the same time. This "character" incorporates all relevant information, including the user's input and the model's own output. Its main function is to evaluate whether the text previously generated by the large model contains any erroneous information. If errors are detected, this "character" will be used to automatically correct these errors, thereby correcting the errors generated by the large model itself.

[0042] Specifically, the implementation process of the present application is described through the following embodiments.

[0043] Embodiment 1

[0044] This embodiment discloses a multi-modal large model automatic error correction method, including:

[0045] The information to be input is converted into a feature vector, a placeholder feature vector is concatenated, and then the feature vector is input into the multimodal large model, wherein the multimodal large model is configured to output a corresponding response vector and a placeholder vector according to the input vector; the placeholder vector is a fusion vector including the input information and the response information of the multimodal large model;

[0046] The placeholder vector is decoded. If the result is correct, the reply text corresponding to the reply vector is directly output; if the result is wrong, the placeholder vector is replaced by the placeholder feature vector and re-input into the multimodal large model until the decoding result is correct or the preset number of error corrections is reached.

[0047] The core advantage of this application method lies in its adaptability. By generating and applying this special "character" in real time, the large model can dynamically identify and correct the mistakes it may make in the text generation process. This not only improves the accuracy of text generation, but also enhances the reliability and practicality of the model.

[0048] In an exemplary embodiment, the method comprises the following steps:

[0049] S1, convert the input information into a feature vector and concatenate it with a placeholder feature vector;

[0050] This embodiment takes a multimodal large model that supports image and text input as an example. Take a piece of text as input, optionally input an image, and attach an information placeholder. In this application, the information placeholder is a special label, and the text representation is " <info>”;

[0051] Specifically, a piece of text comes from the user's input, usually a user's instruction, indicating to complete a task or seek some help; the input image also comes from the user, used to complete image-based tasks or questions and answers, and the image resolution is not limited; in addition, the additional input information placeholder <info>, used to subsequently capture and integrate all information input by the user.

[0052] Furthermore, in this embodiment, the above-mentioned various forms of input data, such as text, images, etc., are converted into feature vectors to convert the original data into a form that can be "understood" by the computer model.

[0053] In one embodiment, the encoding module is used to encode the above input information to lay the foundation for the subsequent training and application of the deep learning model. In this process, according to different data types (modalities), corresponding encoding strategies are adopted; in this embodiment, the implementation method of the encoding module is as follows: Figure 1 As shown,

[0054] The text is encoded by a tokenizer, which includes:

[0055] First, the text is segmented, and then the character table recorded in the word segmenter is used to convert the segmented units into numerical form through table lookup, usually vectors in high-dimensional space. This process is called word embedding. The vector is designed so that semantically or grammatically similar words are close in the vector space.

[0056] In this embodiment, a special placeholder is added to the character table of the word segmenter, marked as <info>; A character table is usually maintained in the word segmenter to map one or more characters to an id, and each id corresponds to a different vector. <info>Add a special id, for example, set it to -100, and add a learnable vector for this id, independent of the vector in the previous character table. For example, the id of "I" is 1, the id of "is" is 2, and the id of "who" is 3, then "Who am I" <info>" will be mapped to a list "[1,2,3,-100]", and then the corresponding vector is obtained from the character table, and finally represented as a vector list.

[0057] In the present invention, the placeholder <info>It is designed to remind the multimodal large model to capture and integrate various information of user input in the process of generating responses. That is, when the user's input data passes through the word segmenter, <info>This placeholder will also be automatically recognized and converted into the corresponding vector form.

[0058] The image is encoded using an image encoder, such as the visual encoder of the multimodal model CLIP, which is trained by contrastive learning on a large number of image and text pairs, thereby learning to map images and descriptive text into a common vector space.

[0059] In the design of encoding modules, a key challenge in processing data of different modalities is that their feature representations often have different dimensions and characteristics. Due to the essential differences of these modalities, it may not be feasible or meaningful to directly merge or use their feature vectors. To solve this problem, this application introduces the concept of projection layer, such as Figure 1 As shown in Figure 1, its function is to map the feature vectors generated by each modality to a common embedding space, that is, feature vectors with the same dimension. In this way, data from different modalities can be effectively compared and combined in the same dimensional space.

[0060] In an exemplary embodiment, the projection layer is implemented by using a simple neural network structure, such as a linear layer (fully connected layer) followed by a nonlinear activation function, such as ReLU or tanh. Although this structure is simple, it is very effective and can learn the transformation that maps the original feature vector to the target space.

[0061] In practical applications, it is necessary to combine the implementation method of the feature fusion module to determine the projection layer structure and parameters that are most suitable for specific tasks and data. There is no limitation here. Each different encoding method corresponds to a different projection layer.

[0062] To further optimize the above technical solution, the input information may also include audio data, that is, after the audio data is converted into text using the ASR model, it is further converted into a corresponding feature vector using a word segmenter.

[0063] In summary, after the input information of each modality is encoded and converted, it is concatenated with the placeholder feature vector and then input into the multimodal large model.

[0064] S2. The multimodal model of the present application is configured to output a corresponding reply vector and a placeholder vector according to the input vector; the placeholder vector is a fusion vector containing the multimodal model input information and reply information; that is, in this embodiment, the multimodal model will generate a series of vectors (tokens), which contain a reply to the user input and a placeholder for the information. <info>Reserved output vector.

[0065] In this embodiment, the multimodal big model is a big model that can understand information in different modalities. Its function is to understand the user's input and generate corresponding replies. At the same time, it can generate more accurate information based on the feedback of subsequent modules. In one embodiment, one implementation of the big model is to use an open source multimodal big model, such as Tsinghua's ChatGLM series of big models; it can also be an agent-based big language model, that is, it only supports text input, but can convert other modal information into text through other methods. For example, picture input can be replaced by a text describing the picture in detail.

[0066] In this application, the workflow of the multimodal large model is as follows Figure 2 Generally, multimodal large models are autoregressive models based on the Transformer structure, so the output of the multimodal large model is not fixed length. In this embodiment, the output of the multimodal large model is specially processed. <info>Placeholders, which are additional characters added to the input <info>The placeholder can guide the big model to output a special vector in addition to the answer to the user. This special vector is called the "fusion information vector", and the big model will learn this during the training process.

[0067] Specifically, the last vector in the output list is taken as a pair <info>The other vectors in the list are considered to be direct answers to the user's query (e.g. Figure 3 As shown); in a preferred embodiment, each reply vector is converted into a probability vector through a softmax function, and the vector length is the same as the number of characters in the character table. The probability vector is used as the final output reply vector, and the last special vector does not pass through the softmax function and remains unchanged.

[0068] Due to the self-attention and mutual-attention mechanisms in the Transformer architecture, as well as the autoregressive mechanism of the multimodal large model, the last vector output by the multimodal large model can be regarded as a vector that integrates the information of all previous vectors, so it can also be called a "fused information vector".

[0069] It should be noted that the multimodal large model workflow defined in this application requires corresponding training data to fine-tune the entire system before it can be implemented. During fine-tuning, the parameters of the large model and the decoding module need to be adjusted at the same time.

[0070] S3. Further, the reply vector and the placeholder vector are decoded to obtain the reply text and the authenticity (credibility) of the reply text. The result is expressed as correct or wrong, which is used to characterize the accuracy of the multimodal large model reply.

[0071] One implementation of the decoding process is as follows Figure 4 As shown, the reply vector will be converted back into text by the word segmenter, which is the same as the word segmenter in the encoding module, and the operation performed can be regarded as a reverse operation. The fused information vector will be converted into two outputs, correct or incorrect, by the decoder, which can be regarded as a classification or as output text; in this embodiment, the judgment of "correct" or "incorrect" is the judgment of the multimodal large model's own answer, that is, the large model outputs information for judging right and wrong (fused information vector) while outputting the reply, which is actually an automatic error correction mechanism.

[0072] The fused information vector contains the information of the response generated by the multimodal large model. Since the autoregressive model predicts the next vector based on all input and output vectors, the fused information vector also implicitly contains all the user's input information (text and images). Therefore, the decoder can output a "correct" or "wrong" judgment based on this fused information vector. In one embodiment, the decoder is a decoder (Decoder) similar to the Transformer structure, or a decoder of other structures, such as a diffusion model, etc., which can be replaced with a structure that adapts to the user scenario and has better effects according to the actual scenario.

[0073] In this embodiment, if the result is correct, the decoded reply text is directly output; if the result is wrong, the decoded placeholder vector replaces the placeholder feature vector and is re-input into the multimodal large model until the decoding result is correct or the preset number of error corrections is reached, thereby ensuring that the reply text output by the large model is correct or tends to be correct.

[0074] The re-input multimodal model will generate new responses based on the updated information. Due to the introduction of the fused information vector, the responses generated this time will be different from the previous responses. This difference is due to the fact that the fused information vector provides additional, targeted information, helping the model to more accurately understand the user's needs and context, thereby correcting any previous errors.

[0075] The automatic error correction mechanism disclosed in this application not only increases the flexibility of the multimodal large model in processing inputs, but also improves its ability to adapt to different scenarios. In practical applications, this means that the model can more accurately capture the user's intentions and information needs, especially when dealing with complex or ambiguous inputs. By continuously optimizing and adjusting the fused information vector, the performance of the multimodal large model can be continuously improved, enabling it to provide more accurate and reliable outputs in various application scenarios.

[0076] To further optimize the above technical solution, this embodiment provides a fine-tuning solution for a multimodal large model workflow, the steps of which include:

[0077] Construct question-answer pairs with deterministic answers and conduct preliminary training of the multimodal large model automatic error correction system.

[0078] Construct question-answer pairs without deterministic answers, and retrain the preliminarily trained multimodal large model automatic error correction system based on question-answer pairs with deterministic answers and question-answer pairs without deterministic answers.

[0079] In this embodiment, when constructing question-answer pairs, manual construction or commercial large-scale models available on the market are used to construct them and then manual review is performed to check for possible errors. The key to constructing training data is to define which answers are correct and which answers are incorrect.

[0080] Preferably, constructing training data can be divided into two stages:

[0081] The question-answer pairs in the first stage all have deterministic answers, rather than open-ended questions. For example, if you input a picture of a man in blue clothes and ask whether the man in the picture is wearing blue clothes, the correct answer can only be "yes." All other answers are wrong answers.

[0082] The second stage of question-answer pairs includes both deterministic and indeterminate answers. The indeterminate answer problem here means that the same question can have multiple semantically similar answers. For example, when asking "What color is water?", the answer can be "Water is transparent" or "Water is colorless". You can list some correct answers and then use semantic similarity to determine whether the answer of the multimodal model is correct.

[0083] During training, the first stage of data training is used to enable the large model to understand the task of error correction and understand <info>The significance of placeholders and the significance of fused information vectors are used to achieve preliminary error correction, and then the adaptability and generalization of the multimodal large model are further improved through the data of the second stage to improve the accuracy of error correction.

[0084] Embodiment 2

[0085] This embodiment also discloses a multimodal large model automatic error correction system, which applies the multimodal large model automatic error correction method as described above, and the system includes an encoding module, a multimodal large model, a decoding module and a judgment module; Figure 5 As shown,

[0086] The encoding module is used to convert the input information and placeholders into feature vectors, concatenate them, and input them into the multimodal large model;

[0087] A multimodal large model, used for outputting a corresponding reply vector and a placeholder vector according to an input vector;

[0088] A decoding module, used for decoding the reply vector to obtain the reply text, and decoding the placeholder vector to obtain a correct or incorrect judgment result;

[0089] The judgment module is used to directly output the reply text when the judgment result is correct; when the judgment result is wrong, it replaces the placeholder feature vector obtained by the encoding module with the placeholder vector and re-inputs it into the multimodal large model until the decoding result is correct.

[0090] Furthermore, the encoding module includes a first word segmenter and an image encoder, the first word segmenter is used to convert the text and information placeholders into corresponding feature vectors, and the image encoder is used to convert the image into an image feature vector;

[0091] The decoding module includes a second word segmenter and a decoder, the second analyzer is used to convert the reply vector into a reply text, and the decoder is used to decode the placeholder vector into a judgment result.

[0092] The details are consistent with those disclosed in Example 1 and will not be repeated here.

[0093] In the multimodal large model automatic error correction mechanism provided by the present invention, the multimodal large model can not only generate responses, but also self-adjust and optimize according to the feedback of subsequent modules; this adaptive ability makes the results generated by the model more credible; at the same time, it becomes more accurate with the passage of time and the accumulation of data, and can better meet user needs and handle complex situations.

[0094] In addition, the model in this method supports multi-modal input and multi-modal interaction; it can be used not only in the two modes of picture and text, but also in the same way in video and audio, thus further enhancing the interactive experience. Overall, this automatic error correction mechanism marks an important step forward for large model technology towards self-supervision and self-improvement.

[0095] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0096] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.< / info> < / info> < / info> < / info> < / info> < / info> < / info> < / info> < / info> < / info> < / info> < / info>

Claims

1. A multi-modal large model automatic error correction method, characterized in that: The information to be input is converted into a feature vector, a placeholder feature vector is concatenated, and then the feature vector is input into the multimodal large model, wherein the multimodal large model is configured to output a corresponding response vector and a placeholder vector according to the input vector; the placeholder vector is a fusion vector including the input information and the response information of the multimodal large model; The placeholder vector is decoded. If the result is correct, the reply text corresponding to the reply vector is directly output; if the result is wrong, the placeholder vector is replaced by the placeholder feature vector and re-input into the multimodal large model until the decoding result is correct or the preset number of error corrections is reached.

2. A multi-modal large model automatic error correction method according to claim 1, characterized in that: The input information includes text or pictures; The text is converted into a text feature vector by a word segmenter; The picture is converted into an image feature vector through an image encoder.

3. A multi-modal large model automatic error correction method according to claim 2, characterized in that: The word segmenter includes a character table corresponding to the placeholder, which is used to map the placeholder into a learnable feature vector.

4. A multi-modal large model automatic error correction method according to claim 2, characterized in that: The picture is converted into an image feature vector by an image encoder, and further includes: The conversion result of the image encoder is mapped to a common embedding space through a projection layer, so that the feature vectors corresponding to the image and the descriptive text are combined in the same dimensional space to obtain the image feature vector.

5. The multi-modal large model automatic error correction method according to claim 1, characterized in that: The reply vector is converted into a probability vector through a softmax function, and the probability vector is used as the final reply vector, while the placeholder vector remains unchanged.

6. The multi-modal large model automatic error correction method according to claim 1, characterized in that: The placeholder vector is decoded by the decoder and the output is either true or false.

7. A multi-modal large model automatic error correction method according to claim 6, characterized in that: The decoder is a Transformer network model or a diffusion model.

8. A multi-modal large model automatic error correction system, characterized in that: The multimodal large model automatic error correction method according to any one of claims 1 to 7 is applied, comprising an encoding module, a multimodal large model, a decoding module and a judgment module; The encoding module is used to convert the input information and placeholders into feature vectors, concatenate them, and input them into the multimodal large model; A multimodal large model, used for outputting a corresponding reply vector and a placeholder vector according to an input vector; A decoding module, used for decoding the reply vector to obtain the reply text, and decoding the placeholder vector to obtain a correct or incorrect judgment result; The judgment module is used to directly output the reply text when the judgment result is correct; when the judgment result is wrong, it replaces the placeholder feature vector obtained by the encoding module with the placeholder vector and re-inputs it into the multimodal large model until the decoding result is correct.

9. A multi-modal large model automatic error correction system according to claim 8, characterized in that: The encoding module includes a first word segmenter and an image encoder, wherein the first word segmenter is used to convert text and information placeholders into corresponding feature vectors, and the image encoder is used to convert images into image feature vectors; The decoding module includes a second word segmenter and a decoder, the second analyzer is used to convert the reply vector into a reply text, and the decoder is used to decode the placeholder vector into a judgment result.

10. A multi-modal large model automatic error correction system according to claim 8, characterized in that: Construct training data to train the multimodal large model automatic error correction system, and the training steps include: Construct question-answer pairs with deterministic answers and conduct preliminary training of the multimodal large model automatic error correction system. Construct question-answer pairs without deterministic answers, and retrain the preliminarily trained multimodal large model automatic error correction system based on question-answer pairs with deterministic answers and question-answer pairs without deterministic answers.

Images