Visual language large model training method, system and rich text image question answering method for rich text image question answering

Abstract

The present invention discloses a method for training a large visual language model for rich-text image question answering, comprising determining a large visual language model and obtaining a training data set; selecting a number of rich-text image data in the training data set and obtaining text information and description information; using a large language model to obtain question-answer data pairs with an inference process; selecting a number of question-answer data pairs in the training data set and combining the question-answer data pairs to construct a mixed training data set; designing a loss function for training the large visual language model; using the mixed training data set to complete the training of the large visual language model to be trained according to the loss function. The present invention also discloses a system for implementing the method for training the large visual language model for rich-text image question answering, and a rich-text image question answering method including the method for training the large visual language model for rich-text image question answering. The present invention can realize the training of a large visual language model, and has higher reliability and better overall performance.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a visual language large model training method and system for rich text image question answering and a rich text image question answering method. Background Art

[0002] In the real world, a large amount of visual content consists of text-rich images, such as logos, advertisements, electronic invoices, and scanned PDF documents. These images often require complex analysis, as the text within them is crucial to defining the image's context and meaning. Therefore, researchers have proposed a variety of techniques for identifying and parsing textual elements within images.

[0003] In recent years, large vision-language models (LVLMs) have demonstrated remarkable capabilities in integrating text and visual data. These models can simultaneously process both image and text information, thereby deepening understanding of images. Consequently, they have become a core technology for rich-text image question answering.

[0004] Therefore, to improve the reliability and accuracy of rich-text image question answering solutions, researchers have focused their attention on large visual language models to enhance their performance. However, current solutions to improve the performance of large visual language models primarily focus on increasing the resolution of input images to optimize performance by capturing more details. While these solutions achieve more precise understanding through high-resolution input, they ignore the more difficult problem of deep understanding and reasoning about the complex textual content within the image. This problem greatly limits the performance of large visual language models when dealing with visual question answering and reasoning tasks in natural scenes, significantly impacting the usability and reliability of rich-text image question answering solutions. Summary of the Invention

[0005] One of the purposes of the present invention is to provide a method for training a large visual language model for rich text image question answering that is suitable for rich text image question answering and has high reliability and good overall performance.

[0006] A second object of the present invention is to provide a system for implementing the visual language large model training method for rich text image question answering.

[0007] A third object of the present invention is to provide a rich-text image question-answering method that includes the visual language large model training method for rich-text image question-answering.

[0008] The method for training a large visual language model for rich text image question answering provided by the present invention comprises the following steps:

[0009] S1. Determine the visual language model to be trained and obtain a training dataset for the visual language model;

[0010] S2. Select several rich text image data from the training data set obtained in step S1, and obtain the text information and corresponding description information of the selected rich text image;

[0011] S3. Based on the text information and description information obtained in step S2, a large language model is used to obtain question-answer data pairs with an inference process;

[0012] S4. Select several question-answer data pairs from the training dataset obtained in step S1, and combine them with the question-answer data pairs obtained in step S3 to construct a hybrid training dataset;

[0013] S5. Design loss functions for training large visual language models;

[0014] S6. Using the mixed training data set obtained in step S4, and according to the loss function designed in step S5, complete the training of the large visual language model to be trained.

[0015] The step S1 of determining the visual language model to be trained and obtaining the training data set of the visual language model specifically includes the following steps:

[0016] The visual language model to be trained is specifically an original, untrained visual language model;

[0017] The training data set of the large visual language model includes original images and question-answer data pairs corresponding to the images; the original images include rich text images and ordinary images.

[0018] Step S2, in which a plurality of rich text image data are selected from the training data set obtained in step S1, and the text information and corresponding description information of the selected rich text images are obtained, specifically includes the following steps:

[0019] In the training data set obtained in step S1, a number of rich text image data is selected, and the number of rich text images selected is the number of original images in the training data set. n is the set ratio value;

[0020] Using optical character recognition tools to recognize text information in the selected rich text image; the recognized text information includes text coordinates, text content, and recognition confidence;

[0021] A pre-trained visual language model is used to obtain descriptive information of the selected rich text image; the descriptive information includes the type, color, number, action and position of objects in the rich text image, as well as the text information included in the rich text image and the relative position information of the objects.

[0022] Step S3, based on the text information and description information obtained in step S2, uses a large language model to obtain question-answer data pairs with an inference process, specifically including the following steps:

[0023] Design system prompts for large language models, instructing them to generate question-answer data pairs with reasoning processes;

[0024] Design several examples of generating question-answer data pairs with reasoning processes using text information and description information;

[0025] Input the system prompts and examples into the large language model, and then input the text information and description information obtained in step S2 into the large language model to obtain question-answer data pairs with reasoning process generated by the large language model.

[0026] Step S4 selects several question-answer data pairs from the training data set obtained in step S1 and combines them with the question-answer data pairs obtained in step S3 to construct a hybrid training data set, which specifically includes the following steps:

[0027] In the training data set obtained in step S1, a number of question-answer data pairs are selected, and the number of the selected question-answer data pairs is the number of all question-answer data pairs in the training data set. m is the set proportional coefficient, and m>n;

[0028] The selected question-answer data pairs and the question-answer data pairs obtained in step S3 constitute a mixed training data set.

[0029] The loss function for designing the large visual language model training described in step S5 specifically includes the following steps:

[0030] The following formula is used as the coarse-grained loss function:

[0031]

[0032] Where L CG is the coarse-grained loss function value; B is the batch size; sim() is the cosine similarity calculation function; is the average embedding of image features; is the average embedding of the label; τ is the temperature parameter used to adjust the convergence speed of the model;

[0033] The following formula is used as the fine-grained loss function:

[0034]

[0035] Where L FG is the fine-grained loss function value; N t is the total number of text boxes in the batch; is the average embedding of the image block corresponding to the text box; is the average embedding of the text in the text box; w i is the batch normalization weight value, and c i is the detection confidence score of the text box; for the text box of the self-annotated image data, the corresponding w i The value of is 1;

[0036] The following formula is used as the third loss function:

[0037]

[0038] Where L CE is the value of the third loss function; L is the length of all tags predicted by the model; is the true label corresponding to the i-th mark; The probability of the true label corresponding to the i-th tag predicted by the model;

[0039] The following formula is used as the loss function L for training the designed visual language model: total :

[0040] L total =αL CE +βL CG +γL FG

[0041] Where α is the first weight value; β is the second weight value; γ is the third weight value.

[0042] The visual language model to be trained in step S1 is specifically the LLaVAR visual language model;

[0043] The step S2 of obtaining the text information and corresponding description information of the selected rich text image specifically comprises using the PaddleOCR optical character recognition tool to obtain the text information of the selected rich text image, and using ShareGPT4V to obtain the description information corresponding to the selected rich text image;

[0044] The step S3 uses a large language model to obtain question-answer data pairs with reasoning processes, specifically using a GPT3.5 large language model to obtain question-answer data pairs with reasoning processes;

[0045] After getting the average embedding of image features In the process, CLIP visual encoder ViT-L / 14 is used as the visual encoder, and the ViT-L / 14 is a ViT-L pre-trained image encoder model;

[0046] Get the average embedding of the label In the process, the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text;

[0047] Get the average embedding of the text in the text box In the process, PaddleOCR is used to identify the text box in the image, and the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text;

[0048] After getting the average embedding of the image block corresponding to the text box In the process of image processing, the CLIP visual encoder ViT-L / 14 is used to evenly divide the image into several image blocks.

[0049] The present invention also provides a system for implementing the visual language large model training method for rich text image question answering, comprising a data acquisition module, an information acquisition module, a data question answering module, a training set construction module, a training design module and a model training module; the data acquisition module, the information acquisition module, the data question answering module, the training set construction module, the training design module and the model training module are connected in series in sequence; the data acquisition module is used to determine the visual language large model to be trained, and obtain the training data set of the visual language large model, and upload the data information to the information acquisition module; the information acquisition module is used to select a number of rich text image data in the obtained training data set according to the received data information, and obtain the text information and corresponding description information of the selected rich text image, and upload the data information to the data question answering module. Module; the data question and answer module is used to obtain question and answer data pairs with reasoning process based on the received data information, the obtained text information and the description information using the large language model, and upload the data information to the training set construction module; the training set construction module is used to select several question and answer data pairs in the obtained training data set based on the received data information, and construct a mixed training data set based on the obtained question and answer data pairs, and upload the data information to the training design module; the training design module is used to design the loss function for training the visual language large model based on the received data information, and upload the data information to the model training module; the model training module is used to complete the training of the visual language large model to be trained based on the received data information, the obtained mixed training data set, and the designed loss function.

[0050] The present invention also provides a rich-text image question-answering method including the visual language large model training method for rich-text image question-answering, further comprising the following steps:

[0051] S7. Use the trained visual language model obtained in step S6 to perform rich text image question answering.

[0052] The visual language large model training method, system and rich-text image question-answering method provided by the present invention for rich-text image question-answering, by combining reasoning ability stimulation and multi-granularity comparative learning, can use images as input to generate dialogue data with reasoning processes based on image content, and use the generated data and a portion of the original data, further combined with the multi-granularity comparative learning method to fine-tune the visual language large model, thereby achieving an improvement in the performance of the visual language large model for rich-text image question-answering; and the present invention has higher reliability and better overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the training method of the present invention.

[0054] Figure 2 Schematic diagram of the functional modules of the system of the present invention.

[0055] Figure 3 Schematic diagram of the method flow of the question-answering method of the present invention. DETAILED DESCRIPTION

[0056] like Figure 1 The figure shows a flow chart of the training method of the present invention: The visual language large model training method for rich text image question answering provided by the present invention comprises the following steps:

[0057] S1. Determine the visual language model to be trained and obtain a training dataset for the visual language model; specifically, the steps include:

[0058] The visual language model to be trained is specifically an original, untrained visual language model; for example, the LLaVAR visual language model can be used;

[0059] The training dataset of the large visual language model includes original images and question-answer data pairs corresponding to the images; the original images include rich text images and ordinary images; in specific implementation, the LLaVA_instruct_150k_LLaVAR_16k dataset can be used;

[0060] S2. Select several rich text image data from the training data set obtained in step S1, and obtain the text information and corresponding description information of the selected rich text image; specifically comprising the following steps:

[0061] In the training data set obtained in step S1, a number of rich text image data is selected, and the number of rich text images selected is the number of original images in the training data set. n is the set ratio value; in specific implementation, the preferred value of n is 10, and a 16k rich text image is selected;

[0062] Using optical character recognition tools to recognize text information in the selected rich text image; the recognized text information includes text coordinates, text content, and recognition confidence;

[0063] Using a pre-trained visual language model, descriptive information of the selected rich text image is obtained; the descriptive information includes the type, color, number, action and position of objects in the rich text image, as well as the text information included in the rich text image and the relative position information of the objects;

[0064] In specific implementation, the PaddleOCR optical character recognition tool can be used to obtain the text information of the selected rich text image, and ShareGPT4V can be used to obtain the description information corresponding to the selected rich text image;

[0065] S3. Based on the text information and description information obtained in step S2, a large language model is used to obtain question-answer data pairs with an inference process; specifically, the steps include:

[0066] Design system prompts for large language models, instructing them to generate question-answer data pairs with reasoning processes;

[0067] Design several examples of generating question-answer data pairs with reasoning processes using text information and description information. For specific implementation, two examples can be designed.

[0068] Input the system prompts and examples into the large language model, and then input the text information and description information obtained in step S2 into the large language model to obtain question-answer data pairs with reasoning process generated by the large language model;

[0069] In specific implementation, the GPT3.5 large language model can be used to obtain question-answer data pairs with reasoning processes;

[0070] S4. Select several question-answer data pairs from the training dataset obtained in step S1 and combine them with the question-answer data pairs obtained in step S3 to construct a hybrid training dataset. Specifically, the following steps are included:

[0071] In the training data set obtained in step S1, a number of question-answer data pairs are selected, and the number of the selected question-answer data pairs is the number of all question-answer data pairs in the training data set. m is the set proportional coefficient, and m>n; the purpose of setting m>n is to stimulate the reasoning ability of the large model; the preferred value of m is 30;

[0072] The selected question-answer data pairs and the question-answer data pairs obtained in step S3 constitute a mixed training data set;

[0073] S5. Design a loss function for training a large visual language model; specifically, the following steps are involved:

[0074] The following formula is used as the coarse-grained loss function:

[0075]

[0076] Where L CG is the coarse-grained loss function value; B is the batch size; sim() is the cosine similarity calculation function; is the average embedding of image features; is the average embedding of the label; τ is the temperature parameter used to adjust the convergence speed of the model;

[0077] When training a large visual language model, its visual encoder encodes the image into a feature vector and uses the visual-language adapter to convert it into the semantic space of the large language model, converting the feature vector into an embedding of image features. That is, the average value of the embedding of image features; in the specific implementation, the CLIP visual encoder ViT-L / 14 is used as the visual encoder, and the ViT-L / 14 is the ViT-L pre-trained image encoder model;

[0078] When training the visual language model, we obtain the answer part of the question-answering data, use the visual language model's tagger to convert it into a tag sequence, and then use the visual language model's embedding layer to convert it into an embedded sequence. That is, the average value of the label embedding sequence; in the specific implementation, the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text;

[0079] In addition, since the labels include words that are irrelevant to the image and have poor semantics, i.e., stop words, such as articles and prepositions, before calculating the average embedding of the labels, we use a tag-level tag filter to filter out stop words and other specified tags in the tags. Specifically, we obtain a public stop word list, convert it into a tag sequence using a tagger, and then manually select some semantically poor tags that are not in the stop word list and special tags based on the tag distribution of the tags.

[0080] The following formula is used as the fine-grained loss function:

[0081]

[0082] Where L FG is the fine-grained loss function value; N t is the total number of text boxes in the batch; is the average embedding of the image block corresponding to the text box; is the average embedding of the text in the text box; w i is the batch normalization weight value, and c iis the detection confidence score of the text box; for the text box of the self-annotated image data, the corresponding w i The value of is 1;

[0083] Use the text boxes in the labeled image data provided by the training data, or use an optical character recognition tool to identify the text boxes in the image. Use a tagger to tokenize the text in the text box, and then use the embedding layer of the visual language model to convert it into an embedded sequence. It is the average value of the text embedding sequence. In the specific implementation, PaddleOCR is used to identify the text box in the image, and the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text.

[0084] When training a large visual language model, its visual encoder divides the image into several image blocks, encodes each image block into a feature vector, and uses a visual-language adapter to convert it into the semantic space of the large language model, converting the feature vector into an embedding of image features. is the average value of the embedding of the image block features corresponding to the text box; in the specific implementation, the CLIP visual encoder ViT-L / 14 is used to evenly divide the image into several image blocks;

[0085] The following formula is used to express the large language model to generate target reasoning and answer X C Probability of:

[0086]

[0087] Where p(X C |X I ,X prompt ) represents the probability product of all tags corresponding to the reasoning and answer data generated by the large language model; p θ represents the probability of one of the tags generated by the large language model; x i represents the real-time generated tags during the model execution process; L represents the length of the output reasoning and answer data content; θ represents the trainable parameters; X C＜i Represents x i All previous dialogue prompts;

[0088] Use generated target dialogue data X C The probability p(X C |X I ,X prompt ), determine the label l of the predicted data m , and the label l of the real conversation data n , and then use the following formula as the third loss function:

[0089]

[0090] Where L CE is the value of the third loss function; L is the length of all tags predicted by the model; is the true label corresponding to the i-th mark; The probability of the true label corresponding to the i-th tag predicted by the model;

[0091] The following formula is used as the loss function L for training the designed visual language model: total :

[0092] L total =αL CE +βL CG +γL FG

[0093] Where α is the first weight value; β is the second weight value; γ is the third weight value;

[0094] S6. Using the mixed training data set obtained in step S4, and according to the loss function designed in step S5, complete the training of the large visual language model to be trained.

[0095] The advantages of the method of the present invention are described below through an embodiment:

[0096] The LLaVAR model training dataset, LLaVA_instruct_150k_LLaVAR_16k, is used. A portion of the rich text image data is selected to generate question-answering data with an inference process, and a portion of the original training data is used to create a mixed dataset. The LLaVAR model is trained using a multi-granularity contrastive learning method.

[0097] The STVQA, TextVQA, OCRVQA, ESTVQA, DocVQA, InfographicVQA, ChartQA, FUNSD, SROIE, and POIE datasets are used to evaluate the performance of different methods.

[0098] The STVQA, TextVQA, OCRVQA, and ESTVQA datasets are used to evaluate models' real-world question-answering capabilities. STVQA and ESTVQA require models to understand text within images; TextVQA requires reading and reasoning about text within images to answer questions; and OCRVQA focuses on text recognition in book cover images.

[0099] The DocVQA, InfographicVQA, and ChartQA datasets are used to evaluate models' document-based question answering capabilities. DocVQA is built around questions about document images, while InfoVQA is a dataset designed to evaluate comprehension of infographics. ChartQA includes a variety of chart types and topics, combining both human-written and machine-generated question-answer pairs. These datasets primarily target dense text scenarios, emphasizing understanding and parsing capabilities in complex text environments.

[0100] The FUNSD, SROIE, and POIE datasets are used to evaluate models' document-based information extraction capabilities. The SROIE dataset contains a large number of scanned receipt images and is used in an optical character recognition and key information extraction competition. In this competition, participants are required to extract information such as the merchant name, date, address, and total amount from receipts. The FUNSD dataset consists of real, fully annotated scanned forms that may contain distracting elements. The POIE dataset consists of camera-captured images of English product nutrition labels, collecting a large number of images with text examples.

[0101] The implementation results are as follows:

[0102] The test performance comparison data on STVQA, TextVQA, OCRVQA and ESTVQA datasets are shown in Table 1:

[0103] Table 1 Performance comparison data diagram

[0104] method STVQA TextVQA OCRVQA ESTVQA Original LLaVAR 29 28.4 17.9 46.5 Method of the present invention 31.1 29.6 19.2 50.4

[0105] The test performance comparison data on DocVQA, InfographicVQA (InfoVQA), ChartQA, FUNSD, SROIE and POIE datasets are shown in Table 2:

[0106] Table 2 Performance comparison data

[0107]

[0108] Tables 1 and 2 show that after the LLaVAR model is trained using the method of the present invention, the total score of the model is better than that of the original LLaVAR model on ten datasets, with improvements of 2.1%, 1.2%, 1.3% and 3.9% on the STVQA, TextVQA, OCRVQA and ESTVQA datasets, 0.5%, 0.7% and 0.1% on the DocVQA, InfographicVQA and ChartQA datasets, and 0.2%, 0.3% and 0.6% on the FUNSD, SROIE and POIE datasets, respectively. These results demonstrate the effectiveness of the scheme of the present invention.

[0109] like Figure 2 The figure shows a schematic diagram of the functional modules of the system of the present invention: the system disclosed in the present invention for realizing the method for training the large visual language model for rich text image question and answer, comprises a data acquisition module, an information acquisition module, a data question and answer module, a training set construction module, a training design module and a model training module; the data acquisition module, the information acquisition module, the data question and answer module, the training set construction module, the training design module and the model training module are connected in series in sequence; the data acquisition module is used to determine the large visual language model to be trained, and obtain the training data set of the large visual language model, and upload the data information to the information acquisition module; the information acquisition module is used to select a number of rich text image data in the obtained training data set according to the received data information, and obtain the text information and corresponding description information of the selected rich text image, and upload the data to the information acquisition module. Information is uploaded to the data question and answer module; the data question and answer module is used to obtain question and answer data pairs with reasoning process based on the received data information, the obtained text information and the description information using the large language model, and upload the data information to the training set construction module; the training set construction module is used to select several question and answer data pairs in the acquired training data set based on the received data information, and construct a mixed training data set based on the obtained question and answer data pairs, and upload the data information to the training design module; the training design module is used to design the loss function for training the visual language large model based on the received data information, and upload the data information to the model training module; the model training module is used to complete the training of the visual language large model to be trained based on the received data information and the obtained mixed training data set according to the designed loss function.

[0110] like Figure 3 The diagram shows a flow chart of the question-answering method of the present invention: The rich-text image question-answering method disclosed in the present invention, which includes the visual language large model training method for rich-text image question-answering, comprises the following steps:

[0111] S1. Determine the visual language model to be trained and obtain a training dataset for the visual language model;

[0112] S2. Select several rich text image data from the training data set obtained in step S1, and obtain the text information and corresponding description information of the selected rich text image;

[0113] S3. Based on the text information and description information obtained in step S2, a large language model is used to obtain question-answer data pairs with an inference process;

[0114] S4. Select several question-answer data pairs from the training dataset obtained in step S1, and combine them with the question-answer data pairs obtained in step S3 to construct a hybrid training dataset;

[0115] S5. Design loss functions for training large visual language models;

[0116] S6. Using the mixed training data set obtained in step S4, the training of the visual language model to be trained is completed according to the loss function designed in step S5;

[0117] S7. Use the trained visual language model obtained in step S6 to perform rich text image question answering.

Claims

1. A method for training a large visual language model for rich text image question answering, comprising the following steps: S1. Determine the visual language model to be trained and obtain a training dataset for the visual language model; S2. Select several rich text image data from the training data set obtained in step S1, and obtain the text information and corresponding description information of the selected rich text image; S3. Based on the text information and description information obtained in step S2, a large language model is used to obtain question-answer data pairs with an inference process; S4. Select several question-answer data pairs from the training dataset obtained in step S1, and combine them with the question-answer data pairs obtained in step S3 to construct a hybrid training dataset; S5. Design a loss function for training a large visual language model; specifically, the following steps are involved: The following formula is used as the coarse-grained loss function: Where L CG is the coarse-grained loss function value; B is the batch size; sim() is the cosine similarity calculation function; is the average embedding of image features; is the average embedding of the label; τ is the temperature parameter used to adjust the convergence speed of the model; The following formula is used as the fine-grained loss function: Where L FG is the fine-grained loss function value; N t is the total number of text boxes in the batch; is the average embedding of the image block corresponding to the text box; is the average embedding of the text in the text box; w i is the batch normalization weight value, and c i is the detection confidence score of the text box; for the text box of the self-annotated image data, the corresponding w i The value of is 1; The following formula is used as the third loss function: Where L CE is the value of the third loss function; L is the length of all tags predicted by the model; is the true label corresponding to the i-th mark; The probability of the true label corresponding to the i-th tag predicted by the model; The following formula is used as the loss function L for training the designed visual language model: total : L total =αL CE +βL CG +γL FG Where α is the first weight value; β is the second weight value; γ is the third weight value; S6. Using the mixed training data set obtained in step S4, the training of the visual language model to be trained is completed according to the loss function designed in step S5; S7. Use the trained visual language model obtained in step S6 to perform rich text image question answering.

2. The method for training a large visual language model for rich text image question answering according to claim 1 is characterized in that The step S1 of determining the visual language model to be trained and obtaining the training data set of the visual language model specifically includes the following steps: The visual language model to be trained is specifically an original, untrained visual language model; The training data set of the large visual language model includes original images and question-answer data pairs corresponding to the images; the original images include rich text images and ordinary images.

3. The visual language large model training method for rich text image question answering according to claim 2 is characterized in that Step S2, in which a plurality of rich text image data are selected from the training data set obtained in step S1, and the text information and corresponding description information of the selected rich text images are obtained, specifically includes the following steps: In the training data set obtained in step S1, a number of rich text image data is selected, and the number of rich text images selected is the number of original images in the training data set. n is the set ratio value; Using optical character recognition tools to recognize text information in the selected rich text image; the recognized text information includes text coordinates, text content, and recognition confidence; A pre-trained visual language model is used to obtain descriptive information of the selected rich text image; the descriptive information includes the type, color, number, action and position of objects in the rich text image, as well as the text information included in the rich text image and the relative position information of the objects.

4. The method for training a large visual language model for rich text image question answering according to claim 3 is characterized in that Step S3, based on the text information and description information obtained in step S2, uses a large language model to obtain question-answer data pairs with an inference process, specifically including the following steps: Design system prompts for large language models, instructing them to generate question-answer data pairs with reasoning processes; Design several examples of generating question-answer data pairs with reasoning processes using text information and description information; Input the system prompts and examples into the large language model, and then input the text information and description information obtained in step S2 into the large language model to obtain question-answer data pairs with reasoning process generated by the large language model.

5. The method for training a large visual language model for rich text image question answering according to claim 4 is characterized in that Step S4 selects several question-answer data pairs from the training data set obtained in step S1 and combines them with the question-answer data pairs obtained in step S3 to construct a hybrid training data set, which specifically includes the following steps: In the training data set obtained in step S1, a number of question-answer data pairs are selected, and the number of the selected question-answer data pairs is the number of all question-answer data pairs in the training data set. m is the set proportional coefficient, and m>n; The selected question-answer data pairs and the question-answer data pairs obtained in step S3 constitute a mixed training data set.

6. The method for training a large visual language model for rich text image question answering according to claim 1, characterized in that The visual language model to be trained in step S1 is specifically the LLaVAR visual language model; The step S2 of obtaining the text information and corresponding description information of the selected rich text image specifically comprises using the PaddleOCR optical character recognition tool to obtain the text information of the selected rich text image, and using ShareGPT4V to obtain the description information corresponding to the selected rich text image; The step S3 uses a large language model to obtain question-answer data pairs with reasoning processes, specifically using a GPT3.5 large language model to obtain question-answer data pairs with reasoning processes; After getting the average embedding of image features In the process, CLIP visual encoder ViT-L / 14 is used as the visual encoder, and the ViT-L / 14 is a ViT-L pre-trained image encoder model; Get the average embedding of the label In the process, the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text; Get the average embedding of the text in the text box In the process, PaddleOCR is used to identify the text box in the image, and the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text; After getting the average embedding of the image block corresponding to the text box In the process of image processing, the CLIP visual encoder ViT-L / 14 is used to evenly divide the image into several image blocks.

7. A system for implementing the visual language large model training method for rich text image question answering according to any one of claims 1 to 6, characterized in that It includes a data acquisition module, an information acquisition module, a data question and answer module, a training set construction module, a training design module and a model training module; the data acquisition module, the information acquisition module, the data question and answer module, the training set construction module, the training design module and the model training module are connected in series in sequence; the data acquisition module is used to determine the visual language model to be trained, and obtain the training data set of the visual language model, and upload the data information to the information acquisition module; the information acquisition module is used to select a number of rich text image data in the obtained training data set according to the received data information, and obtain the text information and corresponding description information of the selected rich text images, and upload the data information to the data question and answer module; The data question-answering module is used to obtain question-answer data pairs with reasoning processes based on the received data information, the obtained text information and the description information using a large language model, and upload the data information to the training set construction module; the training set construction module is used to select several question-answer data pairs from the obtained training data set based on the received data information, and combine the obtained question-answer data pairs to construct a mixed training data set, and upload the data information to the training design module; The training design module is used to design the loss function for training the visual language model based on the received data information, and upload the data information to the model training module; The model training module is used to complete the training of the visual language model to be trained based on the received data information, the obtained mixed training data set, and the designed loss function.

Images