Visual language large model training method and system for rich text image question and answer and rich text image question and answer method
Through the visual language big model training method, combined with reasoning ability and multi-grained comparison learning, the problem of insufficient understanding of complex text content in the existing technology is solved, and the performance and reliability of rich text image Q&A is improved.
Patent Information
- Application Number
- CN202411299895.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-18
AI Technical Summary
When dealing with visual Q&A and reasoning tasks in natural scenes, existing visual language models ignore the deep understanding of the complex text content in the image, limiting their performance, thus affecting the usability and reliability of the rich text image Q&A scheme.
By designing a visual language big model training method for rich text image question and answer, it includes determining the model to be trained, obtaining training data sets, selecting rich text image data, obtaining text information and description information, building a mixed training data set, and The loss function is designed for model training, combining inference ability and multi-grained comparison learning to improve model performance.
It realizes the generation of dialogue data with inference process based on image content, improves the efficiency and reliability of visual language models, and improves the comprehensive performance of rich text image Q&A.
Smart Images

Figure CN119066178A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a visual language large model training method and system for rich text image question answering and a rich text image question answering method. Background Art
[0002] In the real world, a large amount of visual content consists of images with rich text, such as logos, advertisements, electronic invoices, and scanned PDF documents. Such images usually require complex analysis because the text in them is crucial to defining the context and meaning of the image. Therefore, researchers have proposed a variety of techniques for identifying and parsing text elements within images.
[0003] In recent years, Large Vision-Language Models (LVLMs) have demonstrated their ability to integrate text and visual data. LVLMs can process both image and text information at the same time, thereby deepening the understanding of images. Therefore, LVLMs have become the core technology for rich text image question answering.
[0004] Therefore, in order to improve the reliability and accuracy of rich text image question answering solutions, researchers have focused on large visual language models to improve their performance. However, current solutions to improve the performance of large visual language models mainly focus on increasing the resolution of input images to optimize performance by capturing more details. Although such solutions can obtain more accurate understanding through high-resolution input, they ignore the more difficult problem: deep understanding and reasoning of complex text content in images. When dealing with visual question answering and reasoning tasks in natural scenes, this problem greatly limits the performance of large visual language models, thereby greatly affecting the availability and reliability of rich text image question answering solutions. Summary of the invention
[0005] One of the purposes of the present invention is to provide a visual language large model training method for rich text image question answering that is suitable for a rich text image question answering solution and has high reliability and good comprehensive performance.
[0006] A second objective of the present invention is to provide a system for implementing the visual language large model training method for rich text image question answering.
[0007] A third object of the present invention is to provide a rich-text image question-answering method that includes the visual language large model training method for rich-text image question-answering.
[0008] The visual language large model training method for rich text image question answering provided by the present invention comprises the following steps:
[0009] S1. Determine the visual language model to be trained and obtain a training data set for the visual language model;
[0010] S2. Selecting a number of rich text image data from the training data set obtained in step S1, and obtaining text information and corresponding description information of the selected rich text image;
[0011] S3. Based on the text information and description information obtained in step S2, a large language model is used to obtain question-answer data pairs with reasoning process;
[0012] S4. Select several question-answer data pairs from the training data set obtained in step S1, and combine them with the question-answer data pairs obtained in step S3 to construct a mixed training data set;
[0013] S5. Design loss function for training large visual language models;
[0014] S6. Using the mixed training data set obtained in step S4, and according to the loss function designed in step S5, complete the training of the large visual language model to be trained.
[0015] The step S1 of determining the visual language model to be trained and obtaining the training data set of the visual language model specifically includes the following steps:
[0016] The visual language model to be trained is specifically an original, untrained visual language model;
[0017] The training data set of the large visual language model includes original images and question-answer data pairs corresponding to the images; the original images include rich text images and ordinary images.
[0018] Step S2, in which a plurality of rich text image data are selected from the training data set obtained in step S1, and text information and corresponding description information of the selected rich text images are obtained, specifically comprises the following steps:
[0019] In the training data set obtained in step S1, a number of rich text image data is selected, and the number of rich text images selected is the number of original images in the training data set. n is the set ratio value;
[0020] Using an optical character recognition tool to recognize text information of the selected rich text image; the recognized text information includes text coordinates, text content and recognition confidence;
[0021] A pre-trained visual language large model is used to obtain description information of the selected rich text image; the description information includes the type, color, number, action and position of objects in the rich text image, as well as the text information included in the rich text image and the relative position information of the objects.
[0022] Step S3, based on the text information and description information obtained in step S2, uses a large language model to obtain question-answer data pairs with reasoning processes, specifically including the following steps:
[0023] Design system prompts for large language models to generate question-answer data pairs with reasoning processes;
[0024] Design several examples of generating question-answer data pairs with reasoning process through text information and description information;
[0025] The system prompts and examples are input into the large language model, and then the text information and description information obtained in step S2 are input into the large language model to obtain question-answer data pairs with reasoning process generated by the large language model.
[0026] Step S4 selects a number of question-answer data pairs from the training data set obtained in step S1, and combines the question-answer data pairs obtained in step S3 to construct a mixed training data set, which specifically includes the following steps:
[0027] In the training data set obtained in step S1, a number of question-answer data pairs are selected, and the number of the selected question-answer data pairs is the number of all question-answer data pairs in the training data set. m is the set proportional coefficient, and m>n;
[0028] The selected question-answer data pair and the question-answer data pair obtained in step S3 constitute a mixed training data set.
[0029] The loss function for designing the large visual language model training described in step S5 specifically includes the following steps:
[0030] The following formula is used as the coarse-grained loss function:
[0031]
[0032] Where L CG is the coarse-grained loss function value; B is the batch size; sim() is the cosine similarity calculation function; is the average embedding of image features; is the average embedding of the label; τ is the temperature parameter, which is used to adjust the convergence speed of the model;
[0033] The following formula is used as the fine-grained loss function:
[0034]
[0035] Where L FG is the fine-grained loss function value; N t is the total number of text boxes in the batch; is the average embedding of the image block corresponding to the text box; is the average embedding of the text in the text box; w i is the batch normalization weight value, and c i is the detection confidence score of the text box; for the text box of the self-annotated image data, the corresponding w i The value of is 1;
[0036] The following formula is used as the third loss function:
[0037]
[0038] Where L CE is the value of the third loss function; L is the length of all tags predicted by the model; is the true label corresponding to the i-th tag; The probability of the true label corresponding to the i-th tag predicted by the model;
[0039] The following formula is used as the loss function L for training the designed visual language model: total :
[0040] L total =αL CE +βL CG +γL FG
[0041] In the formula, α is the first weight value; β is the second weight value; γ is the third weight value.
[0042] The visual language model to be trained in step S1 is specifically the LLaVAR visual language model;
[0043] The step S2 of obtaining the text information and corresponding description information of the selected rich text image specifically comprises using the PaddleOCR optical character recognition tool to obtain the text information of the selected rich text image, and using ShareGPT4V to obtain the description information corresponding to the selected rich text image;
[0044] The step S3 uses a large language model to obtain question-answer data pairs with reasoning processes, specifically uses a GPT3.5 large language model to obtain question-answer data pairs with reasoning processes;
[0045] After getting the average embedding of image features In the process, CLIP visual encoder ViT-L / 14 is used as the visual encoder, and the ViT-L / 14 is a ViT-L pre-trained image encoder model;
[0046] Get the average embedding of the label In the process, the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text;
[0047] Get the average embedding of the text in the text box In the process, PaddleOCR is used to recognize the text box in the image, and the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text;
[0048] After getting the average embedding of the image block corresponding to the text box In the process of image segmentation, the CLIP visual encoder ViT-L / 14 is used to evenly divide the image into several image blocks.
[0049] The present invention also provides a system for implementing the visual language large model training method for rich text image question answering, comprising a data acquisition module, an information acquisition module, a data question answering module, a training set construction module, a training design module and a model training module; the data acquisition module, the information acquisition module, the data question answering module, the training set construction module, the training design module and the model training module are connected in series in sequence; the data acquisition module is used to determine the visual language large model to be trained, and obtain the training data set of the visual language large model, and upload the data information to the information acquisition module; the information acquisition module is used to select a number of rich text image data in the obtained training data set according to the received data information, and obtain the text information and corresponding description information of the selected rich text image, and upload the data information to the data question answering module. Module; the data question and answer module is used to obtain question and answer data pairs with reasoning process according to the received data information, the obtained text information and the description information, using the large language model, and upload the data information to the training set construction module; the training set construction module is used to select several question and answer data pairs in the acquired training data set according to the received data information, and construct a mixed training data set in combination with the obtained question and answer data pairs, and upload the data information to the training design module; the training design module is used to design the loss function of the visual language large model training according to the received data information, and upload the data information to the model training module; the model training module is used to complete the training of the visual language large model to be trained according to the received data information, using the obtained mixed training data set and the designed loss function.
[0050] The present invention also provides a rich text image question answering method including the visual language large model training method for rich text image question answering, and further includes the following steps:
[0051] S7. Use the trained visual language model obtained in step S6 to perform rich text image question answering.
[0052] The visual language large model training method, system and rich-text image question-answering method for rich-text image question-answering provided by the present invention can use images as input to generate dialogue data with reasoning processes based on image content, by combining reasoning ability stimulation and multi-granularity contrastive learning. At the same time, the generated data and a part of the original data are used, and the multi-granularity contrastive learning method is further combined to fine-tune the visual language large model, thereby improving the performance of the visual language large model for rich-text image question-answering. Moreover, the present invention has higher reliability and better overall performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 The figure is a flow chart of the training method of the present invention.
[0054] Figure 2 Schematic diagram of the functional modules of the system of the present invention.
[0055] Figure 3 Schematic diagram of the method flow of the question-answering method of the present invention. DETAILED DESCRIPTION
[0056] like Figure 1 The figure shows a flow chart of the training method of the present invention: the visual language large model training method for rich text image question answering provided by the present invention comprises the following steps:
[0057] S1. Determine the visual language model to be trained and obtain the training data set of the visual language model; specifically, the steps include:
[0058] The visual language model to be trained is specifically an original, untrained visual language model; for example, the LLaVAR visual language model can be used;
[0059] The training data set of the visual language large model includes original images and question-answer data pairs corresponding to the images; the original images include rich text images and ordinary images; in specific implementation, the LLaVA_instruct_150k_LLaVAR_16k data set can be used;
[0060] S2. Selecting a number of rich text image data from the training data set obtained in step S1, and obtaining text information and corresponding description information of the selected rich text images; specifically comprising the following steps:
[0061] In the training data set obtained in step S1, a number of rich text image data is selected, and the number of rich text images selected is the number of original images in the training data set. n is the set ratio value; in specific implementation, the preferred value of n is 10, and a 16k rich text image is selected;
[0062] Using an optical character recognition tool to recognize text information of the selected rich text image; the recognized text information includes text coordinates, text content and recognition confidence;
[0063] Using a pre-trained visual language large model, the description information of the selected rich text image is obtained; the description information includes the type, color, number, action and position of the objects in the rich text image, as well as the text information included in the rich text image and the relative position information of the objects;
[0064] In specific implementation, the PaddleOCR optical character recognition tool can be used to obtain the text information of the selected rich text image, and ShareGPT4V can be used to obtain the description information corresponding to the selected rich text image;
[0065] S3. Based on the text information and description information obtained in step S2, a large language model is used to obtain a question-answer data pair with an inference process; specifically, the steps include:
[0066] Design system prompts for large language models to generate question-answer data pairs with reasoning processes;
[0067] Design several examples of generating question-answer data pairs with reasoning processes through text information and description information. In specific implementation, two examples can be designed.
[0068] Input the system prompt and examples into the large language model, and then input the text information and description information obtained in step S2 into the large language model to obtain a question-answer data pair with an inference process generated by the large language model;
[0069] In specific implementation, the GPT3.5 large language model can be used to obtain question-answer data pairs with reasoning processes;
[0070] S4. Select several question-answer data pairs from the training data set obtained in step S1, and combine them with the question-answer data pairs obtained in step S3 to construct a mixed training data set; specifically, the following steps are included:
[0071] In the training data set obtained in step S1, a number of question-answer data pairs are selected, and the number of the selected question-answer data pairs is the number of all question-answer data pairs in the training data set. m is the set proportionality coefficient, and m>n; the purpose of setting m>n is to stimulate the reasoning ability of the large model; the preferred value of m is 30;
[0072] The selected question-answer data pair and the question-answer data pair obtained in step S3 constitute a mixed training data set;
[0073] S5. Design the loss function for training the large visual language model; specifically, the following steps are included:
[0074] The following formula is used as the coarse-grained loss function:
[0075]
[0076] Where L CG is the coarse-grained loss function value; B is the batch size; sim() is the cosine similarity calculation function; is the average embedding of image features; is the average embedding of the label; τ is the temperature parameter, which is used to adjust the convergence speed of the model;
[0077] When training the large vision-language model, its visual encoder encodes the image into a feature vector and uses the vision-language adapter to convert it into the semantic space of the large language model, converting the feature vector into an embedding of image features. That is, the average value of the embedding of the image features; in the specific implementation, the CLIP visual encoder ViT-L / 14 is used as the visual encoder, and the ViT-L / 14 is a ViT-L pre-trained image encoder model;
[0078] When training the visual language model, we obtain the answer part of the question-answering data, use the visual language model's tagger to convert it into a tag sequence, and then use the visual language model's embedding layer to convert it into an embedded sequence. It is the average value of the label embedding sequence. In the specific implementation, the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text.
[0079] In addition, since the labels include words that are irrelevant to the image and have poor semantics, i.e., stop words, such as articles and prepositions, etc. Before calculating the average embedding of the labels, the tag-level tag filter is used to filter the stop words and other specified tags in the labels; specifically, a public stop word list is obtained, and a tagger is used to convert it into a tag sequence, and then some semantically poor tags that are not in the stop word list and special tags are manually selected according to the tag distribution of the tags;
[0080] The following formula is used as the fine-grained loss function:
[0081]
[0082] Where L FG is the fine-grained loss function value; N t is the total number of text boxes in the batch; is the average embedding of the image block corresponding to the text box; is the average embedding of the text in the text box; w i is the batch normalization weight value, and c iis the detection confidence score of the text box; for the text box of the self-annotated image data, the corresponding w i The value of is 1;
[0083] Use the text boxes in the labeled image data that comes with the training data, or use an optical character recognition tool to identify the text boxes in the image. Use a tagger to tokenize the text in the text box, and then use the embedding layer of the visual language model to convert it into an embedded sequence. It is the average value of the text embedding sequence. In the specific implementation, PaddleOCR is used to recognize the text box in the image, and the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text.
[0084] When training the large visual language model, its visual encoder divides the image into several image blocks, encodes each image block into a feature vector, and uses the visual-language adapter to convert it into the semantic space of the large language model, converting the feature vector into the embedding of image features. is the average value of the embedding of the image block features corresponding to the text box; in the specific implementation, the CLIP visual encoder ViT-L / 14 is used to evenly divide the image into several image blocks;
[0085] The following formula is used to express the large language model to generate target reasoning and answer X C Probability:
[0086]
[0087] Where p(X C |X I ,X prompt ) represents the probability product of all tags corresponding to the reasoning and answer data generated by the large language model; p θ represents the probability of one of the tags generated by the large language model; x i represents the tags generated in real time during the model execution process; L represents the length of the output reasoning and answer data content; θ represents the trainable parameters; X C<i Represents x i All previous dialogue prompts;
[0088] Use the generated target dialogue data X C The probability p(X C |X I ,X prompt ), determine the label l of the predicted data m , and the label l of the real conversation data n , and then use the following formula as the third loss function:
[0089]
[0090] Where L CE is the value of the third loss function; L is the length of all tags predicted by the model; is the true label corresponding to the i-th tag; The probability of the true label corresponding to the i-th tag predicted by the model;
[0091] The following formula is used as the loss function L for training the designed visual language model: total :
[0092] L total =αL CE +βL CG +γL FG
[0093] Where α is the first weight value; β is the second weight value; γ is the third weight value;
[0094] S6. Using the mixed training data set obtained in step S4, and according to the loss function designed in step S5, complete the training of the large visual language model to be trained.
[0095] The advantages of the method of the present invention are described below by an embodiment:
[0096] The training dataset of the LLaVAR model is used: LLaVA_instruct_150k_LLaVAR_16k dataset. A portion of the rich text image data is selected to generate question-answering data with reasoning process, and a portion of the original training data is used to make a mixed dataset. The LLaVAR model is trained in combination with the multi-granularity contrastive learning method.
[0097] The STVQA, TextVQA, OCRVQA, ESTVQA, DocVQA, InfographicVQA, ChartQA, FUNSD, SROIE and POIE datasets are used to evaluate the performance of different methods.
[0098] The STVQA, TextVQA, OCRVQA, and ESTVQA datasets are used to evaluate the model's real-world question-answering capabilities. STVQA and ESTVQA require the model to understand the text in the image; TextVQA requires reading and reasoning about the text in the image to answer; and OCRVQA focuses on text recognition questions in book cover images.
[0099] DocVQA, InfographicVQA, and ChartQA datasets are used to evaluate the model's document-oriented question-answering capabilities. DocVQA is built around document image-based questions, and InfoVQA is a dataset designed to evaluate the ability to understand infographics. ChartQA contains various types of charts and topics, combining human-written and machine-generated question-answer pairs. These datasets are mainly oriented towards dense text scenarios, emphasizing the ability to understand and parse in complex text environments.
[0100] The FUNSD, SROIE, and POIE datasets are used to evaluate the model's document-oriented information extraction capabilities. SROIE contains a large number of scanned receipt images for optical character recognition and key information extraction competitions. In this competition, contestants need to extract information such as merchant name, date, address, and total amount from receipts. The FUNSD dataset consists of real and fully annotated scanned forms that may contain interference elements. The POIE dataset covers camera-captured images from English nutrition labels of products, collecting a large number of images with text instances.
[0101] The implementation results are as follows:
[0102] The test performance comparison data on STVQA, TextVQA, OCRVQA and ESTVQA datasets are shown in Table 1:
[0103] Table 1 Performance comparison data table
[0104] method STVQA TextVQA OCRVQA ESTVQA Original LLaVAR 29 28.4 17.9 46.5 Method of the present invention 31.1 29.6 19.2 50.4
[0105] The test performance comparison data on DocVQA, InfographicVQA (InfoVQA), ChartQA, FUNSD, SROIE and POIE datasets are shown in Table 2:
[0106] Table 2 Performance comparison data table
[0107]
[0108] Tables 1 and 2 show that after the LLaVAR model is trained using the method of the present invention, the total score of the model is better than that of the original LLaVAR model on ten data sets, and is improved by 2.1%, 1.2%, 1.3% and 3.9% on the STVQA, TextVQA, OCRVQA and ESTVQA data sets, by 0.5%, 0.7% and 0.1% on the DocVQA, InfographicVQA and ChartQA data sets, and by 0.2%, 0.3% and 0.6% on the FUNSD, SROIE and POIE data sets, respectively. These results demonstrate the effectiveness of the scheme of the present invention.
[0109] like Figure 2 The figure shows a schematic diagram of the functional modules of the system of the present invention: the system disclosed in the present invention for implementing the visual language large model training method for rich text image question and answer comprises a data acquisition module, an information acquisition module, a data question and answer module, a training set construction module, a training design module and a model training module; the data acquisition module, the information acquisition module, the data question and answer module, the training set construction module, the training design module and the model training module are connected in series in sequence; the data acquisition module is used to determine the visual language large model to be trained, and obtain the training data set of the visual language large model, and upload the data information to the information acquisition module; the information acquisition module is used to select a number of rich text image data in the obtained training data set according to the received data information, and obtain the text information and corresponding description information of the selected rich text image, and upload the data to the information acquisition module. Information is uploaded to the data question and answer module; the data question and answer module is used to obtain question and answer data pairs with reasoning process based on the received data information, the obtained text information and the description information, using the large language model, and upload the data information to the training set construction module; the training set construction module is used to select several question and answer data pairs in the acquired training data set according to the received data information, and construct a mixed training data set based on the obtained question and answer data pairs, and upload the data information to the training design module; the training design module is used to design the loss function of the visual language large model training according to the received data information, and upload the data information to the model training module; the model training module is used to complete the training of the visual language large model to be trained according to the received data information, using the obtained mixed training data set and the designed loss function.
[0110] like Figure 3 The diagram is a flow chart of the question-answering method of the present invention: the rich-text image question-answering method disclosed in the present invention, which includes the visual language large model training method for rich-text image question-answering, comprises the following steps:
[0111] S1. Determine the visual language model to be trained and obtain a training data set for the visual language model;
[0112] S2. Selecting a number of rich text image data from the training data set obtained in step S1, and obtaining text information and corresponding description information of the selected rich text image;
[0113] S3. Based on the text information and description information obtained in step S2, a large language model is used to obtain question-answer data pairs with reasoning process;
[0114] S4. Select several question-answer data pairs from the training data set obtained in step S1, and combine them with the question-answer data pairs obtained in step S3 to construct a mixed training data set;
[0115] S5. Design loss function for training large visual language models;
[0116] S6. Using the mixed training data set obtained in step S4, and according to the loss function designed in step S5, complete the training of the visual language model to be trained;
[0117] S7. Use the trained visual language model obtained in step S6 to perform rich text image question answering.
Claims
1. A method for training a large visual language model for rich text image question answering, comprising the following steps: S1. Determine the visual language model to be trained and obtain a training data set of the visual language model; S2. Selecting a number of rich text image data from the training data set obtained in step S1, and obtaining text information and corresponding description information of the selected rich text image; S3. Based on the text information and description information obtained in step S2, a large language model is used to obtain a question-answer data pair with an inference process; S4. Select several question-answer data pairs from the training data set obtained in step S1, and combine them with the question-answer data pairs obtained in step S3 to construct a mixed training data set; S5. Design loss function for training large visual language models; S6. Using the mixed training data set obtained in step S4, and according to the loss function designed in step S5, complete the training of the large visual language model to be trained.
2. The visual language large model training method for rich text image question answering according to claim 1 is characterized in that The step S1 of determining the visual language model to be trained and obtaining the training data set of the visual language model specifically includes the following steps: The visual language model to be trained is specifically an original, untrained visual language model; The training data set of the large visual language model includes original images and question-answer data pairs corresponding to the images; the original images include rich text images and ordinary images.
3. The visual language large model training method for rich text image question answering according to claim 2 is characterized in that Step S2, in which a plurality of rich text image data are selected from the training data set obtained in step S1, and text information and corresponding description information of the selected rich text images are obtained, specifically comprises the following steps: In the training data set obtained in step S1, a number of rich text image data is selected, and the number of rich text images selected is the number of original images in the training data set. n is the set ratio value; Using an optical character recognition tool to recognize text information of the selected rich text image; the recognized text information includes text coordinates, text content and recognition confidence; A pre-trained visual language large model is used to obtain description information of the selected rich text image; the description information includes the type, color, number, action and position of objects in the rich text image, as well as the text information included in the rich text image and the relative position information of the objects.
4. The visual language large model training method for rich text image question answering according to claim 3 is characterized in that Step S3, based on the text information and description information obtained in step S2, uses a large language model to obtain question-answer data pairs with reasoning processes, specifically including the following steps: Design system prompts for large language models to generate question-answer data pairs with reasoning processes; Design several examples of generating question-answer data pairs with reasoning processes through text information and description information; The system prompts and examples are input into the large language model, and then the text information and description information obtained in step S2 are input into the large language model to obtain question-answer data pairs with reasoning process generated by the large language model.
5. The visual language large model training method for rich text image question answering according to claim 4 is characterized in that Step S4 selects a number of question-answer data pairs from the training data set obtained in step S1, and combines the question-answer data pairs obtained in step S3 to construct a mixed training data set, which specifically includes the following steps: In the training data set obtained in step S1, a number of question-answer data pairs are selected, and the number of the selected question-answer data pairs is the number of all question-answer data pairs in the training data set. m is the set proportional coefficient, and m>n; The selected question-answer data pair and the question-answer data pair obtained in step S3 constitute a mixed training data set.
6. The visual language large model training method for rich text image question answering according to claim 5 is characterized in that The loss function for designing the large visual language model training described in step S5 specifically includes the following steps: The following formula is used as the coarse-grained loss function: Where L CG is the coarse-grained loss function value; B is the batch size; sim() is the cosine similarity calculation function; is the average embedding of image features; is the average embedding of the label; τ is the temperature parameter, which is used to adjust the convergence speed of the model; The following formula is used as the fine-grained loss function: Where L FG is the fine-grained loss function value; N t is the total number of text boxes in the batch; is the average embedding of the image block corresponding to the text box; is the average embedding of the text in the text box; w i is the batch normalization weight value, and c i is the detection confidence score of the text box; for the text box of the self-annotated image data, the corresponding w i The value of is 1; The following formula is used as the third loss function: Where L CE is the value of the third loss function; L is the length of all tags predicted by the model; is the true label corresponding to the i-th tag; The probability of the true label corresponding to the i-th tag predicted by the model; The following formula is used as the loss function L for training the designed visual language model: total : L total =αL CE +βL CG +γL FG In the formula, α is the first weight value; β is the second weight value; γ is the third weight value.
7. The visual language large model training method for rich text image question answering according to claim 6 is characterized in that The visual language model to be trained in step S1 is specifically the LLaVAR visual language model; The step S2 of obtaining the text information and corresponding description information of the selected rich text image specifically comprises using the PaddleOCR optical character recognition tool to obtain the text information of the selected rich text image, and using ShareGPT4V to obtain the description information corresponding to the selected rich text image; The step S3 uses a large language model to obtain question-answer data pairs with reasoning processes, specifically uses a GPT3.5 large language model to obtain question-answer data pairs with reasoning processes; After getting the average embedding of image features In the process, CLIP visual encoder ViT-L / 14 is used as the visual encoder, and the ViT-L / 14 is a ViT-L pre-trained image encoder model; Get the average embedding of the label In the process, the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text; Get the average embedding of the text in the text box In the process, PaddleOCR is used to recognize the text box in the image, and the pre-trained Token embedding model in the LLaMA-2 model is used to embed the text; After getting the average embedding of the image block corresponding to the text box In the process of image segmentation, the CLIP visual encoder ViT-L / 14 is used to evenly divide the image into several image blocks.
8. A system for implementing the visual language large model training method for rich text image question answering as described in any one of claims 1 to 7, characterized in that It includes a data acquisition module, an information acquisition module, a data question and answer module, a training set construction module, a training design module and a model training module; the data acquisition module, the information acquisition module, the data question and answer module, the training set construction module, the training design module and the model training module are connected in series in sequence; the data acquisition module is used to determine the visual language large model to be trained, and obtain the training data set of the visual language large model, and upload the data information to the information acquisition module; the information acquisition module is used to select a number of rich text image data in the obtained training data set according to the received data information, and obtain the text information and corresponding description information of the selected rich text image, and upload the data information to the data question and answer module; The data question-answering module is used to obtain question-answering data pairs with reasoning processes based on the received data information, the obtained text information and the description information using a large language model, and upload the data information to the training set construction module; the training set construction module is used to select a number of question-answering data pairs from the obtained training data set based on the received data information, and construct a mixed training data set based on the obtained question-answering data pairs, and upload the data information to the training design module; The training design module is used to design the loss function of the visual language large model training based on the received data information, and upload the data information to the model training module; The model training module is used to complete the training of the large visual language model to be trained based on the received data information, the obtained mixed training data set, and the designed loss function.
9. A rich text image question answering method comprising the visual language large model training method for rich text image question answering according to any one of claims 1 to 7, characterized in that The following steps are also included: S7. Use the trained visual language model obtained in step S6 to perform rich text image question answering.
Citation Information
Patent Citations
Image question and answer data acquisition method based on visual language large model
CN117972053A
Systems and methods for video and language pre-training
US20230154146A1
Cited By
MTU data decoupling formation and model training method, system and device and medium
CN119832579A
Training method, application method and training device of commodity processing model
CN120014381A
Large model training data generation method and device, medium and electronic equipment
CN120913010A
