Mobile phone screenshot text content question-answer pair generation method based on multi-modal large model

Abstract

The invention relates to the technical field of natural language processing, and provides a mobile phone screenshot text content question-answer pair generation method based on a multi-modal large model. The method comprises the following steps: extracting a picture text content, generating a text content description of a mobile phone screenshot picture by using a multi-modal large model, and obtaining content information related to a text in the picture; taking the mobile phone screenshot picture and the generated text content description as input of a multi-mode large model, and constructing an instruction to drive the multi-mode large model to generate question and answer pairs about the text content of the mobile phone screenshot picture; and screening the generated question and answer pairs by combining the generated picture text content and the generated question and answer pairs with a construction instruction. According to the method, the multi-mode large model and the large language model are used for automatically processing the mobile phone screen capture process, high-quality question and answer pair training data can be efficiently generated, and the performance of the multi-mode large model in a mobile terminal digital text understanding task is helped to be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model. Background Art

[0002] With the continuous advancement of science and technology, mobile phones have become one of the most commonly used items in people's daily lives. Training models to complete mobile interface text content question-answering tasks will help the model better understand the digital text content on the mobile terminal, thereby further improving the efficiency of tasks such as annotation, summarization and navigation of user interface elements, which is of great significance to the development of human-computer interaction and human communication. At present, with the rapid development of multimodal large language models, the answering of text questions has made significant progress. Many closed-source leading models have even surpassed the human level in some aspects and demonstrated excellent performance. However, the performance of most open-source models is still significantly behind these closed-source models. One of the main reasons is that open-source models lack sufficient training data for comprehensive training. Correspondingly, in the task of answering digital text content on the mobile terminal, due to problems such as the difficulty of manual annotation, it is even more difficult to train multimodal large models to complete related tasks.

[0003] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model.

[0005] The present invention adopts the following technical solution:

[0006] A method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model, comprising:

[0007] In step 201, extract the text content of the picture, and use the multimodal large model to generate a text content description of the mobile phone screenshot picture to obtain content information related to the text in the picture;

[0008] In step 202, the mobile phone screenshot image and the text content description generated in step 201 are used as inputs of the multimodal large model, and instructions are constructed to drive the multimodal large model to generate question-answer pairs about the text content of the mobile phone screenshot image;

[0009] In step 203, the question-answer pairs generated in step 202 are screened by using the text content description generated in step 201 and the question-answer pairs generated in step 202 in combination with construction instructions.

[0010] Preferably, the extracting of the text content of the image and using the multimodal large model to generate a text content description of the mobile phone screenshot image to obtain content information related to the text in the image specifically includes:

[0011] Use the mobile phone screenshot as the input of the multimodal large model, so that the multimodal large model can describe the text content in the input image;

[0012] The obtained text content description is used as the input of the multimodal large model again, so that the multimodal large model can directly determine whether the text content description contains meaningless and repeated text content;

[0013] If the text content description does not contain any repeated parts, the multimodal large model successfully obtains the text content description of the mobile phone screenshot; otherwise, the multimodal large model fails to successfully obtain the text content description of the mobile phone screenshot.

[0014] Preferably, the mobile phone screenshot image and the text content description generated in step 201 are used as inputs of the multimodal large model, and instructions are constructed to drive the multimodal large model to generate question-answer pairs about the text content of the mobile phone screenshot image, specifically including:

[0015] Build an instruction library containing multiple instructions, and randomly extract an instruction from the instruction library each time a question-answer pair is generated;

[0016] After the multimodal large model finishes generating question-answer pairs, the generated question-answer pairs are used as input again, and the multimodal large model is used to determine whether the number of question-answer pairs in the final generated result reaches the specified number, and to check whether the output content meets the preset format requirements;

[0017] If the number of question-answer pairs in the output of the multimodal large model does not reach the specified number or the content format of the multimodal large model output does not meet the format requirements, the question-answer pairs generated in this round will not be used as the final result. Instead, a non-repeated instruction will be randomly extracted from the instruction library again to drive the multimodal large model to complete the generation of question-answer pairs.

[0018] Repeat the above steps until the question-answer pairs generated by the multimodal large model meet the quantity and format requirements;

[0019] If no expected results are obtained after using all the instructions in the instruction library, it is directly determined that the generation of the question-answer pair for the image has failed.

[0020] Preferably, the instructions in the instruction library are divided into two types: instructions for strengthening thought chains and instructions for specifying the identity of the multimodal macro model.

[0021] Preferably, the method for constructing the instruction for strengthening the thinking chain specifically includes:

[0022] Instructions emphasize that large multimodal models require careful step-by-step thinking to complete tasks;

[0023] Emphasize in the instruction that the input image is a mobile phone screenshot;

[0024] Determine whether the text content description is obtained in step 201. If the text content description of the picture is obtained, add the content to the instruction as input, and emphasize in the instruction that the text information of the entire picture needs to be summarized based on the input text content and the picture; otherwise, the instruction drives the multimodal large model to first describe the content of the entire picture, and then summarize the text information of the entire picture based on the input picture and the picture content generated by the multimodal large model;

[0025] The instructions indicate that the multimodal large model needs to generate a preset number of meaningful and correct question-answer pairs based on the summarized image text information and the input image, and ensure that the generated question-answer pairs do not introduce external knowledge as much as possible;

[0026] Provide an output example in the instructions and emphasize that the multimodal large model should output the summarized image text information and the generated question-answer pairs in the provided sample format.

[0027] Preferably, the method for constructing the instruction for specifying the identity of the multimodal large model specifically includes:

[0028] Specify the identity of the big model as a mobile assistant or a mobile user;

[0029] When the identity of the multimodal large model is specified as a mobile phone assistant, the instruction indicates that the input mobile phone screenshot picture is the mobile phone interface currently being used by the mobile phone user, and indicates that the task of the multimodal large model is to help the user use the mobile phone; detect whether step 201 successfully obtains the text content description of the mobile phone screenshot picture; if step 201 successfully obtains the text content description, then this part of the content is added to the instruction as external knowledge, and it is emphasized that the multimodal large model needs to summarize the text information contained in the entire mobile phone screen according to the text content description and the input mobile phone screenshot picture; otherwise, the instruction drives the multimodal large model to read the text content in the mobile phone screenshot as detailed as possible and summarize the entire picture text information; the instruction indicates that the multimodal large model needs to simulate the problems encountered by the mobile phone user when using the mobile phone interface according to the picture and the text information of the picture, and as a mobile phone assistant, it gives corresponding meaningful answers that do not introduce external knowledge according to the picture to help the user solve the problem; the instruction includes an output sample, and emphasizes that the multimodal large model needs to output the summarized mobile phone screenshot picture text information and the simulated question and answer pairs according to the provided sample format;

[0030] When the identity of the multimodal large model is specified as a mobile phone user, the instruction indicates that the input mobile phone screenshot picture is the mobile phone interface currently being used by the user; whether the text content description of the mobile phone screenshot picture is successfully obtained in step 201 is detected; if the text content description of the picture is obtained in step 201, the instruction indicates that the multimodal large model needs to summarize the text information seen when using the mobile phone screenshot according to the text content description and the input mobile phone screenshot; otherwise, the instruction indicates that the multimodal large model needs to read the mobile phone screenshot carefully and summarize the text information therein in detail;

[0031] The multimodal large model combines mobile phone screenshots to simulate the questions that mobile phone users have when they see the mobile phone screen during use, and gives corresponding, meaningful answers to these questions without introducing external knowledge by combining pictures and text information;

[0032] Provide an output sample in the instruction, and ask the multimodal large model to output the image text information summarized by the multimodal large model and the generated question and answer pairs according to the provided sample format.

[0033] Preferably, the method of using the text content description generated in step 201 and the question-answer pairs generated in step 202 in combination with the construction instruction to screen the question-answer pairs generated in step 202 specifically includes:

[0034] Construct an instruction-driven multimodal large model to self-correct the generated question-answer pairs using thought chains;

[0035] The instruction-driven large language model is constructed to filter out the generated question-answer pairs that are irrelevant to the text in the mobile phone screenshot.

[0036] Preferably, the construction instruction drives the multimodal large model to self-correct the generated question-answer pairs using thought chains, specifically including:

[0037] In the instructions, the multimodal big model’s identity was indicated as an expert in image analysis;

[0038] Input the mobile phone screenshot, the image text information output in step 202, and the questions in the generated question-answer pair into the multimodal large model;

[0039] The instructions require the multimodal large model to combine the image and the image text information to give the most concise answer to the input question. The generated answer should only be related to the image text information and no external knowledge should be introduced.

[0040] The instruction gives the reason for the answer and indicates that the reason for the answer needs to be able to find the corresponding clues in the input picture;

[0041] Provide output samples in the instructions and emphasize that the multimodal large model outputs questions, answers to questions, and clues to answers one by one in the sample format;

[0042] The final result of this step is used to replace the answer generated in step 202 for subsequent screening.

[0043] Preferably, the construction instruction drives the large language model to filter out the question-answer pairs generated that are irrelevant to the text in the mobile phone screenshot image, specifically including:

[0044] Determine whether the text content description of the mobile phone screenshot image is successfully obtained in step 201;

[0045] If the text content description of the mobile phone screenshot is successfully obtained, the text content description and the question generated in the previous step are used as input;

[0046] The instruction requires the large language model to determine whether the question is related to the input text content description;

[0047] If the text content description in the mobile phone screenshot image is not successfully obtained in step 201, the image text information summarized in step 202 and the question generated in the previous step are used as input, and the construction instruction requires the large language model to determine whether the input question is related to the input image text information;

[0048] Based on the judgment of the large language model, questions and answers that are irrelevant to the text content in the question and answer pairs and the mobile phone screenshots are filtered out, and finally high-quality question and answer pair data is obtained.

[0049] In a second aspect, the present invention further provides a device for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model, which is used to implement the method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model described in the first aspect, and the device includes:

[0050] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model as described in the first aspect.

[0051] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0052] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory to execute the method of the first aspect.

[0053] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method of the first aspect.

[0054] The present invention uses a multimodal large model to extract text content information from mobile phone screenshots, takes text content information and mobile phone screenshots as input, and constructs a plurality of instructions to form an instruction library by building an emphasizing thinking chain, specifying the identity of a large model, and other methods, so as to drive the multimodal large model to generate question-answer pairs with appropriate quantity and format related to the text content of the picture. In addition, the multimodal large model and the large language model are used to self-correct and screen the generated question-answer pairs, so as to ensure the accuracy of the question-answer pairs finally generated. This process of automatically processing mobile phone screenshots using a multimodal large model and a large language model can efficiently generate high-quality question-answer pair training data to help improve the performance of the multimodal large model in mobile digital text understanding tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a flowchart of a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided by an embodiment of the present invention;

[0057] Figure 2 It is a flowchart of a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided by an embodiment of the present invention;

[0058] Figure 3 It is a flowchart of a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided by an embodiment of the present invention;

[0059] Figure 4 It is a schematic diagram of a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided by an embodiment of the present invention;

[0060] Figure 5 It is a schematic diagram of a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided by an embodiment of the present invention;

[0061] Figure 6 It is a schematic diagram of a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided by an embodiment of the present invention;

[0062] Figure 7 It is a schematic diagram of a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided by an embodiment of the present invention;

[0063] Figure 8 It is a schematic diagram of the architecture of a device for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0065] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0066] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0067] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0068] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0069] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0070] Embodiment 1:

[0071] This embodiment proposes a concise and efficient instruction-driven data generation method. By inputting mobile phone screenshots and specific instructions into the multimodal large model, it generates high-quality screenshot image text content question-answer pairs, thereby efficiently generating question-answer pair training data for the model in the mobile digital text understanding task.

[0072] A method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model provided in Example 1 of the present invention includes:

[0073] In step 201, the text content of the picture is extracted, and a text content description of the mobile phone screenshot picture is generated using a multimodal large model to obtain content information related to the text in the picture; the mobile phone screenshot picture is also referred to as an input picture or picture in subsequent embodiments.

[0074] In step 202, the mobile phone screenshot image and the text content description generated in step 201 are used as inputs of the multimodal large model, and instructions are constructed to drive the multimodal large model to generate question-answer pairs about the text content of the mobile phone screenshot image; the question-answer pairs include questions and answers.

[0075] In step 203, the question-answer pairs generated in step 202 are screened by using the text content description generated in step 201 and the question-answer pairs generated in step 202 in combination with construction instructions.

[0076] The extraction of the text content of the image uses a multimodal large model to generate a text content description of the mobile phone screenshot image to obtain content information related to the text in the image, such as Figure 2 As shown, specifically including:

[0077] In step 301, a screenshot of a mobile phone is used as an input of a multimodal large model, so that the multimodal large model describes the text content in the input image.

[0078] In step 302, the obtained text content description is used as the input of the multimodal large model again, so that the multimodal large model can directly determine whether the text content description contains meaningless and repetitive text content.

[0079] In step 303, if the text content description does not contain a repeated part (i.e., the meaningless and repetitive text content mentioned above), the multimodal large model successfully obtains the text content description of the mobile phone screenshot; otherwise, the multimodal large model fails to successfully obtain the text content description of the mobile phone screenshot, and this part of the content will not be used in subsequent steps.

[0080] In an optional implementation, the mobile phone screenshot image and the text content description generated in step 201 are used as inputs of the multimodal large model, and instructions are constructed to drive the multimodal large model to generate question-answer pairs about the text content of the mobile phone screenshot image, such as Figure 3 As shown, specifically including:

[0081] In step 401, an instruction library containing multiple instructions is constructed, and an instruction is randomly selected from the instruction library for use each time a question-answer pair is generated.

[0082] In step 402, after the multimodal large model finishes generating question-answer pairs, the generated question-answer pairs are used as input again, and the multimodal large model is used to determine whether the number of question-answer pairs in the final generated result reaches the specified number, and check whether the output content meets the preset format requirements; the specified number and preset format requirements are obtained by technical personnel in this field based on demand analysis.

[0083] In step 403, if the number of question-answer pairs in the output result of the multimodal large model fails to reach the specified number or the content format of the multimodal large model output does not meet the format requirements, the question-answer pairs generated in this round are not used as the final result. Instead, a non-repeating instruction is randomly selected from the instruction library again to drive the multimodal large model to complete the generation of the question-answer pairs.

[0084] In step 404, the above steps are repeated until the question-answer pairs generated by the multimodal large model meet the quantity requirements and format requirements.

[0085] In step 405, if no expected result is obtained after using all the instructions in the instruction library, it is directly determined that the generation of the question-answer pair for the image has failed.

[0086] Among them, the instructions in the instruction library are divided into two types: instructions for strengthening the thinking chain and instructions for specifying the identity of the multimodal large model.

[0087] The method for constructing the instruction for strengthening the thinking chain specifically includes:

[0088] The instructions emphasize that large multimodal models require careful step-by-step thinking to complete the task.

[0089] It is emphasized in the instruction that the input image is a mobile phone screenshot.

[0090] It is determined whether step 201 has obtained a text content description (i.e., a text content description of the mobile phone screenshot image). If the text content description of the image is obtained, this part of the content is added to the instruction as input, and the instruction emphasizes the need to summarize the text information of the entire image based on the input text content and image; otherwise, the instruction drives the multimodal large model to first describe the content of the entire image, and then summarize the entire image text information based on the input image and the image content generated by the multimodal large model.

[0091] The instructions indicate that the multimodal large model needs to generate a preset number of meaningful and correct question-answer pairs based on the summarized image text information and the input image, and ensure that the generated question-answer pairs do not introduce external knowledge as much as possible; the preset number is obtained by technical personnel in this field based on empirical analysis.

[0092] Provide an output example in the instructions and emphasize that the multimodal large model should output the summarized image text information and the generated question-answer pairs in the provided sample format.

[0093] The method for constructing the instruction for specifying the identity of the multimodal large model specifically includes:

[0094] Specify the identity of the big model as a mobile assistant or a mobile user.

[0095] When the identity of the multimodal large model is specified as a mobile phone assistant, the instruction indicates that the input mobile phone screenshot picture is the mobile phone interface currently being used by the mobile phone user, and indicates that the task of the multimodal large model is to help the user use the mobile phone; detect whether step 201 successfully obtains the text content description of the mobile phone screenshot picture; if step 201 successfully obtains the text content description, then this part of the content is added to the instruction as external knowledge, and emphasizes that the multimodal large model needs to summarize the text information contained in the entire mobile phone screen based on the text content description and the input mobile phone screenshot picture; otherwise, the instruction drives the multimodal large model to read the text content in the mobile phone screenshot as detailed as possible and summarize the entire picture text information; the instruction indicates that the multimodal large model needs to simulate the problems encountered by the mobile phone user when using the mobile phone interface based on the picture and the text information of the picture, and as a mobile phone assistant, it gives corresponding meaningful answers based on the picture without introducing external knowledge to help the user solve the problem; the instruction includes an output sample, and emphasizes that the multimodal large model needs to output the summarized mobile phone screenshot picture text information and the simulated question and answer pairs according to the provided sample format.

[0096] When the identity of the multimodal large model is specified as a mobile phone user, the instruction indicates that the input mobile phone screenshot is the mobile phone interface currently being used by the user; whether step 201 successfully obtains the text content description of the mobile phone screenshot; if step 201 obtains the text content description of the picture, the instruction indicates that the multimodal large model needs to summarize the text information seen when using the mobile phone screenshot based on the text content description and the input mobile phone screenshot; otherwise, the instruction indicates that the multimodal large model needs to read the mobile phone screenshot carefully and summarize the text information therein in detail.

[0097] The multimodal large model combines mobile phone screenshots to simulate the questions that mobile phone users will have when they see the mobile phone screen while using the mobile phone, and combines pictures and text information to give corresponding, meaningful answers to these questions without introducing external knowledge.

[0098] Provide an output sample in the instruction, and ask the multimodal large model to output the image text information summarized by the multimodal large model and the generated question and answer pairs according to the provided sample format.

[0099] In some embodiments, the use of the text content description generated in step 201 and the question-answer pairs generated in step 202 in combination with the construction instructions to screen the question-answer pairs generated in step 202 specifically includes:

[0100] Constructing an instruction-driven multimodal large model uses thought chains to self-correct the generated question-answer pairs.

[0101] The instruction-driven large language model is constructed to filter out the generated question-answer pairs that are irrelevant to the text in the mobile phone screenshot.

[0102] In an optional implementation, the construction instruction drives the multimodal large model to self-correct the generated question-answer pairs using thought chains, specifically including:

[0103] The instruction indicates that the multimodal big model is an expert in image analysis; input the mobile phone screenshot, the image text information output in step 202, and the question in the generated question-answer pair to the multimodal big model; the instruction requires the multimodal big model to combine the image and the image text information to give an answer that is as concise as possible corresponding to the input question, and the generated answer should only be related to the image text information without introducing external knowledge; the instruction gives the reason for the answer and indicates that the reason for the answer needs to be able to find corresponding clues in the input image; the instruction provides an output sample and emphasizes that the multimodal big model outputs the question, the answer to the question, and the clue to the answer one by one according to the sample format; the final result of this step is used to replace the answer generated in step 202 for subsequent screening.

[0104] In actual application scenarios, the construction instructions drive the large language model to filter out the generated question-answer pairs that are irrelevant to the text in the mobile phone screenshot, specifically including:

[0105] Determine whether step 201 has successfully obtained the text content description of the mobile phone screenshot image; if the text content description of the mobile phone screenshot image is successfully obtained, the text content description and the question generated in the previous step are used as input; in the instruction, the large language model is required to determine whether the question is related to the input text content description.

[0106] If the text content description in the mobile phone screenshot image is not successfully obtained in step 201, the image text information summarized in step 202 and the question generated in the previous step are used as input, and the construction instruction requires the large language model to determine whether the input question is related to the input image text information.

[0107] Based on the judgment of the large language model, questions and answers that are irrelevant to the text content in the question and answer pairs and the mobile phone screenshots are filtered out, and finally high-quality question and answer pair data is obtained.

[0108] Step 202 in this embodiment can be understood as: by inputting corresponding instructions into the multimodal large model, the result of the corresponding stage (i.e., the output result of the multimodal large model) is obtained, and step 203 can be understood as: by inputting corresponding instructions into the multimodal large model, the result of the corresponding stage (i.e., the output result of the large language model) is obtained.

[0109] This embodiment uses a multimodal large model to extract text content information from mobile phone screenshots, takes text content information and mobile phone screenshots as input, and constructs a plurality of instructions to form an instruction library by building an emphasis on thought chains, specifying the identity of a large model, and other methods, to drive the multimodal large model to generate question-answer pairs with appropriate quantity and format related to the text content of the picture. In addition, the multimodal large model and the large language model are used to self-correct and screen the generated question-answer pairs to ensure the accuracy of the final generated question-answer pairs. This process of automatically processing mobile phone screenshots using a multimodal large model and a large language model can efficiently generate high-quality question-answer pair training data to help improve the performance of the multimodal large model in mobile digital text understanding tasks.

[0110] Embodiment 2:

[0111] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0112] The present invention proposes a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model. By constructing instructions, the multimodal large model is driven to generate question-answer pairs of high-quality mobile phone screenshot image text content, such as Figure 4 As shown, the following steps are included:

[0113] Step 1: Extract the text content of the image, use the multimodal large model to generate a text content description of the mobile phone screenshot image, and obtain content information related to the text in the image.

[0114] Step 2: Generate question-answer pairs. Use the mobile phone screenshot and the text content description generated in step 1 as the input of the multimodal large model, and construct instructions to drive the multimodal large model to generate question-answer pairs about the text content of the mobile phone screenshot.

[0115] Step 3: Filter question-answer pairs. Use the image text content generated in step 1 and the question-answer pairs generated in step 2 to build instructions and filter the question-answer pairs generated in step 2.

[0116] Among them, when using the multimodal large model to generate the text content description of the mobile phone screenshot in step one, the mobile phone screenshot needs to be used as the input of the multimodal large model first, so that the multimodal large model can describe the text content in the input picture; then the obtained text content description needs to be used as the input of the multimodal large model, so that the multimodal large model can directly determine whether the text content description contains meaningless repeated parts. If the text content description does not contain repeated parts, the multimodal large model successfully obtains the text content description of the mobile phone screenshot, and this part of the content will be used in subsequent steps; otherwise, the multimodal large model fails to successfully obtain the text content of the mobile phone screenshot, and this part of the information will not be used in subsequent steps.

[0117] When constructing an instruction-driven multimodal large model to generate question-answer pairs in step 2, first, it is necessary to build an instruction library containing multiple instructions, and randomly extract an instruction from the instruction library each time a question-answer pair is generated; then, after the multimodal large model generates question-answer pairs, it is necessary to use the generated question-answer pairs as input again, and use the multimodal large model to determine whether the number of question-answer pairs in the final generated result reaches the specified number, and check whether the output content meets the preset format requirements. If the number of question-answer pairs in the output result of the multimodal large model fails to reach the specified number or the content format of the multimodal large model output does not meet the format requirements, the question-answer pairs generated in this round are not used as the final result, but a non-repeated instruction is randomly extracted from the instruction library again to drive the multimodal large model to complete the generation of the question-answer pairs. Repeat the above steps until the question-answer pairs generated by the multimodal large model meet the number requirements and format requirements. If all the instructions in the instruction library are used and no expected results are obtained, it is directly determined that the generation of the question-answer pair for the image has failed.

[0118] Moreover, when constructing instructions to drive the multimodal large model to generate question-answer pairs in step 2, the instructions in the constructed instruction library are divided into two types: instructions for strengthening the thinking chain and instructions for specifying the identity of the multimodal large model.

[0119] When constructing instructions to strengthen the thinking chain, first of all, the instructions should emphasize that the multimodal large model needs to think carefully step by step to complete the task; secondly, it is necessary to emphasize in the instructions that the input image is a screenshot of a mobile phone; then, it is necessary to determine whether the text content description is obtained in step 1. If the text content description of the image is obtained, then add this part of the content to the instruction as input, and emphasize in the instructions that the text information of the entire image needs to be summarized based on the input text content and the image; otherwise, the instructions should drive the multimodal large model to first describe the content of the entire image, and then summarize the entire image text information based on the input image and the image content generated by the multimodal large model. Then, the instructions should indicate that the multimodal large model needs to generate a certain number of meaningful and correct question-answer pairs based on the summarized image text information and the input image, and ensure that the generated question-answer pairs do not introduce external knowledge as much as possible. Finally, an output example needs to be provided in the instructions, and it is emphasized that the multimodal large model should output the summarized image text information and the generated question-answer pairs in the provided sample format.

[0120] When constructing an instruction to specify the identity of a multimodal big model, you need to specify the identity of the big model as a mobile assistant or a mobile user.

[0121] When specifying the identity of the multimodal large model as a mobile phone assistant, first of all, it is necessary to indicate in the instruction that the input mobile phone screenshot is the mobile phone interface currently used by the mobile phone user, and that the task of the multimodal large model is to help the user use the mobile phone. Secondly, it is necessary to detect whether the text content description of the mobile phone screenshot is successfully obtained in step 1. If the text content description is successfully obtained in step 1, this part of the content is added to the instruction as external knowledge, and it is emphasized that the multimodal large model needs to summarize the text information contained in the entire mobile phone screen based on the text content description and the input mobile phone screenshot; otherwise, the instruction should drive the multimodal large model to read the text content in the mobile phone screenshot as detailed as possible and summarize the entire picture text information. Next, the instruction should indicate that the multimodal large model needs to simulate the problems that the mobile phone user may encounter when using the mobile phone interface based on the picture and the text information of the picture, and as a mobile phone assistant, give corresponding meaningful answers based on the picture without introducing external knowledge to help the user solve the problem. Finally, the instruction should include an output sample, and emphasize that the multimodal large model needs to output the summarized mobile phone screenshot text information and the simulated question and answer pairs according to the provided sample format.

[0122] When the identity of the multimodal large model is specified as a mobile phone user, first of all, it is necessary to indicate in the instruction that the input mobile phone screenshot is the mobile phone interface that the user is using. Secondly, it is necessary to detect whether step one successfully obtains the text content description of the mobile phone screenshot. If step one obtains the text content of the picture, the instruction should indicate that the multimodal large model needs to summarize the text information seen when using this mobile phone screenshot based on the text content description and the input mobile phone screenshot; otherwise, the instruction should indicate that the multimodal large model needs to read the mobile phone screenshot carefully and summarize the text information in detail. Next, the multimodal large model should combine the mobile phone screenshot to simulate the questions that the mobile phone user may have when seeing the mobile phone screen during the use of the mobile phone, and give corresponding, meaningful, and non-external knowledge answers to these questions in combination with the picture and text information. Finally, an output sample needs to be provided in the instruction, and the multimodal large model is required to output the picture text information summarized by the multimodal large model and the generated question and answer pairs according to the provided sample format.

[0123] When filtering the question-answer pairs generated in step 2 in step 3, it is first necessary to build a command-driven multimodal large model to use the thought chain to self-correct the generated question-answer pairs. Then, it is necessary to build a command-driven large language model to filter out the generated question-answer pairs that are irrelevant to the text in the mobile phone screenshot.

[0124] When constructing the instruction-driven multimodal big model in step 3 to use the thinking chain to self-correct the generated question-answer pairs, first of all, it is necessary to indicate in the instruction that the identity of the multimodal big model is an image analysis expert; secondly, input the mobile phone screenshot image, the image text information output in step 2, and the question in the generated question-answer pair to the multimodal big model; then, the instruction should require the multimodal big model to combine the image and the text information of the image to give the answer to the input question as concisely as possible, only related to the image text information, and without introducing external knowledge; then, the instruction gives the reason for the answer and indicates that the reason for the answer needs to find the corresponding clue in the input image. Then, the instruction should provide an output sample and emphasize that the multimodal big model should output the question, the answer to the question, and the reason for the answer one by one according to the sample format. The result obtained at the end of this step will replace the answer generated in step 2 for subsequent screening.

[0125] When constructing instructions to drive the large language model to filter out the generated question-answer pairs that are unrelated to the text in the mobile phone screenshot, it is first necessary to determine whether step one has successfully obtained the text content description of the mobile phone screenshot. If the text content description of the mobile phone screenshot is successfully obtained, the text content of the picture and the question generated in the previous step are used as input. Then, the instruction requires the large language model to determine whether the question is related to the input text content description; if the text content description in the mobile phone screenshot is not successfully obtained in step one, the picture text information summarized in step two and the question generated in the previous step are used as input, and the instruction is constructed to require the large language model to determine whether the input question is related to the input picture text information. Finally, according to the judgment of the large language model, the questions and answers in the question-answer pairs that are unrelated to the text content in the mobile phone screenshot are filtered out, and high-quality question-answer pair data is finally obtained.

[0126] In general, the above technical solutions conceived by the present invention have the following advantages compared with the prior art:

[0127] Beneficial effects:

[0128] The present invention uses a multimodal large model to extract text content information from mobile phone screenshots, takes text content information and mobile phone screenshots as input, and constructs a plurality of instructions to form an instruction library by building an emphasizing thinking chain, specifying the identity of a large model, and other methods, so as to drive the multimodal large model to generate question-answer pairs with appropriate quantity and format related to the text content of the picture. In addition, the multimodal large model and the large language model are used to self-correct and screen the generated question-answer pairs, so as to ensure the accuracy of the question-answer pairs finally generated. This process of automatically processing mobile phone screenshots using a multimodal large model and a large language model can efficiently generate high-quality question-answer pair training data to help improve the performance of the multimodal large model in mobile digital text understanding tasks.

[0129] More specifically, Figure 4 As shown, the embodiment of the present invention proposes a method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model, comprising the following steps:

[0130] Step 1: Extract the text content of the image, use the multimodal large model to generate a text content description of the mobile phone screenshot image, and obtain content information related to the text in the image.

[0131] Specifically, the process of using a multimodal large model to generate a text description of a mobile phone screenshot is as follows: Figure 5As shown. First, the mobile phone screenshot image needs to be used as the input of the multimodal large model, and the multimodal large model needs to describe the text content in the input image; then the obtained text content description needs to be used as the input of the multimodal large model, and the multimodal large model needs to directly determine whether the text content description contains meaningless repeated parts. If the text content description does not contain repeated parts, the multimodal large model successfully obtains the text content description of the mobile phone screenshot image, and this part of the content will be used in subsequent steps; otherwise, the multimodal large model fails to successfully obtain the text content of the mobile phone screenshot image, and this part of the information will not be used in subsequent steps.

[0132] Step 2: Generate question-answer pairs. Use the mobile phone screenshot and the text content description generated in step 1 as the input of the multimodal large model, and construct instructions to drive the multimodal large model to generate question-answer pairs about the text content of the mobile phone screenshot.

[0133] Specifically, the flowchart for constructing a command-driven multimodal large model to generate question-answer pairs in step 2 is as follows: Figure 6 Used. First, you need to build an instruction library containing multiple instructions. Each time a question-answer pair is generated, a random instruction is extracted from the instruction library for use; then, after the multimodal large model has finished generating the question-answer pair, the generated question-answer pair needs to be used as input again, and the multimodal large model is used to determine whether the number of question-answer pairs in the final generated result reaches the specified number, and check whether the output content meets the preset format requirements. If the number of question-answer pairs in the output result of the multimodal large model does not reach the specified number or the content format output by the multimodal large model does not meet the format requirements, the question-answer pair generated in this round is not used as the final result. Instead, a non-repeated instruction is randomly extracted from the instruction library again to drive the multimodal large model to complete the generation of the question-answer pair. Repeat the above steps until the question-answer pair generated by the multimodal large model meets the number requirements and format requirements. If all the instructions in the instruction library are used and no expected results are obtained, it is directly determined that the generation of the question-answer pair for the image has failed.

[0134] When constructing specific instructions to drive the multimodal large model to generate question-answer pairs, the instructions in the constructed instruction library are divided into two types: instructions for strengthening the thinking chain and instructions for specifying the identity of the multimodal large model.

[0135] In the whole step, when constructing instructions to strengthen the thinking chain, first of all, it should be emphasized in the instructions that the multimodal large model needs to think carefully step by step to complete the task; secondly, it is necessary to emphasize in the instructions that the input image is a screenshot of the mobile phone; then, it is necessary to determine whether the text content description is obtained in step 1. If the text content description of the image is obtained, then this part of the content is added to the instruction as input, and the instruction emphasizes that the text information of the entire image needs to be summarized based on the input text content and the image; otherwise, the instruction should drive the multimodal large model to first describe the content of the entire image, and then summarize the entire image text information based on the input image and the image content generated by the multimodal large model. Then, the instruction should indicate that the multimodal large model needs to generate a certain number of meaningful and correct question-answer pairs based on the summarized image text information and the input image, and ensure that the generated question-answer pairs do not introduce external knowledge as much as possible. Finally, it is necessary to provide an output sample in the instruction, and emphasize that the multimodal large model should output the summarized image text information and the generated question-answer pairs according to the provided sample format.

[0136] In the entire step, when constructing the instruction to specify the identity of the multimodal big model, it is necessary to specify the identity of the big model as a mobile phone assistant or a mobile phone user.

[0137] When specifying the identity of the multimodal large model as a mobile phone assistant, first of all, it is necessary to indicate in the instruction that the input mobile phone screenshot is the mobile phone interface currently used by the mobile phone user, and that the task of the multimodal large model is to help the user use the mobile phone. Secondly, it is necessary to detect whether the text content description of the mobile phone screenshot is successfully obtained in step 1. If the text content description is successfully obtained in step 1, this part of the content is added to the instruction as external knowledge, and it is emphasized that the multimodal large model needs to summarize the text information contained in the entire mobile phone screen based on the text content description and the input mobile phone screenshot; otherwise, the instruction should drive the multimodal large model to read the text content in the mobile phone screenshot as detailed as possible and summarize the entire picture text information. Next, the instruction should indicate that the multimodal large model needs to simulate the problems that the mobile phone user may encounter when using the mobile phone interface based on the picture and the text information of the picture, and as a mobile phone assistant, give corresponding meaningful answers based on the picture without introducing external knowledge to help the user solve the problem. Finally, the instruction should include an output sample, and emphasize that the multimodal large model needs to output the summarized mobile phone screenshot text information and the simulated question and answer pairs according to the provided sample format.

[0138] When the identity of the multimodal large model is specified as a mobile phone user, first of all, it is necessary to indicate in the instruction that the input mobile phone screenshot is the mobile phone interface that the user is using. Secondly, it is necessary to detect whether step one successfully obtains the text content description of the mobile phone screenshot. If step one obtains the text content of the picture, the instruction should indicate that the multimodal large model needs to summarize the text information seen when using this mobile phone screenshot based on the text content description and the input mobile phone screenshot; otherwise, the instruction should indicate that the multimodal large model needs to read the mobile phone screenshot carefully and summarize the text information in detail. Next, the multimodal large model should combine the mobile phone screenshot to simulate the questions that the mobile phone user may have when seeing the mobile phone screen during the use of the mobile phone, and give corresponding, meaningful, and non-external knowledge answers to these questions in combination with the picture and text information. Finally, an output sample needs to be provided in the instruction, and the multimodal large model is required to output the picture text information summarized by the multimodal large model and the generated question and answer pairs according to the provided sample format.

[0139] Step 3: Filter question-answer pairs. Use the image text content generated in step 1 and the question-answer pairs generated in step 2 to build instructions and filter the question-answer pairs generated in step 2.

[0140] Specifically, the flowchart for screening the question-answer pairs generated in step 2 is as follows: Figure 7 First, we need to build a large instruction-driven multimodal model to use the thought chain to self-correct the generated question-answer pairs, and then we need to build a large instruction-driven language model to filter out the generated question-answer pairs that are irrelevant to the text in the mobile phone screenshot.

[0141] Among them, when constructing an instruction-driven multimodal big model to use the thinking chain to self-correct the generated question-answer pairs, first of all, it is necessary to indicate in the instruction that the identity of the multimodal big model is an image analysis expert; secondly, input the mobile phone screenshot image, the image text information output in step 2, and the question in the generated question-answer pair to the multimodal big model; then, the instruction should require the multimodal big model to combine the image and the text information of the image to give the answer corresponding to the input question as concisely as possible, only related to the image text information, and without introducing external knowledge; then, the instruction gives the reason for the answer and indicates that the reason for the answer needs to be able to find the corresponding clue in the input image. Then, the instruction should provide an output sample and emphasize that the multimodal big model should output the question, the answer to the question, and the reason for the answer one by one according to the sample format. The result obtained at the end of this step will replace the answer generated in step 2 for subsequent screening.

[0142] Among them, when constructing instructions to drive the large language model to filter out the generated question-answer pairs that are unrelated to the text in the mobile phone screenshot, it is first necessary to determine whether step one has successfully obtained the text content description of the mobile phone screenshot. If the text content description of the mobile phone screenshot is successfully obtained, the text content of the picture and the question generated in the previous step are used as input. Then, the large language model is required in the instruction to determine whether the question is related to the input text content description; if the text content description in the mobile phone screenshot is not successfully obtained in step one, the picture text information summarized in step two and the question generated in the previous step are used as input, and the construction instruction requires the multimodal large model to determine whether the input question is related to the input picture text information. Finally, according to the judgment of the large language model, the questions and answers in the question-answer pairs that are unrelated to the text content in the mobile phone screenshot are filtered out, and high-quality question-answer pair data is finally obtained.

[0143] Embodiment 3:

[0144] like Figure 8 , is a schematic diagram of the architecture of a device for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to an embodiment of the present invention. The device for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to this embodiment includes one or more processors 21 and a memory 22. Figure 8 A processor 21 is taken as an example.

[0145] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0146] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model in Example 1. The processor 21 executes the method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model by running the non-volatile software programs and instructions stored in the memory 22.

[0147] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0148] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the method for generating mobile phone screenshot text content question-answer pairs based on a multimodal large model in the above-mentioned embodiment 1 is executed.

[0149] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0150] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model, characterized in that: include: In step 201, extract the text content of the picture, and use the multimodal large model to generate a text content description of the mobile phone screenshot picture to obtain content information related to the text in the picture; In step 202, the mobile phone screenshot image and the text content description generated in step 201 are used as inputs of the multimodal large model, and instructions are constructed to drive the multimodal large model to generate question-answer pairs about the text content of the mobile phone screenshot image; In step 203, the question-answer pairs generated in step 202 are screened by using the text content description generated in step 201 and the question-answer pairs generated in step 202 in combination with construction instructions.

2. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 1 is characterized in that: The extraction of the text content of the image and the use of the multimodal large model to generate a text content description of the mobile phone screenshot image to obtain content information related to the text in the image specifically include: Use the mobile phone screenshot as the input of the multimodal large model, so that the multimodal large model can describe the text content in the input image; The obtained text content description is used as the input of the multimodal large model again, so that the multimodal large model can directly determine whether the text content description contains meaningless and repeated text content; If the text content description does not contain any repeated parts, the multimodal large model successfully obtains the text content description of the mobile phone screenshot; otherwise, the multimodal large model fails to successfully obtain the text content description of the mobile phone screenshot.

3. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 1 is characterized in that: The mobile phone screenshot image and the text content description generated in step 201 are used as inputs of the multimodal large model, and instructions are constructed to drive the multimodal large model to generate question-answer pairs about the text content of the mobile phone screenshot image, specifically including: Build an instruction library containing multiple instructions, and randomly extract an instruction from the instruction library each time a question-answer pair is generated; After the multimodal large model finishes generating question-answer pairs, the generated question-answer pairs are used as input again, and the multimodal large model is used to determine whether the number of question-answer pairs in the final generated result reaches the specified number, and to check whether the output content meets the preset format requirements; If the number of question-answer pairs in the output of the multimodal large model does not reach the specified number or the content format of the multimodal large model output does not meet the format requirements, the question-answer pairs generated in this round will not be used as the final result. Instead, a non-repeated instruction will be randomly extracted from the instruction library again to drive the multimodal large model to complete the generation of question-answer pairs. Repeat the above steps until the question-answer pairs generated by the multimodal large model meet the quantity and format requirements; If no expected results are obtained after using all the instructions in the instruction library, it is directly determined that the generation of the question-answer pair for the image has failed.

4. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 3 is characterized in that: The instructions in the instruction library are divided into two types: instructions for strengthening thought chains and instructions for specifying the identity of a multimodal large model.

5. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 4 is characterized in that: The method for constructing the instruction for strengthening the thinking chain specifically includes: Instructions emphasize that large multimodal models require careful step-by-step thinking to complete tasks; Emphasize in the instruction that the input image is a mobile phone screenshot; Determine whether the text content description is obtained in step 201. If the text content description of the picture is obtained, add the content to the instruction as input, and emphasize in the instruction that the text information of the entire picture needs to be summarized based on the input text content and the picture; otherwise, the instruction drives the multimodal large model to first describe the content of the entire picture, and then summarize the text information of the entire picture based on the input picture and the picture content generated by the multimodal large model; The instructions indicate that the multimodal large model generates a preset number of question-answer pairs based on the summarized image text information and the input image; Provide an output example in the instruction, and emphasize that the multimodal large model outputs the summarized image text information and the generated question-answer pairs in the provided sample format.

6. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 4 is characterized in that: The method for constructing the instruction for specifying the identity of the multimodal large model specifically includes: Specify the identity of the big model as a mobile assistant or a mobile user; When the identity of the multimodal large model is specified as a mobile phone assistant, the instruction indicates that the input mobile phone screenshot picture is the mobile phone interface currently being used by the mobile phone user, and indicates that the task of the multimodal large model is to help the user use the mobile phone; detect whether step 201 successfully obtains the text content description of the mobile phone screenshot picture; if step 201 successfully obtains the text content description, then this part of the content is added to the instruction as external knowledge, and emphasizes that the multimodal large model needs to summarize the text information contained in the entire mobile phone screen based on the text content description and the input mobile phone screenshot picture; otherwise, the instruction drives the multimodal large model to read the text content in the mobile phone screenshot and summarize the entire picture text information; the instruction indicates that the multimodal large model needs to simulate the problems encountered by the mobile phone user when using the mobile phone interface based on the picture and the text information of the picture, and as a mobile phone assistant, it gives corresponding answers based on the picture to help the user solve the problem; the instruction includes an output sample, and emphasizes that the multimodal large model needs to output the summarized mobile phone screenshot picture text information and the simulated question and answer pairs according to the provided sample format.

7. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 6 is characterized in that: When the identity of the multimodal large model is specified as a mobile phone user, the instruction indicates that the input mobile phone screenshot picture is the mobile phone interface currently being used by the user; whether the text content description of the mobile phone screenshot picture is successfully obtained in step 201 is detected; if the text content description of the picture is obtained in step 201, the instruction indicates that the multimodal large model summarizes the text information seen when using the mobile phone screenshot according to the text content description and the input mobile phone screenshot; otherwise, the instruction indicates that the multimodal large model reads the mobile phone screenshot and summarizes the text information therein; The multimodal large model combines mobile phone screenshots to simulate the questions that mobile phone users have when they see the mobile phone screen during use, and gives answers to these questions in combination with pictures and text information; Provide an output sample in the instruction, and ask the multimodal large model to output the image text information summarized by the multimodal large model and the generated question and answer pairs according to the provided sample format.

8. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 1, characterized in that: The text content description generated in step 201 and the question-answer pairs generated in step 202 are combined with the construction instructions to screen the question-answer pairs generated in step 202, specifically including: Construct an instruction-driven multimodal large model to self-correct the generated question-answer pairs using thought chains; The instruction-driven large language model is constructed to filter out the generated question-answer pairs that are irrelevant to the text in the mobile phone screenshot.

9. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 8, characterized in that: The construction instruction drives the multimodal large model to self-correct the generated question-answer pairs using the thought chain, specifically including: Input the mobile phone screenshot, the image text information output in step 202, and the questions in the generated question-answer pair into the multimodal large model; The instructions require the multimodal large model to combine the image and the image text information to give an answer corresponding to the input question, and the generated answer is only related to the image text information; The instruction gives the reason for the answer and indicates that the reason for the answer needs to be able to find the corresponding clues in the input picture; Provide output samples in the instructions and emphasize that the multimodal large model outputs questions, answers to questions, and clues to answers one by one in the sample format; The final result of this step is used to replace the answer generated in step 202 for subsequent screening.

10. The method for generating question-answer pairs of mobile phone screenshot text content based on a multimodal large model according to claim 8, characterized in that: The construction instruction drives the large language model to filter out the question-answer pairs generated that are irrelevant to the text in the mobile phone screenshot image, specifically including: Determine whether the text content description of the mobile phone screenshot image is successfully obtained in step 201; If the text content description of the mobile phone screenshot is successfully obtained, the text content description and the question generated in the previous step are used as input; in the instruction, the large language model is required to determine whether the question is related to the input text content description; If the text content description in the mobile phone screenshot image is not successfully obtained in step 201, the image text information summarized in step 202 and the question generated in the previous step are used as input, and the construction instruction requires the large language model to determine whether the input question is related to the input image text information; Based on the judgment of the large language model, questions and answers that are irrelevant to the text content in the question and answer pairs and the mobile phone screenshots are filtered out, and finally high-quality question and answer pair data is obtained.

Images