Question solving large model training method, device, equipment, medium and product

Abstract

The embodiment of the invention provides a problem solving large model training method and device, equipment, a medium and a product, and the scheme can comprise the steps: obtaining a first training sample comprising a first sample problem and a first standard answer, the problem type of the first sample problem can comprise a plurality of graph-containing problem types, the first standard answer comprises a plurality of derivation steps and a final result. Character role information, question type information and guiding information for guiding the pre-trained large model to perform answer derivation according to specific question solving logic corresponding to the question type information are fused into the built prompt word information used for solving the first sample question. Inputting the first cue word information into a pre-trained large model to obtain a first prediction answer; and according to the first standard answer and the first predicted answer, performing fine tuning training on the pre-trained large model to obtain a trained problem solving large model. The perceptual ability of a large model to graphs in topics with graphs is improved, and the accuracy of solving the topics with graphs is improved.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of artificial intelligence, and in particular to a problem-solving large model training method, device, equipment, medium and product. BACKGROUND

[0002] With the rapid iteration of artificial intelligence technology, large language models have gradually penetrated from general fields to education scenarios and become important tools for assisting teaching, homework correction, and personalized tutoring. When large language models process pure text problems, based on the powerful natural language capabilities of large language models, the models can exhibit excellent performance.

[0003] However, when large language models face problems with graphs, such as typical problems with graphs in K12 education scenarios, such as geometric figures, function images, picture listing, picture counting, three-view figures, and statistical charts, the problem requires high graph perception capabilities of large language models. When the graph perception capability of the large language model is insufficient, the information in the graph cannot be effectively utilized to solve the problem, resulting in a low accuracy rate of the large language model for solving problems with graphs.

[0004] Therefore, how to improve the accuracy of solving problems with graphs has become a technical problem to be solved. SUMMARY Therefore, how to improve the accuracy of solving problems with graphs has become a technical problem to be solved.

[0005] According to a first aspect of an embodiment of the present specification, a problem-solving large model training method is provided, comprising: obtaining a first training sample, the first training sample comprising a first sample problem and a first standard answer corresponding to the first sample problem, the problem type of the first sample problem comprising at least one of a geometric figure type, a function image type, a picture listing type, a picture counting type, a three-view figure type, and a statistical chart type; the first standard answer comprising a plurality of deduction steps and a final result; constructing first prompt word information for solving the first sample problem, the first prompt word information comprising at least character role information, problem type information of the first sample problem, and guiding information guiding the pre-training large model to deduce the answer according to the problem-solving logic corresponding to the problem type information, the guiding information comprising information for solving the first sample problem by utilizing known conditions in the text content in the first sample problem and combining the graph information in the first sample problem; inputting the first prompt word information into the pre-training large model to obtain a first predicted answer generated by the pre-training large model for the first sample question; According to the first standard answer and the first predicted answer, the pre-training large model is fine-tuned and trained to obtain a trained problem-solving large model.

[0006] According to a second aspect of the embodiments of the present specification, a problem-solving method for a question is provided, including: obtaining a question of a user, the question type of the question including at least one of a geometric figure type, a function image type, a picture description type, a picture counting type, a three-view type, and a statistical chart type; constructing fourth prompt word information for solving the question, the fourth prompt word information including at least character role information, question type information of the question, and guiding information for guiding the problem-solving large model to derive an answer according to a problem-solving logic corresponding to the question type information, the guiding information including at least information for solving the question by using known conditions in the text content of the question and combining the graphic information in the question; inputting the fourth prompt word information into the problem-solving large model to obtain a target answer generated by the problem-solving large model for the question.

[0007] According to a third aspect of the embodiments of the present specification, a problem-solving large model training device is provided, including: a training sample obtaining module configured to obtain a first training sample, the first training sample including a first sample question and a first standard answer corresponding to the first sample question, the question type of the first sample question including at least one of a geometric figure type, a function image type, a picture description type, a picture counting type, a three-view type, and a statistical chart type; the first standard answer including a plurality of derivation steps and a final result; a first prompt word information constructing module configured to construct first prompt word information for solving the first sample question, the first prompt word information including at least character role information, question type information of the first sample question, and guiding information for guiding a pre-training large model to derive an answer according to a problem-solving logic corresponding to the question type information, the guiding information including at least information for solving the first sample question by using known conditions in the text content of the first sample question and combining the graphic information in the first sample question; a first prompt word information inputting module configured to input the first prompt word information into the pre-training large model to obtain a first predicted answer generated by the pre-training large model for the first sample question; The fine-tuning training module is configured to fine-tune the pre-trained large model according to the first standard answer and the first predicted answer to obtain a trained problem-solving large model.

[0008] According to a fourth aspect of the embodiments of the present specification, a problem-solving device for a target problem is provided, comprising: The question obtaining module is configured to obtain a target problem of a user, and the question type of the target problem includes at least one of a geometric figure type, a function image type, a picture description type, a picture counting type, a three-view type, and a statistical chart type. The fourth prompt information constructing module is configured to construct fourth prompt information for solving the target problem, wherein the fourth prompt information at least includes character role information, question type information of the target problem, and guiding information for guiding the problem-solving large model to derive an answer according to the problem-solving logic corresponding to the question type information, and the guiding information at least includes information for solving the target problem by using known conditions in the text content of the target problem and combining the graphic information in the target problem. The fourth prompt information input module is configured to input the fourth prompt information into the problem-solving large model to obtain a target answer generated by the problem-solving large model for the target problem.

[0009] According to a fifth aspect of the embodiments of the present specification, a computing device is provided, comprising: a memory and a processor; The memory is used to store a computer program or instructions, and the processor is used to execute the computer program or instructions, and the computer program or instructions realize the steps of the above problem-solving large model training method or the problem-solving method for a target problem when executed by the processor.

[0010] According to a sixth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores a computer program or instructions, and the computer program or instructions realize the steps of the above problem-solving large model training method or the problem-solving method for a target problem when executed by the processor.

[0011] According to a seventh aspect of the embodiments of the present specification, a computer program product is provided, comprising a computer program or instructions, and the computer program or instructions realize the steps of the above problem-solving large model training method or the problem-solving method for a target problem when executed by the processor.

[0012] One embodiment of the present specification realizes that the pre-trained large model is fine-tuned and trained by using the first training sample, wherein the sample questions in the first training sample include various types of questions with figures, such as geometric figure questions, function image questions, figure listing questions, figure counting questions, three-view questions and statistical chart questions. In addition, the standard answer corresponding to each sample question can include detailed derivation steps and the final answer. Through the diversified training of questions with figures and the standardized answer guidance, the large model can fully learn and master the problem solving ideas and methods for different types of questions with figures, so as to improve the perception and understanding ability of the large model to the figure information in the questions with figures.

[0013] In addition, the prompt word information constructed for the sample question includes character role information, question type information and guiding information for guiding the pre-trained large model to derive the answer according to the specific problem solving logic corresponding to the question type information. Through this prompt word information construction method, the large model can accurately grasp the problem solving direction and follow the correct problem solving steps, so as to improve the accuracy of the large model in solving the questions with figures. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of a problem solving large model training method provided by one embodiment of the present specification; Figure 2 is a flowchart of a problem solving method for a question title provided by one embodiment of the present specification; Figure 3 is a flowchart of a problem solving method for a question title provided by one embodiment of the present specification; Figure 4 is a structural schematic diagram of a problem solving large model training device provided by one embodiment of the present specification; Figure 5 is a structural schematic diagram of a problem solving device for a question title provided by one embodiment of the present specification; Figure 6 is a structural block diagram of a computing device provided by one embodiment of the present specification. DETAILED DESCRIPTION

[0015] In the following description, many specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced in many different ways beyond the specific embodiments described herein, and it is understood that persons having ordinary skill in the art can make similar modifications without departing from the scope of the present specification.

[0016] The terminology used in this description of one or more embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the description. As used in this description of one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0017] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal sequence, but to distinguish between different types of information. For example, without departing from the scope of one or more embodiments of the description, first can be termed second, and similarly, second can be termed first. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining."

[0018] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the description are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0019] First, the terms involved in one or more embodiments of the description are explained.

[0020] Large Language Model (LLM) is an artificial intelligence model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. The characteristic of LLM is large scale, containing billions of parameters, which can help LLM learn complex patterns in language data. Large language models are usually built based on deep learning architectures, generally composed of transformers.

[0021] Multimodal, Multimodal usually refers to data from multiple different sources or formats. The types of these data sources can include text, images, audio, video, and even more complex data types.

[0022] With picture title, it refers to the title containing visual elements such as images, charts, geometric shapes, etc.

[0023] K12 education refers to the basic education stage from kindergarten to 12th grade, covering all stages of primary school, junior high school, and high school.

[0024] Pre-training refers to the initial training of a large language model on a large dataset, enabling the model to learn general language and visual representation capabilities.

[0025] Fine-tuning refers to further training of a pre-trained large model on specific task data to adapt it to specific application scenarios.

[0026] Reinforcement learning is a machine learning method that optimizes model behavior through interaction with the environment based on reward signals.

[0027] Binary Cross Entropy Loss (BEC LOSS) is a classic loss function used in binary classification tasks in machine learning, mainly used to measure the difference between the model's predicted binary probability distribution and the true label.

[0028] In this specification, a problem-solving large model training method is provided. This specification also relates to a problem-solving method for a specific purpose, a problem-solving large model training device, a problem-solving device for a specific purpose, a computing device, and a computer-readable storage medium. Each of these will be described in detail in the following embodiments.

[0029] Figure 1 This is a flowchart of a problem-solving large model training method provided by an embodiment of this specification. From a hardware perspective, the execution subject of this flow can be a device for training a problem-solving large model, or from a software perspective, the execution subject of this flow can be an application program installed on a device for training a problem-solving large model.

[0030] As shown in Figure 1 , the flow can include the following steps.

[0031] Step 102: Obtain a first training sample, which includes a first sample question and a first standard answer corresponding to the first sample question.

[0032] Wherein, the question type of the first sample question includes at least one of geometric figure type, function image type, picture description type, picture counting type, three-view type, and statistical chart type; the first standard answer includes several deduction steps and a final result.

[0033] In the embodiments of the present specification, the first training sample can be a training sample selected from a preset question bank. The first training sample can include a plurality of training samples. Each training sample can include a first sample question and a first standard answer.

[0034] In the embodiments of the present specification, the first sample question can include any type of question in the K12 education stage that needs to be solved in combination with graphics. The first sample question can include text content and graphic content. The text content can reflect semantic information of the question, such as problem requirements and known conditions. The graphic content is used to visually represent problem-solving elements that are difficult to represent through text in the question, such as structural relationships in geometric figures, image features of function images, quantity distribution of objects, spatial correspondence of three-view diagrams, and visualization of statistical data. These graphic contents are indispensable key basis for solving problems, and together with the text content, they constitute complete question information.

[0035] In actual applications, geometric figures can include regular figures such as triangles, quadrilaterals, and circles, as well as irregular figures. Function images can include linear function images and quadratic function images. Figure-based expression can include figure-based calculation expression and figure-based equation. Figure-based counting can count the total number of objects in the figure or the number of objects marked in the figure. Three-view diagrams can include analysis of top view, bottom view, front view, and side view for three-dimensional figures. Statistical charts can include chart statistics for column charts, line charts, and pie charts.

[0036] In the embodiments of the present specification, the first standard answer can include derivation step identification and final result identification. In actual applications, the derivation step identification and the final result identification can be represented by keywords, such as "detailed explanation" for the derivation step identification and "answer" for the final result identification. The derivation step identification and the final result identification can each occupy a line. Between the derivation step identification and the final result identification, there can be several derivation steps for answering the first sample question, each of which can occupy a line. The line below the final result identification can be the actual answer content.

[0037] In the embodiments of the present specification, at least sample graphical information can be included in the first sample question, the visual detail information serving the solving of the sample graphical information is associated with labeled information in natural language form, and the labeled information is used to reflect the role of the visual detail information in the solving process. Wherein, the visual detail information can include labeled numerical values, labeled functional symbols, labeled structure identifiers, and spatial relationship clues in the graph, etc. The labeled numerical values can include labeled lengths, angles, scales, etc., the labeled functional symbols can include right angle symbols, auxiliary lines, solid and dashed lines, etc., the labeled structure identifiers can include grouping boxes, shaded borders, etc., and the spatial relationship in the graph can include corresponding lines in the three-view diagram, etc. In other words, the visual detail information can be used to exclude decorative visual content unrelated to solving, such as background texture, non-functional color filling, etc.

[0038] For example, if the sample question is a geometry type question, and the line segment AB is labeled with a 3cm label, the labeled information added for the visual detail information 3cm can be "the length of the line segment AB is 3cm". Through this sample graph with labeled information, in the fine-tuning training, the visual details in the graph can be displayed to the large language model in natural language form, so that the large language model can rely on its powerful natural language understanding ability to accurately capture the role of the visual detail information in the solving process, and then the perception ability of the large language model to the key visual details in the graph can be strengthened, so as to improve the performance of the large language model in solving the graph question, and finally the accuracy of solving the graph question can be improved.

[0039] Step 104: constructing first prompt word information for solving the first sample question.

[0040] Wherein, the first prompt word information at least includes character role information, question type information of the first sample question, and guide information guiding the pre-trained large model to derive answers according to the solving logic corresponding to the question type information, and the guide information at least includes information for solving the first sample question by using known conditions in the text content in the first sample question and combining the graphical information in the first sample question.

[0041] In the embodiments of the present specification, the first prompt word information can be information generated based on the input information of the user and the prompt word template pre-stored by the system. In actual application, the system can configure a dedicated prompt word template for the first sample question of different question types, that is, one prompt word template for one question type, and establish an association relationship between the question type and the prompt word template.

[0042] The prompt word template can include reusable general information and replaceable unique information, wherein the general information can include character role information, topic type information, and guidance information, etc. Since one prompt word template can be configured for one topic type, the character role information and the topic type information can be reused in the prompt word template. Guidance information can also be reused for one topic type, which can be configured for a set of questions of the type. The character role information is used to reflect the background knowledge of the large model in answering the sample question. The guidance information can be information that guides the large model to derive the answer according to a certain problem solving logic, wherein the problem solving logic can be a fixed problem solving logic step determined according to the topic type, or a problem solving logic step determined according to the known conditions and graphical information in the sample question, which is not limited. The replaceable information can refer to the content of the question. Since the known conditions in the question are different under the same topic type, the actual content of the question will not be the same, and therefore the actual content of the question can be replaceable. It should be noted that the guidance information can also be configured as replaceable information, which is not limited.

[0043] The input information of the user can only include the first sample question, or the input information of the user can also include the first sample question and summary information for indicating that the pre-trained large model answers the first sample question, such as "please answer this question" and the like. The pre-trained large model can be a model trained on a basis large model using a plurality of types of questions in the K12 education stage. After the system receives the input information of the user, the semantic analysis of the text content of the first sample question can be used to determine the topic type corresponding to the sample question, and then based on the association between the topic type and the prompt word template, the prompt word template corresponding to the topic type can be determined, and then the prompt word information for answering the sample question can be generated based on the prompt word template. In this way, standard prompt word information for answering the sample question can be generated without the user having the ability to provide standard prompt word information, thereby improving the accuracy of the model in answering the sample question.

[0044] In the embodiments of the present specification, the system can be configured with a corresponding exclusive prompt word template for different topic types, and the corresponding guidance information can be different. In actual applications, the system can pre-configure corresponding guidance information for each topic type, and then construct corresponding prompt word information.

[0045] In order to facilitate those skilled in the art to understand the present scheme, one or more specific examples are proposed in the present specification for the constructed first prompt word information.

[0046] Example 1: The question type is geometry. For geometry questions, the role information in the prompt template could be: You are a math expert who solves geometry problems. The guiding information in the prompt template could be: Please analyze the semantic information of the text in the question to obtain the known conditions. Analyze the graphic information in the question. If the graphic in the question is a regular graphic, please use the formula corresponding to regular graphics to calculate. If the graphic in the question is an irregular graphic, please add auxiliary lines to divide the original graphic into smaller regular graphics, calculate for each smaller regular graphic, and summarize the calculation results to obtain the answer for the original graphic. The final first prompt information could be: Question content + You are a math expert who solves geometry problems + This is a question about geometry + Please analyze the semantic information of the text in the question to obtain the known conditions. Analyze the graphic information in the question. If the graphic in the question is a regular graphic, please use the formula corresponding to regular graphics to calculate. If the graphic in the question is an irregular graphic, please add auxiliary lines to divide the original graphic into smaller regular graphics, calculate for each smaller regular graphic, and summarize the calculation results to obtain the answer for the original graphic.

[0047] Example 2: The question type is function graph. For function graph questions, the role information in the prompt template could be: You are a math expert who solves function graph questions. The guiding information in the prompt template could be: Please analyze the semantic information of the text content in the question, obtain the known conditions, analyze the function type corresponding to the function graph in the question, and use the function knowledge related to that function type to solve the function graph question. The final first prompt information could be: Question content + You are a math expert who solves function graph questions + This is a question about function graph types + Please analyze the semantic information of the text content in the question, obtain the known conditions, analyze the function type corresponding to the function graph in the question, and use the function knowledge related to that function type to solve the function graph question.

[0048] Example 3: The question type is "Picture-based equation writing". For picture-based equation writing questions, the character information in the prompt template could be: "You are a math expert who solves picture-based equation writing questions." The guiding information in the prompt template could be: "Please analyze the quantitative relationships between the objects in the picture, determine the matching operation type, and write the corresponding calculation formula based on the quantitative relationship and operation type." The final first prompt information could be: "Question content + You are a math expert who solves picture-based equation writing questions + This is a picture-based equation writing question + Please analyze the quantitative relationships between the objects in the picture, determine the matching operation type, and write the corresponding calculation formula based on the quantitative relationship and operation type."

[0049] Example 4: The question type is picture counting. For picture counting questions, the character information in the prompt template could be: You are a math expert who solves picture counting problems. The guiding information in the prompt template could be: Please identify the types of objects in the picture, count the quantity of each type of object, and generate the answer based on the counted quantity. The final first prompt information could be: Question content + You are a math expert who solves picture counting problems + This is a picture counting question + Please identify the types of objects in the picture, count the quantity of each type of object, and generate the answer based on the counted quantity.

[0050] Example 5: The question type is three-view drawing. For three-view drawing questions, the role information in the prompt template could be: You are a math expert who solves three-view drawing questions. The guiding information in the prompt template could be: Please analyze the semantic information of the text content in the question, obtain the known conditions, analyze the viewing direction of the figure in the question, determine the solid and dashed lines in the figure in the viewing direction, and combine the solid and dashed lines to solve the problem. The final first prompt information could be: Question content + You are a math expert who solves three-view drawing questions + This is a question about three-view drawing + Please analyze the semantic information of the text content in the question, obtain the known conditions, analyze the viewing direction of the figure in the question, determine the solid and dashed lines in the figure in the viewing direction, and combine the solid and dashed lines to solve the problem.

[0051] Example 6: The question type is statistical charts. For statistical chart questions, the role information in the prompt template could be: You are a math expert who solves statistical chart questions. The guiding information in the prompt template could be: Please analyze the semantic information of the text in the question to obtain the known conditions. Analyze the type of statistical chart in the question, such as bar charts, line charts, pie charts, etc. Determine the focus of the question based on the chart type. For example, bar charts focus on quantity comparison, line charts focus on changing trends, and pie charts focus on proportions. Extract effective data from the chart based on the focus of the question, and solve the problem based on the extracted effective data. The final first prompt information can be: Question content + You are a math expert who solves statistical chart problems + This is a question about statistical chart types + Please analyze the semantic information of the text content in the question, obtain the known conditions, analyze the chart type in the question, such as bar chart, line chart, pie chart, etc., and determine the focus of the question based on the chart type. For example, bar charts focus on quantity comparison, line charts focus on changing trends, pie charts focus on proportion relationships, etc. Based on the focus of the question, extract effective data from the chart, and solve the problem based on the extracted effective data.

[0052] It should be noted that Examples 1 to 6 above are merely illustrative examples and are not intended to limit the first prompt word information corresponding to the question type. In practical applications, the first prompt word information can also take other forms, and this specification does not impose specific limitations on them.

[0053] Step 106: Input the first prompt word information into the pre-trained large model to obtain the first predicted answer generated by the pre-trained large model for the first sample question.

[0054] Step 108: Based on the first standard answer and the first predicted answer, fine-tune the pre-trained large model to obtain the trained problem-solving large model.

[0055] In the embodiments of this specification, the methods for fine-tuning training a pre-trained large model may include the following. For the pre-trained large model, the model parameters are randomly initialized, and the first prompt word information obtained in the above steps is input into the pre-trained large model. After receiving the input data, the pre-trained large model outputs the first predicted answer corresponding to the first sample question in each training sample. The loss function value between the first standard answer and the first predicted answer is calculated, and then the parameters of the pre-trained large model are adjusted according to the loss function value until the preset training termination condition is met, resulting in a fine-tuned problem-solving large model. For example, based on the derivation steps and final result in the answer, the first standard answer and the first predicted answer are mapped to the same probability space to obtain the label probability corresponding to the first standard answer and the prediction probability corresponding to the first predicted answer. Then, based on the difference between the label probability and the prediction probability, an appropriate loss function (such as cross-entropy loss) is used to calculate the loss value. Based on this loss value, the gradient descent algorithm is used for backpropagation to update the learnable parameters inside the pre-trained large model. This process is repeated until the preset training termination condition is met, resulting in a trained problem-solving large model.

[0056] Optionally, the preset training termination condition may include reaching the maximum number of iterations in the model training, or alternatively, the preset training termination condition may include no further performance improvement on the model training validation set.

[0057] It should be understood that the order of some steps in the methods described in one or more embodiments of this specification may be interchanged according to actual needs, or some steps may be omitted or deleted.

[0058] Figure 1The method described above uses a first training sample to fine-tune the pre-trained large model. This first training sample includes various types of questions with graphs, such as geometric figures, function graphs, equations based on graphs, counting problems based on graphs, three-view diagrams, and statistical charts. Furthermore, the standard answer for each sample question includes detailed derivation steps and the final answer. Through this diverse training with graph-based questions and standardized answer guidance, the large model can fully learn and master the problem-solving strategies and methods for different types of graph-based questions, thereby improving its ability to perceive and understand graphical information in such questions.

[0059] Furthermore, the prompts constructed for the sample questions incorporate character information, question type information, and guidance information to help the pre-trained large model deduce the answer according to the specific problem-solving logic corresponding to the question type information. This method of constructing prompts effectively guides the large model to accurately grasp the problem-solving direction and follow the correct steps, thereby improving the accuracy of the large model in solving questions with images.

[0060] based on Figure 1 In addition to the method described herein, this specification also provides some specific implementation methods of the method, which will be described below.

[0061] In the embodiments of this specification, an embodiment for training and generating pre-trained large models is also proposed.

[0062] Optionally, before obtaining the first training sample, the method may further include: obtaining a second training sample, the second training sample including a second sample question and a second standard answer corresponding to the second sample question, the second sample question including multiple types of questions in the education stage, the education stage including at least one of preschool education, primary education, secondary education and university education; constructing second prompt word information for answering the second sample question, the second prompt word information including at least the question type information of the second sample question; inputting the second prompt word information into the multimodal large model to obtain a second predicted answer generated by the multimodal large model for the second sample question; and training the multimodal large model according to the second standard answer and the second predicted answer to obtain the pre-trained large model.

[0063] In this embodiment of the specification, the second training sample can be a training sample selected from a preset question bank. The second training sample can include multiple training samples, and each training sample can include a second sample question and a second standard answer. The second sample question can include various types of questions from different subjects in different educational stages, including preschool education, primary education, secondary education, and university education. The subjects can include mathematics, Chinese, English, physics, chemistry, biology, history, geography, and moral education, etc. The various types of questions can include questions with pictures and questions without pictures.

[0064] The second standard answer can include formatted information and result information that conforms to the subject-specific answering standards. For example, for math application questions, the formatted information can include necessary reasoning steps that conform to the problem-solving logic, key calculation formulas, unified units, and a conclusion statement guided by the word "answer"; for Chinese reading comprehension questions, the formatted information can include the keyword "answer", and if the question requires a layered answer, the formatted information can also include bullet points, such as 1, 2, 3, etc.

[0065] In the embodiments of this specification, the explanation of the second prompt word information can refer to the explanation of the first prompt word information described above, and the explanation of training the multimodal large model can refer to the explanation of training the pre-trained large model described above, and will not be repeated here. The multimodal large model can be the basic large model.

[0066] To improve the accuracy of the large problem-solving model, further reinforcement training can be performed on the large problem-solving model generated by fine-tuning training.

[0067] Optionally, after fine-tuning the pre-trained large model, the process may further include: acquiring user test questions, wherein the test questions are of at least one type, including geometric figures, function graphs, graph equations, graph counting, three-view diagrams, and statistical charts; constructing third prompt words for answering the test questions; inputting the third prompt words into the solution large model to obtain a third predicted answer generated by the solution large model for the test questions; acquiring the user's evaluation information for the third predicted answer; and performing reinforcement training on the solution large model based on the evaluation information to obtain the reinforced solution large model.

[0068] In this embodiment of the specification, the test questions can be image-based questions provided by the user. The explanations of the test questions and the third prompt word information can refer to the explanations of the first sample questions and the first prompt word information described above, and will not be repeated here.

[0069] In the embodiments of this specification, the evaluation information of the user for the third test answer can be the evaluation information given by the user based on the standard answer for the test answer. The evaluation information may include whether the format of the test answer is qualified, whether the derivation steps of the test answer are complete, and whether the final result of the test answer is correct.

[0070] In the embodiments of this specification, the relevant learnable parameters of the problem-solving model can be adjusted based on the evaluation information provided by the user for the third test answer output by the problem-solving model, and the above-mentioned user evaluation-parameter adjustment process can be repeated until the preset reinforcement training termination conditions are met, such as the performance index of the problem-solving model reaching the standard or the number of iterations reaching the preset threshold, so as to obtain the problem-solving model after reinforcement training.

[0071] In this embodiment, a basic large-scale model can first be pre-trained based on massive amounts of general subject data to obtain a pre-trained large-scale model. Then, K12 problems with images are used to fine-tune the pre-trained large-scale model, resulting in an initial problem-solving large-scale model. Finally, user feedback on the test answers of the initial problem-solving large-scale model is used to optimize its learnable parameters, resulting in an enhanced problem-solving large-scale model. This progressive training method can improve the accuracy of the problem-solving large-scale model in solving problems with images.

[0072] Figure 2 This is a flowchart illustrating a method for solving a target problem according to an embodiment of this specification. From a hardware perspective, the entity executing this process can be a device used to solve the target problem; or, from a software perspective, the entity executing this process can be an application program installed on the device used to solve the target problem.

[0073] like Figure 2 As shown, the process may include the following steps.

[0074] Step 202: Obtain the user's target question.

[0075] The target question types include at least one of the following: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type.

[0076] In this embodiment of the specification, the user can be a student, parent, or educator, etc. The target question can be a picture-based question encountered by the user during the learning process, such as a question involving calculating the area of ​​a shape encountered while completing homework. The explanation of the question type for the target question can refer to the explanation of the question type for the first sample question mentioned above, and will not be repeated here. Step 204: Construct fourth prompt information for answering the target question.

[0077] The fourth prompt information includes at least character information, the question type information of the target question, and guidance information that guides the problem-solving model to deduce the answer according to the problem-solving logic corresponding to the question type information. The guidance information includes at least information on solving the target question by using the known conditions in the text content of the target question and combining them with the graphic information in the target question.

[0078] In the embodiments of this specification, the explanation of the fourth prompt word information can be referred to the explanation of the first prompt word information mentioned above, and will not be repeated here.

[0079] Step 206: Input the fourth prompt word information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question.

[0080] In the embodiments of this specification, the large problem-solving model can be a model generated using the above-described large problem-solving model training method.

[0081] In this embodiment of the specification, the fourth prompt word information, adapted to the problem-solving requirements of the target question, is input into the problem-solving model obtained through the aforementioned training. Based on the problem-solving logic guided by the fourth prompt word information, the problem-solving model performs reasoning operations to generate the target answer for the target question. The problem-solving model can be a model obtained through pre-training and fine-tuning, or a model obtained through pre-training, fine-tuning, and reinforcement training.

[0082] Figure 2 The method described above utilizes a large-scale problem-solving model, after rigorous training, to answer target questions encountered by users. Because this model learns and masters problem-solving strategies and methods for different types of image-based questions during the training phase, it enhances its ability to perceive and understand graphic information within these questions. Furthermore, the prompts constructed for the target questions incorporate character information, question type information, and guidance information to direct the model to deduce the answer according to the specific problem-solving logic corresponding to the question type. This method of constructing prompts effectively guides the model to accurately grasp the problem-solving direction and follow the correct steps, thereby improving the accuracy of its solutions to image-based questions.

[0083] based on Figure 2 In addition to the method described herein, this specification also provides some specific implementation methods of the method, which will be described below.

[0084] The target question provided by the user can be a question embedded in a target image. The target image can be an image obtained by the user taking a picture of the target question, or it can be an image obtained by the user taking a screenshot of the target question.

[0085] In the embodiments of this specification, the large-scale problem-solving model used to answer the target question can be a problem-solving module in a one-stop problem-solving system, wherein the problem-solving system may include an image morphology detection module, an image content detection module, a problem-solving module, and an answer detection module.

[0086] Figure 3 This is a flowchart illustrating a problem-solving system based on an embodiment of this specification, illustrating a process for solving a target problem. It includes the following steps.

[0087] Step 302: Input the target image containing the target title into the image morphology detection module.

[0088] In the embodiments described in this specification, the image morphology detection module can be a functional module used to receive the target image and detect whether the image morphology of the target image is qualified. In practical applications, the system will only construct corresponding prompt information for the target question when the image morphology of the target image meets the preset requirements.

[0089] Optionally, the construction of the fourth prompt information for answering the target question may specifically include: checking the image morphology of the target image to obtain a first check result; the image morphology includes at least one of image size, image format, and image resolution; if the first check result reflects that the image morphology of the target image is qualified, then the fourth prompt information for answering the target question is constructed.

[0090] In the embodiments of this specification, the detected image shape may include at least one of the detected image size, detected image format, and detected image resolution. An image shape being in a qualified state may include at least one of scenario 1, scenario 2, and scenario 3. Scenario 1: If the image size meets the preset size requirement, the image shape is considered qualified. Scenario 2: If the image format meets the preset format requirement, the image shape is considered qualified. Scenario 3: If the image resolution meets the preset resolution requirement, the image shape is considered qualified.

[0091] In practical applications, if the target image is in a qualified state, prompts can be generated based on the target question within the image to answer the question, triggering the large-scale problem-solving model to solve it. If the target image is in a substandard state, the user is notified of the substandard image and prompted to resubmit a qualified one. Solving the target question only when the target image is in a qualified state avoids interference from substandard image quality, thus improving the accuracy of the large-scale problem-solving model's solutions.

[0092] Step 304: Input the target image into the image content detection module.

[0093] In the embodiments of this specification, the image content detection module can refer to a functional module used to detect whether the image content within a target image is qualified. The input terminal of the image content detection module can be connected to the output terminal of the image morphology detection module, forming a serial connection. Alternatively, the image content detection module and the image morphology detection module can also be connected in parallel, that is, the image content detection module can directly receive the target image provided by the user, without limitation.

[0094] In practical applications, the system will only construct corresponding prompt information for the target question when the image content of the target image meets the preset requirements.

[0095] Optionally, the target question is a question contained in the target image. The construction of the fourth prompt information for answering the target question may specifically include: checking the image content of the target image to obtain a second check result. The image content includes at least one of image clarity, integrity of question information within the image, readability of text within the image, and integrity of graphic elements within the image. If the second check result reflects that the image content of the target image is qualified, then the fourth prompt information for answering the target question is constructed.

[0096] In the embodiments of this specification, detecting image content may include at least one of detecting image clarity, detecting the completeness of question information within the image, detecting the readability of text within the image, and detecting the completeness of graphic elements within the image. The image content being in a qualified state may include at least one of scenarios 4, 5, 6, and 7. Scenario 4: If the image clarity meets a preset clarity requirement, the image content is considered qualified. Scenario 5: If the completeness of question information within the image meets a first preset requirement, such as the question information containing both known and unknown conditions, the image content is considered qualified. Scenario 6: If the readability of text within the image meets a second preset requirement, such as all text within the image being readable natural language, the image content is considered qualified. Scenario 7: If the completeness of graphic elements within the image meets a third preset requirement, such as the graphic elements within the image being unobstructed and undamaged, the image content is considered qualified.

[0097] In practical applications, if the target image content is acceptable, prompts can be generated based on the target question within the image to answer it, triggering the large-scale problem-solving model to solve the problem. If the target image content is unacceptable, the user is notified of the unacceptability and prompted to resubmit a acceptable image. Only when the target image content is acceptable will the model attempt to solve the target question, thus avoiding interference from unacceptable image content and improving the accuracy of the model's solutions.

[0098] In practical applications, the target image provided by the user can first be input into the image shape detection module. After receiving the target image, the image shape detection module will detect the image shape of the target image. If the image shape detection module detects that the image shape of the target image is qualified, the image shape detection module will send the target image to the image content detection module. After receiving the target image, the image content detection module will detect the image content of the target image. If the image content detection module detects that the image content of the target image is qualified, the target image will be sent to the problem-solving module.

[0099] Step 306: If the image shape and / or image content of the target image are in a qualified state, then input the target image into the problem-solving module.

[0100] In this embodiment, the problem-solving module can be a functional module for answering a target question within a target image. The input of the problem-solving module can be connected to the output of the image morphology detection module, or the input of the problem-solving module can be connected to the output of the image content detection module. The problem-solving module can be configured with a large-scale problem-solving model, which can be a model obtained using the aforementioned large-scale problem-solving model training method. After receiving the target image, the large-scale problem-solving model can parse the target question from the target image, construct prompt information for answering the target question, and then answer the target question based on the prompt information to obtain the target answer.

[0101] Step 308: Input the target answer generated for the target question in the target image into the answer detection module.

[0102] In this embodiment of the specification, the answer detection module can be a functional module used to detect whether the target answer is qualified. The input of the answer detection module can be connected to the output of the problem-solving module. If the target answer is qualified, the target answer will be output to the user; if the target answer is unqualified, the user will be prompted to re-trigger the problem-solving process.

[0103] Methods for detecting whether a target answer is acceptable may include detecting whether the target answer contains keywords that do not conform to the semantic information of the target question.

[0104] Optionally, after inputting the fourth prompt word information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question, the step may further include: determining whether there are keywords in the target answer that do not conform to the semantic information of the target question, and obtaining a first determination result; if the first determination result indicates that there are no keywords in the target answer that do not conform to the semantic information of the target question, then outputting the target answer to the user.

[0105] In the embodiments of this specification, keywords can be words composed of consecutive characters retrieved from the target answer, and there can be one or more keywords retrieved from the target answer.

[0106] In practical applications, methods for determining whether there are keywords in the target answer that do not conform to the semantic information of the target question can include: determining the first semantic information of the keywords extracted from the target answer, determining the second semantic information of the target question, and determining the semantic similarity between the first semantic information and the second semantic information. If the semantic similarity is less than a preset threshold, it means that there are keywords in the target answer that do not conform to the semantic information of the target question; if the semantic similarity is greater than or equal to the preset threshold, it means that there are no keywords in the target answer that do not conform to the semantic information of the target question.

[0107] For example, if the target question is to determine the bottom view in three views, and the target answer contains the keyword "top view", then it can be said that the target answer contains keywords that do not conform to the semantic information of the target question.

[0108] In this embodiment of the specification, if the target answer does not contain any keywords that do not conform to the semantic information of the target question, the target answer is output to the user; if the target answer contains keywords that do not conform to the semantic information of the target question, the user is given feedback of the failure to answer the question, so as to prompt the user to re-trigger the answering process for the target question.

[0109] Other methods for checking whether a target answer is acceptable may include checking whether the answer format meets the requirements.

[0110] Optionally, after inputting the fourth prompt information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question, the method may further include: determining whether the format of the target answer conforms to the answer format requirements preset for the target question, and obtaining a second determination result; if the second determination result indicates that the format of the target answer conforms to the answer format requirements preset for the target question, then outputting the target answer to the user.

[0111] In this embodiment of the specification, for each type of target question, the answer format requirements for the corresponding target answer can be predefined. For example, for mathematical questions with diagrams, the answer format requirements may include at least one of the following: reasoning steps, calculation formulas, unified units, and a conclusion introduced by the word "Answer". The answer format of the target answer is checked against the predefined answer format requirements for the target question. If the format of the target answer conforms to the predefined answer format requirements for the target question, the target answer is output to the user; if the format of the target answer does not conform to the predefined answer format requirements for the target question, feedback is given to the user indicating a failure to solve the problem, prompting the user to re-trigger the solution process for the target question.

[0112] Other methods for detecting whether the target answer is acceptable include judging whether the confidence level generated by the large problem-solving model for the target answer meets the preset requirements.

[0113] Optionally, after inputting the fourth prompt word information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question, the method may further include: obtaining the confidence level generated synchronously by the problem-solving model when generating the answer, wherein the confidence level is used to quantify the degree of confidence of the problem-solving model that the target answer is the correct answer to the target question; determining whether the confidence level is greater than or equal to a preset threshold to obtain a third judgment result; if the third judgment result indicates that the confidence level is greater than or equal to the preset threshold, then outputting the target answer to the user.

[0114] In the embodiments of this specification, confidence level can be a quantitative indicator calculated and output simultaneously by the problem-solving model while generating the target answer. Confidence level is the degree of confidence that the problem-solving model believes the target answer is the correct answer to the target question, converted into a measurable numerical value, such as a value in the range of 0-100 points or a value in the range of 0-1. The higher the value, the greater the confidence that the problem-solving model believes its generated target answer is correct; the lower the value, the less confidence the problem-solving model believes its generated target answer is correct.

[0115] In the embodiments of this specification, if the confidence level generated by the problem-solving model when generating the target answer is greater than or equal to a preset threshold, the target answer is output to the user; if the confidence level is less than the preset threshold, the user is fed back a result of failure to solve the problem, so as to prompt the user to re-trigger the solution process for the target question.

[0116] Step 310: If the target answer meets the preset requirements, then output the target answer to the user.

[0117] In this embodiment of the specification, the target answer can be output to the user when the target answer meets at least one of conditions 1, 2, and 3. Condition 1: The target answer does not contain keywords that do not conform to the semantic information of the target question; Condition 2: The format of the target answer conforms to the answer format requirements preset for the target question; Condition 3: The confidence level generated synchronously for the target answer is greater than or equal to a preset threshold.

[0118] In practical applications, large-scale problem-solving models may not need to generate confidence scores for the target answer when generating the target answer. This reduces the workload of detecting the target answer and increases the speed at which the large-scale problem-solving model outputs the target answer to the user.

[0119] In the embodiments of this specification, the problem-solving system for answering target questions in target images can automatically perform comprehensive detection on user-uploaded target images, covering compliance checks of image shape and content to ensure that the input image meets the problem-solving requirements. After passing the detection, the problem-solving system can extract target question information from the target image, automatically construct suitable problem-solving hints, call the built-in problem-solving model to answer the target question, and generate the target answer. The problem-solving system can also simultaneously perform corresponding operations such as semantic consistency verification, format compliance check, and confidence assessment of the target answer. The entire process, from target image detection to target question answering and target answer verification, is completed internally without the need to call external tools or modules. This enables a one-stop service from target image input to target question answering to target answer output, simplifying the operation process and ensuring the continuity and security of data processing, thereby improving the efficiency of answering target questions.

[0120] Corresponding to the above method embodiments, this specification also provides embodiments of a large-scale problem-solving model training device. Figure 4 This is a schematic diagram of the structure of a large-scale problem-solving model training device provided in one embodiment of this specification. Figure 4 As shown, the device may include: The training sample acquisition module 402 is configured to acquire a first training sample, which includes a first sample question and a first standard answer corresponding to the first sample question. The question type of the first sample question includes at least one of the following: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type. The first standard answer includes several derivation steps and the final result. The first prompt word information construction module 404 is configured to construct first prompt word information for answering the first sample question. The first prompt word information includes at least character information, question type information of the first sample question, and guidance information that guides the pre-trained large model to deduce the answer according to the problem-solving logic corresponding to the question type information. The guidance information includes at least information that uses the known conditions in the text content of the first sample question and combines the graphic information in the first sample question to answer the first sample question. The first prompt word information input module 406 is configured to input the first prompt word information into the pre-trained large model to obtain the first predicted answer generated by the pre-trained large model for the first sample question. The fine-tuning training module 408 is configured to fine-tune the pre-trained large model based on the first standard answer and the first predicted answer to obtain the trained problem-solving large model.

[0121] Optionally, the device may further include: The second training sample acquisition module is configured to acquire a second training sample, which includes a second sample question and a second standard answer corresponding to the second sample question. The second sample question includes multiple types of questions in the education stage, which includes at least one of the preschool education stage, primary education stage, secondary education stage and university education stage.

[0122] The second prompt word information construction module is configured to construct second prompt word information for answering the second sample question, wherein the second prompt word information includes at least the question type information of the second sample question.

[0123] The second prompt word information input module is configured to input the second prompt word information into the multimodal large model to obtain the second predicted answer generated by the multimodal large model for the second sample question.

[0124] The multimodal large model training module is configured to train the multimodal large model based on the second standard answer and the second predicted answer to obtain the pre-trained large model.

[0125] Optionally, the device may further include: The test question acquisition module is configured to acquire the user's test questions. The test question types include at least one of the following: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type.

[0126] The third prompt word information construction module is configured to construct third prompt word information for answering the test questions.

[0127] The third prompt word information input module is configured to input the third prompt word information into the solution model to obtain the third predicted answer generated by the solution model for the test question.

[0128] The evaluation information acquisition module is configured to acquire the user's evaluation information for the third predicted answer.

[0129] The problem-solving large model training module is configured to perform reinforcement training on the problem-solving large model based on the evaluation information, so as to obtain the reinforcement-trained problem-solving large model.

[0130] The above is a schematic scheme of a problem-solving large model training device according to this embodiment. It should be noted that the technical solution of this problem-solving large model training device and the technical solution of the problem-solving large model training method described above belong to the same concept. For details not described in detail in the technical solution of the problem-solving large model training device, please refer to the description of the technical solution of the problem-solving large model training method described above.

[0131] Corresponding to the above method embodiments, this specification also provides embodiments of a problem-solving apparatus for the target problem. Figure 5 This is a schematic diagram of a problem-solving device for a target problem provided in one embodiment of this specification. Figure 5 As shown, the device may include: The question acquisition module 502 is configured to acquire the user's target question, and the target question type includes at least one of the following: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type.

[0132] The fourth prompt information construction module 504 is configured to construct fourth prompt information for answering the target question. The fourth prompt information includes at least character information, question type information of the target question, and guidance information that guides the problem-solving model to deduce the answer according to the problem-solving logic corresponding to the question type information. The guidance information includes at least information on how to answer the target question by using known conditions in the text content of the target question and combining them with the graphic information in the target question.

[0133] The fourth prompt word information input module 506 is configured to input the fourth prompt word information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question.

[0134] Optionally, the target title is a title contained in the target image.

[0135] Optionally, the fourth prompt word information construction module 504 may specifically include: An image morphology inspection unit is configured to inspect the image morphology of the target image and obtain a first inspection result; the image morphology includes at least one of image size, image format, and image resolution.

[0136] The first and fourth prompt word information construction unit is configured to construct fourth prompt word information for answering the target question if the first inspection result reflects that the image morphology of the target image is in a qualified state.

[0137] Optionally, the fourth prompt word information construction module 504 may specifically include: The image content inspection unit is configured to inspect the image content of the target image and obtain a second inspection result. The image content includes at least one of image clarity, integrity of title information within the image, readability of text within the image, and integrity of graphic elements within the image.

[0138] The second and fourth prompt word information construction unit is configured to construct fourth prompt word information for answering the target question if the second inspection result reflects that the image content of the target image is in a qualified state.

[0139] Optionally, the device may further include: The first judgment module is configured to determine whether there are keywords in the target answer that do not conform to the semantic information of the target question, and obtain a first judgment result.

[0140] The first output module is configured to output the target answer to the user if the first judgment result indicates that there are no keywords in the target answer that do not conform to the semantic information of the target question.

[0141] Optionally, the device may further include: The second judgment module is configured to judge whether the format of the target answer conforms to the answer format requirements preset for the target question, and obtain a second judgment result.

[0142] The second output module is configured to output the target answer to the user if the second judgment result indicates that the format of the target answer conforms to the answer format requirements preset for the target question.

[0143] Optionally, the device may further include: The confidence level acquisition module is configured to acquire the confidence level generated synchronously by the problem-solving model when generating the answer. The confidence level is used to quantitatively reflect the degree of confidence of the problem-solving model in that the target answer is the correct answer to the target question.

[0144] The third judgment module is configured to determine whether the confidence level is greater than or equal to a preset threshold, and obtain a third judgment result.

[0145] The third output module is configured to output the target answer to the user if the third judgment result indicates that the confidence level is greater than or equal to the preset threshold.

[0146] The above is an illustrative scheme of a problem-solving device for a target problem according to this embodiment. It should be noted that the technical solution of this problem-solving device for a target problem and the technical solution of the problem-solving method for a target problem described above belong to the same concept. For details not described in detail in the technical solution of the problem-solving device for a target problem, please refer to the description of the technical solution of the problem-solving method for a target problem described above.

[0147] Figure 6 This is a structural block diagram of a computing device according to one embodiment of this specification. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0148] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0149] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0150] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0151] The processor 620 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned large-scale problem-solving model training method or the problem-solving method for the target problem.

[0152] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the large-scale problem-solving model training method or the problem-solving method for the target problem, so the description is relatively simple. Relevant parts can be referred to the descriptions of the large-scale problem-solving model training method or the problem-solving method for the target problem.

[0153] An embodiment of this specification also provides a computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the above-described problem-solving large model training method or the problem-solving method for the target problem.

[0154] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are described simply because they are substantially similar to the large-scale problem-solving model training method or the problem-solving method for the target problem. Relevant details can be found in the descriptions of the large-scale problem-solving model training method or the problem-solving method for the target problem.

[0155] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described problem-solving large model training method or problem-solving method for a target problem.

[0156] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the above-mentioned problem-solving large model training method or problem-solving method for the target problem. For details not described in detail in the technical solution of the computer program product, please refer to the description of the above-mentioned problem-solving large model training method or problem-solving method for the target problem.

[0157] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0158] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0159] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0160] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0161] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

[0162] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0163] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0164] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0165] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0171] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0172] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0173] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0174] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0176] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for training a large-scale problem-solving model, characterized in that, The method includes: Obtain a first training sample, which includes a first sample question and a first standard answer corresponding to the first sample question. The question type of the first sample question includes at least one of the following: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type. The first standard answer includes several derivation steps and the final result. Construct first prompt word information for answering the first sample question. The first prompt word information includes at least character information, question type information of the first sample question, and guidance information that guides the pre-trained large model to deduce the answer according to the problem-solving logic corresponding to the question type information. The guidance information includes at least information on answering the first sample question by using the known conditions in the text content of the first sample question and combining them with the graphic information in the first sample question. The first prompt word information is input into the pre-trained large model to obtain the first predicted answer generated by the pre-trained large model for the first sample question; Based on the first standard answer and the first predicted answer, the pre-trained large model is fine-tuned to obtain the trained problem-solving large model.

2. The method as described in claim 1, characterized in that, Before obtaining the first training sample, the process also includes: Obtain a second training sample, which includes a second sample question and a second standard answer corresponding to the second sample question. The second sample question includes multiple types of questions in the education stage, which includes at least one of the preschool education stage, primary education stage, secondary education stage and university education stage. Construct second prompt word information for answering the second sample question, wherein the second prompt word information includes at least the question type information of the second sample question; The second prompt word information is input into the multimodal large model to obtain the second predicted answer generated by the multimodal large model for the second sample question; The multimodal large model is trained based on the second standard answer and the second predicted answer to obtain the pre-trained large model.

3. The method as described in claim 1, characterized in that, After fine-tuning the pre-trained large model, the process further includes: Obtain the user's test questions, wherein the test questions include at least one of the following types: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type; Construct third-party prompt information for answering the test questions; The third prompt word information is input into the solution model to obtain the third predicted answer generated by the solution model for the test question. Obtain the user's evaluation information regarding the third predicted answer; Based on the evaluation information, the problem-solving model is reinforced and trained to obtain the reinforced problem-solving model.

4. A problem-solving method for a specific target question, characterized in that, The method includes: Obtain the user's target question, wherein the target question type includes at least one of the following: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type; Construct a fourth prompt word information for answering the target question. The fourth prompt word information includes at least character information, question type information of the target question, and guidance information that guides the problem-solving model to deduce the answer according to the problem-solving logic corresponding to the question type information. The guidance information includes at least information on how to answer the target question by using the known conditions in the text content of the target question and combining them with the graphic information in the target question. The fourth prompt word information is input into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question.

5. The method as described in claim 4, characterized in that, The target question is a question contained in the target image, and the construction of the fourth prompt word information for answering the target question specifically includes: The image morphology of the target image is inspected to obtain a first inspection result; the image morphology includes at least one of image size, image format, and image resolution. If the first inspection result reflects that the image morphology of the target image is qualified, then a fourth prompt word information is constructed to answer the target question.

6. The method as described in claim 4, characterized in that, The target question is a question contained in the target image, and the construction of the fourth prompt word information for answering the target question specifically includes: The image content of the target image is inspected to obtain a second inspection result. The image content includes at least one of the following: image clarity, integrity of the title information in the image, readability of the text in the image, and integrity of the graphic elements in the image. If the second inspection result reflects that the image content of the target image is qualified, then a fourth prompt word information is constructed to answer the target question.

7. The method as described in claim 4, characterized in that, After inputting the fourth prompt word information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question, the method further includes: Determine whether the target answer contains keywords that do not conform to the semantic information of the target question, and obtain a first determination result; If the first judgment result indicates that there are no keywords in the target answer that do not conform to the semantic information of the target question, then the target answer is output to the user.

8. The method as described in claim 4, characterized in that, After inputting the fourth prompt word information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question, the method further includes: Determine whether the format of the target answer conforms to the pre-defined answer format requirements for the target question, and obtain a second determination result; If the second judgment result indicates that the format of the target answer conforms to the answer format requirements preset for the target question, then the target answer is output to the user.

9. The method as described in claim 4, characterized in that, After inputting the fourth prompt word information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question, the method further includes: The confidence level generated synchronously by the problem-solving model when generating the answer is obtained. The confidence level is used to quantitatively reflect the degree of confidence of the problem-solving model in that the target answer is the correct answer to the target question. Determine whether the confidence level is greater than or equal to a preset threshold to obtain a third determination result; If the third judgment result indicates that the confidence level is greater than or equal to the preset threshold, then the target answer is output to the user.

10. A large-scale problem-solving model training device, characterized in that, The device includes: The training sample acquisition module is configured to acquire a first training sample, which includes a first sample question and a first standard answer corresponding to the first sample question. The question type of the first sample question includes at least one of the following: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type. The first standard answer includes several derivation steps and the final result. The first prompt word information construction module is configured to construct first prompt word information for answering the first sample question. The first prompt word information includes at least character information, question type information of the first sample question, and guidance information that guides the pre-trained large model to deduce the answer according to the problem-solving logic corresponding to the question type information. The guidance information includes at least information that uses the known conditions in the text content of the first sample question and combines the graphic information in the first sample question to answer the first sample question. The first prompt word information input module is configured to input the first prompt word information into the pre-trained large model to obtain the first predicted answer generated by the pre-trained large model for the first sample question. The fine-tuning training module is configured to fine-tune the pre-trained large model based on the first standard answer and the first predicted answer, so as to obtain the trained problem-solving large model.

11. A problem-solving device for a target question, characterized in that, The device includes: The question acquisition module is configured to acquire the user's target questions. The target questions include at least one of the following types: geometric figure type, function graph type, graph expression type, graph counting type, three-view type, and statistical chart type. The fourth prompt word information construction module is configured to construct fourth prompt word information for answering the target question. The fourth prompt word information includes at least character information, question type information of the target question, and guidance information that guides the problem-solving model to deduce the answer according to the problem-solving logic corresponding to the question type information. The guidance information includes at least information on how to answer the target question by using known conditions in the text content of the target question and combining them with the graphic information in the target question. The fourth prompt word information input module is configured to input the fourth prompt word information into the problem-solving model to obtain the target answer generated by the problem-solving model for the target question.

12. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-9.

13. A computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1-9.

14. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1-9.

Images