Image-based problem solving method and device

Abstract

The embodiment of the invention provides an image-based problem solving method and device, and the method comprises the steps: determining to-be-solved image data, carrying out the analysis of the to-be-solved image data, and obtaining to-be-solved text data corresponding to the to-be-solved image data; determining similar question text data corresponding to the to-be-solved image data based on the to-be-solved text data; the similar question text data and the to-be-solved image data are input into a solving model, and the solving model is a fine-tuned visual language model; and outputting answering information corresponding to the to-be-solved image data through the solving model. According to the method, external prior knowledge is introduced, and when original question image information is incomplete or ambiguous, a clear and complete similar question stem text can provide key information supplementation and semantic clarification, so that the ability of a system to understand question intentions is effectively enhanced. The problem solving model can complete correct problem solving of image data to be solved under the assistance of similar problems.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of image processing technology, and in particular to an image-based problem-solving method. Background Technology

[0002] With the widespread adoption of smartphones and the continuous improvement of mobile hardware performance, the extensive use of high-definition cameras and the significant enhancement of device computing power have enabled users to conveniently and clearly capture and upload images of various paper or electronic problem sets. This provides a solid hardware foundation and rich application scenarios for image-based intelligent problem-solving technologies. Against the backdrop of increasingly widespread digital education and online learning, users' demand for quickly obtaining solutions, analyses, and even explanations of knowledge points through photography is growing. This technology not only helps improve learning efficiency but also promotes the equitable access to educational resources to some extent. However, in practical applications, the complete and accurate extraction of problem information often faces challenges due to factors such as shooting conditions, printing quality, complex layout (including formulas, charts, special symbols, etc.), and varying degrees of blurring and obstruction. Therefore, how to improve the accuracy of solving problems using incomplete images is a pressing issue that needs to be addressed. Summary of the Invention

[0003] In view of this, embodiments of this specification provide an image-based problem-solving method. One or more embodiments of this specification also relate to an image-based problem-solving apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, an image-based problem-solving method is provided, comprising: The image data of the problem to be solved is determined, and the image data of the problem to be solved is parsed to obtain the text data of the problem to be solved corresponding to the image data of the problem to be solved; Based on the text data of the problem to be solved, determine the similar text data of the problem to be solved corresponding to the image data of the problem to be solved; The similar question text data and the question image data to be solved are input into the problem-solving model, wherein the problem-solving model is a fine-tuned visual language model; The problem-solving model outputs the solution information corresponding to the image data of the problem to be solved.

[0005] According to a second aspect of the embodiments of this specification, an image-based problem-solving apparatus is provided, comprising: The parsing module is configured to determine the image data of the problem to be solved, parse the image data of the problem to be solved, and obtain the text data of the problem to be solved corresponding to the image data of the problem to be solved; The determination module is configured to determine similar question text data corresponding to the question image data based on the question text data to be solved; The input module is configured to input the similar question text data and the question image data to be solved into the problem-solving model, wherein the problem-solving model is a fine-tuned visual language model; The output module is configured to output the solution information corresponding to the image data of the problem to be solved through the problem-solving model.

[0006] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image-based problem-solving method.

[0007] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described image-based problem-solving method.

[0008] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described image-based problem-solving method.

[0009] One embodiment of this specification obtains the text data of the problem to be solved by parsing the image data of the problem to be solved. Based on the text data of the problem to be solved, corresponding similar problem text data is determined. This realizes the introduction of external prior knowledge. When the original problem image information is incomplete or ambiguous, the clear and complete similar problem stem text can provide key information supplementation and semantic clarification, thereby effectively enhancing the system's ability to understand the intent of the problem. The similar problem text data and the image data of the problem to be solved are input into the problem-solving model. With the assistance of the similar problem text data, the problem-solving model can correctly solve the problem image data of the problem to be solved, thereby outputting the solution information corresponding to the image data of the problem to be solved, providing a more stable and reliable solution for image problem-solving tasks. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an image-based problem-solving method provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating the processing procedure of an image-based problem-solving method provided in one embodiment of this specification. Figure 3 This is a schematic diagram of the structure of an image-based problem-solving device provided in one embodiment of this specification; Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0011] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0012] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0013] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0014] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0015] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0016] Visual Language Model (VLM): A Visual Language Model (VLM) is an artificial intelligence model that integrates computer vision and natural language processing capabilities. It can simultaneously understand and process information from both image and text modalities. It typically accepts images (or videos) and text as combined input, understands the relationship between the two through an internal fusion mechanism, and generates text as output. This enables VLM to perform tasks that require combining visual content with linguistic semantics, such as image description, visual question answering, and image-based dialogue. In the embodiments of this specification, after fine-tuning for a problem-solving task, the visual language model can achieve the fusion of image and text information to deeply understand the question and generate a solution.

[0017] Supervised Fine-Tuning (SFT) is a core training technique in machine learning, particularly in large-scale model applications. It refers to further training a model (i.e., a pre-trained model) that has already been "pre-trained" on massive amounts of unlabeled data and thus possesses broad basic capabilities, using a high-quality dataset with explicit annotations (i.e., "supervision signals") for a specific task. The goal of this process is to focus and adapt the capabilities of a general-purpose model to a specific downstream task, enabling it to predict or generate correct outputs that meet the requirements of that task based on the input data of the new task. In the embodiments of this specification, supervised fine-tuning is used to specifically train a pre-trained visual language model using a labeled dataset consisting of (question images, similar question texts, and standard solutions), thereby obtaining a problem-solving model capable of performing problem-solving tasks.

[0018] With the acceleration of digitalization in education and the widespread adoption of mobile smart terminals, online problem-solving technology based on intelligent image analysis has become an important tool for assisting learning and improving efficiency. The continuous improvement in the pixel count and image quality of smartphone cameras, coupled with the significant enhancement of mobile chip computing power, enables users to conveniently and clearly capture and upload images of various problems containing complex formulas, geometric figures, tables, and special symbols. This provides a solid hardware foundation and broad application scenarios for applications such as photo-based problem searching and intelligent Q&A. This technology not only helps students quickly obtain solutions to difficult problems but also promotes the sharing and universal access to high-quality educational resources to a certain extent, possessing significant social value and market prospects.

[0019] Currently, the technical implementation of image-based problem-solving tasks in the industry mainly follows the following two paths: Firstly, this can be summarized as a cascaded "recognition-solution" approach. This method first relies on optical character recognition (OCR) technology to extract textual information from the question image, and then performs subsequent semantic understanding and problem-solving calculations based on the recognized text. This path is logically clear and the modules are well-defined, but its effectiveness highly depends on the accuracy of the preceding OCR (Optical Character Recognition) stage. In practical applications, due to issues such as uneven lighting during shooting, paper wrinkles, blurry printing, and difficulties in recognizing complex formulas or handwritten characters, the OCR process inevitably encounters problems such as misrecognition, missed recognition, or loss of formatting information. This is especially true for questions containing illustrations, charts, or special layouts, where visual information is easily stripped away during the conversion to text, leading to incomplete input information in subsequent problem-solving stages, thus affecting the accuracy and reliability of the final solution.

[0020] Secondly, there are end-to-end visual understanding solutions. With the development of multimodal artificial intelligence technology, especially the emergence of large-scale visual language models, a technical approach has emerged that directly uses the question image as input and outputs the solution content from the model. This type of method is typically based on a large-scale pre-trained visual language model, which is then supervised and fine-tuned to adapt to the problem-solving task, thus avoiding errors and information loss that may be introduced by the OCR process. However, this type of method still faces the challenge of generalization in real-world scenarios: when the input image is of high quality and the content is clear, the model usually performs well; however, when encountering blurry images, partial occlusion, low contrast, or tilted shooting angles, the model struggles to reliably extract key information from the image, resulting in a significant drop in its solution performance. Essentially, this type of method still primarily relies on the quality of information that can be obtained from the image itself.

[0021] In summary, existing technical solutions all have certain limitations when processing problem images in realistic and complex scenarios. Whether it's the unavoidable information loss and error propagation in cascaded methods, or the high sensitivity of end-to-end methods to image quality, both restrict the stability and accuracy of problem-solving systems in practical applications. Therefore, how to enhance the system's robustness to imperfect image input, and maintain reliable understanding and solution capabilities even when image information is missing or blurred, has become a key technical problem urgently needing to be solved in this field.

[0022] Based on this, this specification provides an image-based problem-solving method. This specification also relates to an image-based problem-solving apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0023] See Figure 1 , Figure 1 A flowchart of an image-based problem-solving method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0024] Step 102: Determine the image data to be solved, parse the image data to be solved, and obtain the text data to be solved corresponding to the image data to be solved.

[0025] The image data containing the problem to be solved can be understood as a digital image captured by the user's terminal device (such as a mobile phone or tablet) camera, containing the problem to be solved. This data not only contains the text of the problem but may also contain various modal information such as formulas, geometric figures, charts, and images, serving as the raw input for all subsequent processing. Parsing the image data involves applying Optical Character Recognition (OCR) technology. This process typically includes: preprocessing the image (such as denoising, correction, and binarization), then using a trained OCR model to identify the text regions in the image and converting the pixel sequences in these regions into corresponding character sequences (strings). The text data containing the problem is the text information extracted from the image through the above parsing process. It initially expresses the content of the problem, but may contain recognition errors, missing characters, or formatting errors due to image quality issues (e.g., recognizing the formula "x²" as "x2"). Nevertheless, it remains the most direct semantic representation of the problem content that can be utilized by traditional information retrieval systems.

[0026] In practical applications, in real-world image-based problem-solving scenarios, user-uploaded problem images often suffer from issues such as blurriness, tilt, uneven lighting, and partial occlusion due to shooting environment, equipment performance, or original printing quality. Directly inputting these imperfect images into a visual model for problem-solving can lead to failure or incorrect answers because the model struggles to accurately and completely extract the semantic information of the problem. Therefore, to build a problem-solving system more robust to image quality, it is essential to first extract structured semantic clues from the original image as much as possible. This provides a reliable basis for the subsequent crucial step of "searching for similar problems." The core objective of this step is to convert the image into machine-readable text information through image parsing, laying the foundation for the entire information augmentation process.

[0027] Furthermore, in order to extract text data from complex, unstructured image data, it is necessary to first determine the text regions containing text in the image. Specifically, the image data to be solved is parsed to obtain the text data corresponding to the image data to be solved, including: parsing the image data to be solved and determining the text regions in the image data to be solved; performing character recognition on the text regions to obtain the recognized text data corresponding to the text regions; and determining the text data to be solved corresponding to the image data to be solved based on the recognized text data.

[0028] In this context, a text region can be understood as a rectangular or arbitrarily shaped area within the image data of the question to be solved, occupied by a visual block containing one or more consecutive characters. Determining a text region involves not only locating its coordinate boundaries but also often inferring the logical relationships and reading order between different regions (e.g., which part of the text is the main body of the question stem, and which part is the option or caption). Recognized text data can be understood as the raw string result directly output after optical character recognition (OCR) of a single text region image. This data may contain all characters within the region but has not yet been sequentially integrated with the results from other regions, nor has it undergone deep error correction or formatting for the question's semantics. The text data of the question to be solved can be understood as the structured text ultimately generated after integrating the recognized text data from all text regions and their positional and sequential relationships, representing the entire question's semantics. It not only contains complete text content but may also implicitly or explicitly retain the question's formatting information (such as paragraphing and option numbering), making it a high-quality semantic representation that can be directly used in subsequent processes.

[0029] In practical applications, a single, end-to-end recognition process is prone to producing low-quality text when faced with diverse layouts and varying quality of question images in real-world scenarios. This is often due to interference from non-textual elements, inaccurate text region localization, or accumulated recognition errors. Consequently, all subsequent steps (especially similar question retrieval) are built on a foundation of errors. Therefore, decomposing the parsing process into an ordered pipeline of "region detection, character recognition, and text synthesis" aims to achieve accurate extraction and reconstruction of image information through modular and phased processing.

[0030] In practice, the first step is to identify the text region to eliminate non-textual interference such as illustrations, stains, and decorative lines, focusing on the true semantic carrier. Then, the purified text region is identified, improving the specificity and accuracy of character recognition. Finally, the text is reconstructed based on the recognition results and region information to restore the original logical structure of the question (such as the distinction between the question stem and options), forming structured text data suitable for deep semantic processing. This solves the problem of information confusion and structural loss that can result from directly converting images into text in a "black box" manner.

[0031] In one specific embodiment of this specification, after analyzing the image, the system locates five main text regions: a larger upper region (the question stem) and four smaller, horizontally arranged lower regions (options A to D).

[0032] The system identifies the question stem region and obtains the original result: "An object starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the first second is 2m. Then its displacement in the third second is ___." Identifying option A yields: "A. 6m"; identifying option B yields: "B. 10m" (Options C and D are omitted, assumed to be correct). Based on the region's position (question stem at the top, options at the bottom, arranged left-right), the system synthesizes the above identification results into a "question stem + option list" structure. The final structured text data of the problem to be solved is generated: "An object starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the first second is 2m. Then its displacement in the third second is ___. A. 6m B. 10m C. 14m D. 18m."

[0033] Based on this, even if the original image has a complex background or irrelevant markings, by first locating the text region, the system effectively isolates the recognition target, avoiding inputting non-textual noise into the recognition model, thus laying the foundation for obtaining cleaner recognition text data. It not only extracts the text content but also automatically identifies and preserves the logical structure of the question (distinguishing between the question stem and options). This structured question text data contains more information than unstructured plain text paragraphs, which is crucial for subsequent understanding of question types and accurate matching of similar questions (especially multiple-choice questions). If a serious error occurs in the recognition of a certain region (e.g., a certain option is completely misidentified), this process can confine the error to that region without contaminating other correctly identified parts (such as the question stem). At the same time, the structured output also facilitates possible subsequent error correction (such as rule verification based on the assumption that all options should be numerical).

[0034] Furthermore, after identifying the text data, to avoid errors in generating the text data to be solved due to incorrect text order, it is necessary to determine the text order of the text data. Specifically, determining the text data to be solved corresponding to the image data to be solved based on the identified text data includes: inputting the identified text data into a semantic recognition model, determining the semantic information corresponding to the identified text data through the semantic recognition model; sorting the identified text data according to the semantic information, and generating the text data to be solved corresponding to the image data to be solved based on the sorting result.

[0035] The semantic recognition model can be understood as a trained natural language processing model. Its input is a text fragment, i.e., a piece of recognized text data, and its output is a functional type determination and / or semantic feature representation of the fragment within the context of the question. Its functions may include: determining whether the fragment is the "body of the question stem," "question sentence," "option A," "figure caption," "known conditions," or "solution target"; or encoding it as a semantically meaningful vector. Semantic information can be understood as a structured understanding of each text fragment output by the semantic recognition model. It contains at least two levels of information: functional labels, identifying the fragment's role in the question; and logical relationship clues, which may include implicit relationships between the fragment and other fragments (e.g., which option belongs to which question). When sorting the recognized text data according to semantic information, the sorting logic considers not only coordinate positions but primarily the logical relationships implied in the semantic information. The basic rule is usually to prioritize the body of the question stem, followed by the question or solution instruction, and then the options (usually in the order of A, B, C, D). This ensures that the generated text conforms to the standard question narrative logic of "stating the background - posing the question - providing the options."

[0036] In practical applications, after obtaining multiple scattered "recognized text data" (such as question stem blocks and option blocks) through OCR, simply sorting them based on their physical positions in the image (such as coordinates from top to bottom) may fail to restore the correct reading and logical order when dealing with complex layouts (such as columns, inserted charts, or formulas). Incorrect text order will lead to semantic confusion and logical breaks in the generated "question text data," severely compromising the accuracy of subsequent similar question retrieval and providing the problem-solving model with an incorrect understanding foundation. Therefore, the embodiments in this specification use a semantic recognition model to sort the text data. The core purpose is to go beyond simple geometric positional sorting and intelligently reconstruct a complete, logically coherent question text that conforms to human reading and comprehension habits by understanding the semantic content of each text block and its functional role in the question. This solves the problem of logical order disorder that may be caused by purely visual layout analysis, ensuring a reliable conversion from image to high-quality semantic representation.

[0037] In a specific embodiment of this specification, referring to the above example, after region detection and recognition, "recognized text data" for five text regions are obtained. However, due to image blurring or special layout, the original recognition order may be disordered: “B.10m” (Recognized text data 1); "An object starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the first second is 2m. Then its displacement in the third second is ___." (Recognize text data 2); “D.18m” (Recognized text data 3); “A.6m” (Recognized text data 4); “C.14m” (Recognized text data 5).

[0038] Input the five text segments above into the semantic recognition model. The model outputs semantic information (functional labels): Recognized text data 1: OPTION_B (Option B); Recognized text data 2: MAIN_STEM (containing both the stem and the question); Recognized text data 3: OPTION_D (Option D); Recognized text data 4: OPTION_A (Option A); Recognized text data 5: OPTION_C (Option C). The sorting rule is: MAIN_STEM has the highest priority and is ranked first. Then, OPTION is sorted alphabetically (A, B, C, D). The sorted order is: Recognized text data 2 - Recognized text data 4 - Recognized text data 1 - Recognized text data 5 - Recognized text data 3. Logically concatenate the sorted texts to generate the final, structured text data for the problem to be solved: "An object starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the first second is 2m. Then its displacement in the third second is ___. A. 6m B. 10m C. 14m D. 18m."

[0039] Based on this, by sorting according to semantic information, regardless of whether the options in the original image are arranged horizontally, vertically, or in other complex layouts, the final output text is forced to be standardized into a uniform structure. This fundamentally corrects the possible disordered order of the original text caused by the OCR process (such as options B and D appearing before the question stem in the example), ensuring the absolute semantic and logical correctness of the final generated "question text data." This is a key quality assurance for generating a reliable and usable question description.

[0040] Step 104: Determine similar question text data corresponding to the question image data based on the question text data to be solved.

[0041] The "question text data" refers to the question text parsed from the original image, which may contain recognition errors or missing information. The "similar question text data" can be understood as the complete question stem text of one or more questions retrieved from a pre-built large-scale structured question bank that are highly similar to the "question text data" in terms of the knowledge points tested, question type structure, key entities, or problem-solving methods. Its text quality is high, its expression is standardized, and it serves as an "information enhancement source."

[0042] In practical applications, the "text data to be solved" obtained in the preceding steps may contain incomplete information, ambiguous expressions, or local errors due to image blurring, occlusion, or OCR recognition errors (such as misidentifying "uniform speed" as "allowed speed"; or option D not being fully captured, resulting in the inability to recognize its value). If this text of varying quality is used as the sole basis for solving the problem, the model's understanding and reasoning foundation will be very fragile, especially when facing questions requiring precise semantics, easily leading to incorrect answers. Therefore, the purpose of this step is to find one or more semantically highly relevant and clearly and accurately expressed "reference examples" for the potentially flawed original question text. By introducing these external high-quality similar question texts, key information supplementation, semantic clarification, and context enhancement can be provided to the subsequent problem-solving model, thereby compensating for possible information loss or noise in the original input and improving the overall robustness of the system.

[0043] In practice, the process involves identifying similar question texts based on the text data of the question to be solved, which manifests as a retrieval and matching process. This typically involves using information retrieval techniques to calculate the similarity between the "text data of the question to be solved" and all question texts in the question bank, and returning one or more of the most similar results.

[0044] Furthermore, determining similar question text data corresponding to the question image data based on the question text data includes: determining the question type corresponding to the question image data based on the question text data, and determining the question database corresponding to the question image data based on the question type; filtering the question database based on the question text data to obtain similar question text data corresponding to the question image data.

[0045] The question type can be understood as a multi-dimensional classification label, such as subject category and question type / structure category. Subject categories can include mathematics, physics, chemistry, history, etc. This determines the knowledge domain and terminology to which the question belongs. Question type / structure categories can include multiple-choice questions, fill-in-the-blank questions, problem-solving questions, proof questions, etc. This determines the basic narrative logic and output format of the question. Type determination can be achieved through text-based classification models, keyword rule matching, or a combination of both. The question database can be understood as a structured, massive collection of questions organized by type labels. It is not a single data pool, but a collection indexed or partitioned based on "question type." For example, there are sub-databases such as "Physics - Multiple Choice" and "Mathematics - Fill-in-the-Blank." Questions in each sub-database have clear metadata (such as subject, question type, and knowledge point) for easy and quick location.

[0046] In practice, the text data of the questions to be solved is filtered within the question database. This involves a precise matching process using text similarity algorithms within the target "question database" (i.e., a specific sub-database) after the target "question database" has been determined. Since the search space has been greatly narrowed and purified (e.g., searching only in "physics-multiple choice questions"), this step can focus more on comparing the semantic similarity of the question content. Calculations can be performed using methods such as TF-IDF (term frequency–inverse document frequency, a commonly used weighting technique for information retrieval and data mining), BM25 (BestMatching 25, a ranking function), or semantic vector models to find the most similar questions.

[0047] In practical applications, the preceding steps have already obtained the text data of the questions to be solved. Directly searching through a large, mixed global question database presents two core problems: low search efficiency and excessive noise in the search results. Questions from different disciplines (such as mathematics, physics, and history) differ greatly in vocabulary, expression, and knowledge system; even within the same discipline, different question types (such as multiple choice, fill-in-the-blank, and calculation questions) have vastly different structures and solution methods. Indiscriminate global searches not only incur high computational costs but may also return a large number of "pseudo-similar" results that do not match the discipline or question type. These results have little value in aiding understanding the original questions and may even interfere with subsequent models. Therefore, in this embodiment, the step of "determining similar question text data" is refined into a two-level process of "classification followed by filtering," aiming to systematically solve the above problems through a layered filtering strategy. By first determining the question type, the original input is quickly categorized, locking in the most relevant search range. Then, filtering is performed in a specific database, with fine-grained similarity matching only performed on a subset of highly relevant questions. This approach can significantly improve the accuracy and efficiency of retrieval, ensuring that the similar questions found are not only similar in content, but also highly consistent in examination format and knowledge scope, thereby maximizing their value as an "information enhancement source".

[0048] In a specific embodiment of this specification, referring to the example above, the system analyzes the text data of the question to be solved. It contains words such as "object," "displacement," "uniformly accelerated linear motion," and "m (meter)," determining the subject category to be "physics." The text ends with a clear format, "A.6m B.10m...", indicating the question type is "multiple choice." Based on this, the question type is determined to be "physics-multiple choice," and the specific question database for retrieval is locked as the "Physics Multiple Choice Question Database." The system uses the text data of the question to be solved as the query, performing a similarity search only within the "Physics Multiple Choice Question Database." Since all questions in this sub-database are physics multiple choice questions, the algorithm can more effectively match kinematic concepts and the expression patterns unique to multiple choice questions. The system retrieves and returns the original text of the question with the highest similarity. The finally determined similar question text data is: "A particle starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the first second is 2 meters. Then its displacement in the third second should be: A. 4 meters B. 6 meters C. 8 meters D. 10 meters." It should be noted that in practice, the returned similar questions may differ in specific values ​​or options, but the physical models and test points are highly consistent.

[0049] Based on this, by first performing rapid classification, the search scope is quickly narrowed from a global database that may contain tens of millions of questions to a specific sub-database (such as a "Physics-Multiple Choice" database). This greatly reduces unnecessary computational comparisons, lowers system latency, and ensures that the returned results are strictly aligned with the original questions in terms of subject and question type, avoiding irrelevant results such as "historical multiple choice questions" or "physics calculation questions," thereby significantly improving the accuracy and relevance of the retrieval. Because the retrieved similar questions strictly match the original questions in terms of type, their textual information has a more direct and powerful auxiliary effect on the problem-solving model. For example, the structure of the stems and options of similar multiple choice questions can directly serve as a reference for the model to understand the format of the original questions. This provides higher-quality and more targeted context for subsequent "information augmentation" steps.

[0050] Furthermore, the process of filtering the text data of the problem to be solved in the question database to obtain similar question text data corresponding to the image data of the problem to be solved includes: calculating the text statistics information corresponding to the text data of the problem to be solved; determining the question similarity between the text data of the problem to be solved and the questions in the question database based on the text statistics information; and selecting similar question text data corresponding to the image data of the problem to be solved in the question database based on the question similarity.

[0051] Textual statistics can be understood as mathematical models or vectors extracted from the text data of the problem to be solved, used to quantify its textual features. This is not the original text string, but a mathematical abstraction of it. The forms of textual statistics can include: "bag-of-words model vector: based on TF (term frequency) statistics"; "TF-IDF vector: based on term frequency, IDF (inverse document frequency) is introduced to reduce the weight of common words and highlight feature words"; "BM25 relevance statistics: a more advanced statistical value of probabilistic retrieval model that considers document length and term frequency saturation"; "word embedding vector or sentence vector: dense vectors with deep semantic meaning obtained through pre-trained language models".

[0052] Question similarity can be understood as a scalar value used to quantify the relevance between two questions (here referring to the "question to be solved" and a question in the question bank). Its calculation method depends on the textual statistical information used. For example, if using TF-IDF vectors, the similarity is usually the cosine similarity between the two vectors. If using BM25, the similarity is the relevance score directly calculated by the model. If using semantic vectors, the similarity can be the cosine similarity between vectors or the reciprocal of the Euclidean distance. Selecting questions based on similarity means that the system sorts the calculated "question similarity" between each candidate question in the question bank and the question to be solved, and ultimately determines the returned "similar question text data" according to a preset strategy (such as selecting the question with the highest score, or the top N questions with scores exceeding a threshold).

[0053] In practical applications, based on a previously locked database of specific question types, simple keyword matching or literal overlap calculations are insufficient to address the common synonyms, lexical variations, and contextual differences in question texts (for example, "uniform acceleration" and "the case where the initial velocity is zero in uniformly accelerated linear motion" are different expressions but share the same core physical model). If the filtering method is not refined enough, the returned "similar questions" may only have partial literal overlap, but do not match in terms of the knowledge points tested, problem-solving logic, or difficulty, thus failing to provide effective enhancement information for subsequent problem-solving models. In one embodiment of this specification, a filtering mechanism based on textual statistical information to calculate question similarity aims to achieve a quantifiable, comparable, and semantically relevant precise matching. This solves the problem of insufficient intelligence in simple matching methods, ensuring that the filtered similar questions are "similar in spirit" (semantics and test points) rather than merely "similar in form" (literal), which is a key technical aspect guaranteeing the overall effectiveness of the invention.

[0054] In a specific embodiment of this specification, referring to the example above, the system preprocesses the "text data to be solved" (word segmentation and stop word removal). Assuming the TF-IDF model is used, the system calculates the TF-IDF vector of the text to be solved, i.e., the text statistical information, based on the vocabulary distribution of the entire "Physics Multiple Choice Question Bank". This vector assigns higher weights to keywords such as "object", "stationary", "uniform acceleration", "linear motion", "displacement", "1st second", "2m", and "3rd second", while common words such as "one", "start", "do", "is", and "then" have very low weights.

[0055] The system iterates through every question in the question bank. For a candidate question in the question bank: "A particle moves from rest in uniformly accelerated linear motion. If its displacement in the first second is 2 meters, find its displacement in the third second. Options omitted." The system also calculates the TF-IDF vector of this candidate question. Then, it calculates the cosine similarity between the vector of the question to be solved and the vector of this candidate question, obtaining a similarity score between 0 and 1, for example, 0.85. The system calculates the similarity score between all candidate questions in the question bank and the question to be solved. Assuming the preset strategy is "select the question with the highest similarity", and the above candidate question has a score of 0.85 as the highest, the system then outputs the complete and clear question stem text of the candidate question as "similar question text data".

[0056] Based on this, the similar questions obtained through the above-mentioned fine-tuning process have textual information that is highly consistent with the original question in terms of knowledge core. When this high-quality similar question text is input into the problem-solving model along with the original blurred image, it can provide the most direct and effective contextual supplementation and semantic guidance, greatly improving the efficiency of the "information enhancement" step.

[0057] Step 106: Input the similar question text data and the question image data to be solved into the problem-solving model, wherein the problem-solving model is a fine-tuned visual language model.

[0058] The similar question text data refers to the clear text obtained through the above steps, used to assist the model's understanding as "context" or "hints." Its role is to provide the model with a reference framework regarding the knowledge points, question types, and standard expressions that the current question may involve. The image data of the question to be solved is the specific target object that needs to be understood and answered. The problem-solving model can be understood as a supervised fine-tuned visual language model (VLM). Its problem-solving ability is reflected in its ability, after training, to map the joint input of image and text to a structured solution output. The problem-solving model can be obtained by fine-tuning a pre-trained visual language model, which possesses the inherent architecture and ability to simultaneously process and understand both image and text modal information and perform cross-modal associations. The fine-tuned visual language model is not a general model, but rather a specially optimized model using a large number of training samples consisting of "(question image, similar question text) as input, and corresponding standard answer as output." Through this fine-tuning, the model learns how to use clear similar question text to guide and assist in understanding potentially blurred target images, and generates answers based on this enhanced understanding.

[0059] In practical applications, the above steps have yielded two key types of information, each with different modalities and qualities: image data of the problem to be solved and text data of similar problems. The image data of the problem to be solved is the original visual information carrier, which may be blurry or occluded, but contains a complete visual representation of the problem (including text, formulas, and chart layout). The text data of similar problems consists of clear text information that is highly semantically related and retrieved from an external question bank, but may differ from the original problem in specific wording and numerical values.

[0060] In practice, relying solely on image data makes it difficult for the model to reliably understand images of poor quality; relying solely on retrieved text results in a loss of grasp of the original question's specific visual details (especially non-textual elements and precise layout). Therefore, this step aims to fuse these two types of information—from different sources, with varying modalities, and complementary strengths—to provide a more comprehensive and robust joint input for generating high-quality solutions. This overcomes the inherent limitations of a single information source and is a crucial step in realizing "information augmentation" from concept to implementation.

[0061] Furthermore, inputting the similar question text data and the question image data to be solved into the problem-solving model includes: generating multimodal problem-solving data based on the similar question text data and the question image data to be solved; and inputting the multimodal problem-solving data into the problem-solving model.

[0062] Multimodal problem-solving data can be understood as a structured, serialized data unit that encapsulates information from both textual and visual modalities, clearly defining the logical relationships between them. It is not a simple data concatenation, but a task-oriented encapsulation format. Typically, this data includes: task instruction text, explicitly telling the model the task to perform (e.g., "Please refer to the following similar question information to answer the question in the given image"); similar question text data, providing clear text as reference information; and a representation of the image data to be solved, which can be raw image pixel data, preprocessed image features, or image placeholders agreed upon by the model. These elements are organized into a sequence according to a predetermined order and separators (such as special tokens).

[0063] In practical applications, the previous step has already obtained two heterogeneous information sources: similar question text data and the image data of the problem to be solved. However, a visual language model without special design usually defaults to accepting a single image or a single text as input, or accepting multimodal input in a specific format. If both types of data are simply input into the model, the model may not understand the relationship between them and their respective roles, resulting in ineffective information fusion or even mutual interference. Therefore, the purpose of this step is to combine the raw, discrete multimodal data into a structured multimodal problem-solving data that carries clear task instructions and contextual relationships, in a format that the model can understand and that meets task requirements. This solves the "format gap" problem between the raw data and the model input interface, ensuring that the model can accurately identify and utilize similar question text as auxiliary information to understand the image of the problem to be solved.

[0064] In a specific embodiment of this specification, referring to the above example, the text data of the similar problem is determined to be "A particle undergoes uniformly accelerated linear motion from rest. If its displacement in the first second is 2 meters, find its displacement in the third second." The image data to be solved is the original, blurry image of the problem (assuming it has been loaded as an image array or tensor in the system). The system calls a preset input template. Assume the template is as follows: "<|System|>Please answer the question in the image based on the similar question information.<|Similar_Problem|>A particle moves from rest in uniformly accelerated linear motion. If its displacement in the first second is 2 meters, find its displacement in the third second.<|Image|>[This is a tensor representation of the image data to be solved or a special image embedding marker]<|Question|>."

[0065] The system populates the template with actual data, fixing the text after the instruction section `<|System|>`. Similar problem text is then inserted after the `<|Similar_Problem|>` tag. The image data (or its feature representation) to be solved is then inserted after the `<|Image|>` tag. Finally, a hybrid data sequence containing a text token sequence and image embeddings is generated, i.e., multimodal problem-solving data. This generated structured sequence is then input into a fine-tuned visual language model, which solves the problem based on the prompt template.

[0066] Based on this, by generating multimodal problem-solving data, the system assigns clear roles and structures to the original data. The model no longer needs to guess which text is a similar question and which is the actual question, eliminating information ambiguity and ensuring that the model's attention is accurately guided to the correct location, which is the foundation for efficient information fusion.

[0067] Furthermore, the training of the problem-solving model includes: collecting sample question images and obtaining sample answers corresponding to the number of sample question images; determining similar question texts corresponding to the sample question images and constructing a training sample set based on the sample question images, the sample answers, and the similar question texts; and using the training sample set to fine-tune the visual language model to obtain the problem-solving model.

[0068] The sample question images can be understood as the basic raw materials used for model training, namely a large number of realistic and diverse question photos. Their characteristic is that they should cover various image qualities (clear, blurry, tilted, occluded, etc.) to force the model to learn to rely on auxiliary information under difficult conditions. The sample answer is a high-quality, structured standard solution corresponding to each sample question image. It usually includes not only the final answer but also the reconstruction and step-by-step parsing of the question text, serving as the target output for model learning. Similar question text can be understood as semantically relevant and clear question text obtained for each sample question image through the aforementioned retrieval process (or manual annotation). During training, it appears as conditional information or hints paired with the sample question images.

[0069] A training sample set can be constructed using sample question images, sample answers, and similar image texts. This training sample set is a structured dataset consisting of numerous "(sample question image, corresponding similar question text) - sample answer" pairs. Each sample is a complete "input-output" mapping instance, serving as the learning material for the model. Subsequent fine-tuning of the pre-trained visual language model involves using the aforementioned training sample set to perform supervised retraining. The core idea is to teach the model a mapping function: given a "question image + similar question text," it generates the corresponding "standard answer." During training, the model's parameters are adjusted to minimize the difference between its predicted output and the sample answers.

[0070] In practical applications, a general-purpose, untrained Visual Language Model (VLM), while possessing basic image and text understanding and generation capabilities, lacks optimization in its knowledge, reasoning patterns, and output format for the highly specialized task of "using clear, similar question text to assist in solving ambiguous question images." Directly using a general-purpose model will fail to understand the specific role and value of similar question text in this scenario, and will be unable to systematically learn how to combine these two information sources to generate structured solutions. Therefore, the purpose of this training step is to transform the general-purpose Visual Language Model into an "expert" proficient in this invention through supervised fine-tuning. This resolves the "capability gap" between the general-purpose model and specific problem-solving tasks.

[0071] In one specific embodiment of this specification, the system collects a slightly blurry photo (sample question image) of a physics multiple-choice question uploaded by a user from the product backend. The sample answers in this photo are annotated by experts or rigorously verified as follows: The problem was rewritten as follows: "An object starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the first second is 2m. What is its displacement in the third second?" Analysis: "According to the law of uniformly accelerated linear motion with an initial velocity of zero, the ratio of displacements in consecutive equal time intervals is 1:3:5:7... The displacement in the first second is 2m, corresponding to ratio 1. Therefore, the displacement in the third second (corresponding to ratio 5) is 2m × 5 = 10m." Answer: "B. 10m". Perform OCR and retrieval on the blurry photo to obtain similar question text: "A particle moves from rest in uniformly accelerated linear motion. If its displacement in the first second is 2 meters, find its displacement in the third second." Construct a training sample: Input: {"Image": blurry photo, "Similar question": "A particle moves from rest in uniformly accelerated linear motion. If its displacement in the first second is 2 meters, find its displacement in the third second."}, Target output: the structured sample answer above.

[0072] Repeat the above process to construct a training sample set containing hundreds of thousands or even millions of such samples. Use this dataset to fine-tune a pre-trained VLM. During training, the model continuously attempts to generate a solution based on the input "blurred image + clear similar question text," compares its output with the "standard answer," and adjusts parameters through backpropagation. After sufficient training, the model learns key patterns: when the image shows "uniformly accelerated" blur, refer to "uniformly accelerated linear motion" in the similar question text; when displacement ratio rules need to be applied, the similar question text provides key contexts such as "second 1" and "second 3." Finally, a problem-solving model specifically designed for this task is obtained.

[0073] Based on this, through learning on massive amounts of samples, the model internalizes how to use clear, similar question text as an aid to complete, clarify, or correct the understanding of blurry images. The model no longer processes images in isolation but learns to actively utilize external textual cues, thus gaining robustness to cope with image quality fluctuations in real-world scenarios. By training with structured sample answers, the model is forced to learn to generate output according to a prescribed format and logic (first reconstructing the question, then analyzing it step-by-step, and finally providing the answer). This ensures that the solutions produced by the problem-solving model are not only correct in content but also neatly formatted, conform to teaching standards, and have direct application value.

[0074] Step 108: Output the solution information corresponding to the image data of the problem to be solved through the problem-solving model.

[0075] The problem-solving model, after fine-tuning the aforementioned process, is a specialized visual language model capable of enhancing image understanding and generating solutions using information from similar questions. The solution information output by the problem-solving model is the reasoning result given for the input multimodal problem-solving data. It is a structured, multi-part text output, typically including the following core elements: Question Reconstruction: Based on enhanced understanding, a clear and accurate restatement of the original ambiguous question content.

[0076] Analysis: This demonstrates the logical reasoning process that led to the answer, usually presented step by step.

[0077] Answer: A clear, definitive final answer (for multiple-choice questions, this includes option identifiers). Additionally, depending on the training data setup, supplementary information may be included, such as summaries of key knowledge points, methodologies, or tips on common mistakes.

[0078] In practical applications, the preceding steps have completed the entire processing flow from the original blurred image to the information-enhanced input, and the problem-solving model has completed internal reasoning based on the fused multimodal information. However, the model's internal understanding must be presented in a form that is perceptible, understandable, and directly usable by the user to realize its final problem-solving value. If the model only knows the answer internally but cannot generate a standardized, complete, and credible explanation, the system still lacks practical value. Therefore, the purpose of this step is to transform the enhanced understanding and reasoning results formed by fusing similar problem texts and the image to be solved by the problem-solving model into a clearly structured, accurate, and formatted solution information for output. This solves the final link in the intelligent system's transition from "internal cognition" to "external service," representing the ultimate realization of the technology's value.

[0079] In a specific embodiment of this specification, the model has received and processed the combined input: "Please refer to the similar question information to answer the question in the image. Similar question: A particle undergoes uniformly accelerated linear motion from rest. If its displacement in the first second is 2 meters, find its displacement in the third second. + [fuzzy original question image]". The model uses the similar question text to clarify that this is a question about "the displacement ratio of uniformly accelerated linear motion with an initial velocity of zero", and combines the numerical value "2m" identified in the image with the option structure to complete the calculation and judgment. The model, according to the format learned during fine-tuning, begins to generate a text sequence as "answer information": Question: An object starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the first second is 2m. What is its displacement in the third second? Options: A. 6m B. 10m C. 14m D. 18m

[0080] Analysis: For uniformly accelerated linear motion with an initial velocity of zero, the ratio of displacements in consecutive equal time intervals is 1:3:5:7…. Given that the displacement in the first second is 2m (corresponding to ratio 1), the displacement in the third second (corresponding to ratio 5) is 2m × 5 = 10m.

[0081] Answer: B. 10m.

[0082] The system outputs the complete text sequence generated by the above model to the user as the solution information corresponding to the image data of the problem to be solved.

[0083] Therefore, the solution information not only includes the answer but also the question confirmation, logical deduction, and conclusion. This structured, step-by-step output makes the solution process transparent, enhancing user trust in the system. Especially when users discover that the model correctly applies relevant rules in the analysis, they have a higher degree of confidence in the correctness of the answer. This step transforms all previous technical processing into intuitive and valuable user perception and satisfaction, completing a closed loop from technological advantages to product value.

[0084] Furthermore, outputting the solution information corresponding to the image data of the problem to be solved through the problem-solving model includes: constructing complete question text data based on the similar question text data and the image data of the problem to be solved through the problem-solving model; solving the problem based on the complete question text data to obtain the answer text data corresponding to the image data of the problem to be solved; generating and outputting the solution information corresponding to the image data of the problem to be solved based on the complete question text data and the answer text data.

[0085] The complete question text data can be understood as a clear, accurate, and unambiguous description of the question text, regenerated by the problem-solving model based on the semantic guidance of "similar question text data" and the visual information of "the image data of the question to be solved." It should contain all elements of the question stem (conditions, questions), and for multiple-choice questions, all options. This data represents the model's "best understanding" of the original fuzzy question. Solving the problem based on the complete question text data is the process by which the problem-solving model, based on this pure text input (the complete question text data), performs logical reasoning, calculation, or analysis to arrive at the final answer. At this point, the solution is based on clear, structured text, avoiding direct processing of image noise. The answer text data is the solution obtained through the above problem-solving steps, specifically for the complete question text data. For multiple-choice questions, this is usually the option identifier and content; for fill-in-the-blank or problem-solving questions, it is the specific numerical value or statement.

[0086] In practical applications, this step, building upon the direct generation of complete solution information, aims to address the dual challenges of ensuring the interpretability of the model's reasoning process and the reliability of the solution. If the model directly generates a final output containing the answer, it's difficult for the user (or system) to verify whether it accurately understands the original question, nor can it confirm whether the answer is derived from a correct reconstruction of the question. Especially in blurry image scenarios, the model might "guess" the answer due to missing information or reason based on an incorrect reconstruction. Therefore, refining the output process into a three-step workflow—"first reconstructing the question, then solving the problem based on the reconstruction, and finally integrating the output"—aims to achieve the following objectives: Separate concerns: Ensure that the model first focuses on accurately reconstructing the question itself from the blurred image and similar question text, which is a separate and crucial subtask.

[0087] Explicit reasoning chain: Decoupling "question reconstruction" from "answer generation" makes the model's reasoning process more traceable. The reconstructed, clear question text can serve as an intermediate verification point.

[0088] Improve the reliability of the solution: Solving the problem based on the clear and complete problem text reconstructed by itself is equivalent to allowing the model to reason on a "clean" semantic basis, which can reduce errors caused by directly processing multimodal noise input.

[0089] In a specific embodiment of this specification, the model has received and processed combined input (similar question text + blurred original image). The model uses the similar question text ("uniformly accelerated linear motion", "displacement in the 1st second", "displacement in the 3rd second") as key semantic cues, carefully analyzing the text regions in the blurred image. The model generates and outputs complete question text data: "An object starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the 1st second is 2m. Then its displacement in the 3rd second is ____. A. 6m B. 10m C. 14m D. 18m". The model takes the clear question text generated in the previous step as input. It recognizes that this is a calculation question about the "displacement ratio of uniformly accelerated linear motion with an initial velocity of zero". Applying physical formulas or proportional relationships (displacement ratio 1:3:5...) for calculation: the displacement of 2m in the 1st second corresponds to a ratio of 1, and the displacement in the 3rd second corresponds to a ratio of 5, therefore the displacement is 2m × 5 = 10m. The answer text data is obtained: "B. 10m". The model combines the "complete question text data" and the "answer text data," and supplements the parsing logic connecting the two to generate the final output. The final output solution information is as follows: Question: An object starts from rest and undergoes uniformly accelerated linear motion. Its displacement in the first second is 2m. What is its displacement in the third second? A. 6m B. 10m C. 14m D. 18m

[0090] Analysis: According to the law of uniformly accelerated linear motion with an initial velocity of zero, the ratio of displacements in consecutive equal time intervals is 1:3:5:7…. Given that the displacement in the first second is 2m (corresponding to ratio 1), the displacement in the third second (corresponding to ratio 5) is 2m × 5 = 10m.

[0091] Answer: B. 10m.

[0092] Based on this, this step first produces complete question text data, that is, the model's understanding of the original blurred image. This provides users or subsequent systems with a readable and verifiable intermediate product. If the reconstructed question is accurate, then the solution based on it is more reliable; if the reconstruction is incorrect, the problem can be detected in advance. This increases the transparency and reliability of the entire system. Because the generation process is structured, the final output solution information naturally includes a clear question statement, logical deduction, and final answer. This greatly enhances the educational significance and user-friendliness of the output results. Users not only get the answer but also see how the model understands the question and how it arrives at the answer step by step, perfectly meeting the core needs of educational assistance scenarios.

[0093] This specification provides an image-based problem-solving method, comprising: determining image data of the problem to be solved; parsing the image data to obtain text data of the problem to be solved corresponding to the image data; determining similar question text data corresponding to the image data based on the text data; inputting the similar question text data and the image data to be solved into a problem-solving model, wherein the problem-solving model is a fine-tuned visual language model; and outputting the solution information corresponding to the image data to be solved through the problem-solving model. By parsing the image data to be solved to obtain the text data of the problem to be solved, and determining the corresponding similar question text data based on the text data of the problem to be solved, external prior knowledge is introduced. When the original question image information is incomplete or ambiguous, clear and complete similar question stem text can provide key information supplementation and semantic clarification, thereby effectively enhancing the system's ability to understand the intent of the question. By inputting both similar question text data and the image data of the problem to be solved into the problem-solving model, the model can correctly solve the problem based on the image data with the assistance of the similar question text data, and thus output the solution information corresponding to the image data. This provides a more stable and reliable solution for image problem-solving tasks.

[0094] The following is in conjunction with the appendix Figure 2 Taking the image-based problem-solving method provided in this specification as an example of its application in image-based problem-solving, the image-based problem-solving method will be further explained. Among other things, Figure 2 The flowchart of an image-based problem-solving method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0095] Step 202: Determine the image data to be solved.

[0096] In one embodiment, Xiaoming encountered a physics problem he couldn't solve. He opened a problem-solving app on his phone, clicked the "photo search" function, and took a picture of the problem in his workbook. The app received the image taken by Xiaoming and officially identified it as the "problem image data to be solved" that needed to be processed.

[0097] Step 204: Analyze the image data to be solved, determine the text region in the image data to be solved, perform character recognition on the text region, and obtain the recognized text data corresponding to the text region.

[0098] In one embodiment, the software uses a layout analysis model to outline three main text regions in the image. Then, it performs OCR recognition on each of these three regions. Due to the blurriness of the image, the recognition results may be incorrect. For example, the question stem region might be recognized as: "A car slides down a ramp from a stationary position" (where "rod" is incorrectly identified as "up"); option A might be recognized as: "A. Speed ​​greater than 4.5 m / s"; and option B might be recognized as: "B. Speed ​​less than 4.5 m / s".

[0099] Step 206: Input the identified text data into the semantic recognition model, determine the semantic information corresponding to the identified text data through the semantic recognition model, sort the identified text data according to the semantic information, and generate the text data to be solved corresponding to the image data to be solved based on the sorting result.

[0100] In one embodiment, the software sends three text segments—"A. A cart slides down an inclined plane from a standstill," "A. Speed ​​greater than 4.5 m / s," and "B. Speed ​​less than 4.5 m / s"—to a semantic recognition model. The model determines that the first segment is the "question stem," and the second and third segments are "Option A" and "Option B," respectively. The software sorts the segments according to the semantic logic of "question stem first, options A, B in sequence." This ultimately generates a structured text data of the question to be solved: "A cart slides down an inclined plane from a standstill. A. Speed ​​greater than 4.5 m / s B. Speed ​​less than 4.5 m / s."

[0101] Step 208: Determine the question type corresponding to the image data to be solved based on the text data to be solved, and determine the question database corresponding to the image data to be solved based on the question type.

[0102] In one embodiment, the software analyzes the keywords "car", "incline", "sliding down at rest" and "speed" in the text to determine that the question type is "physics-multiple choice question", and based on this, it locks the "physics multiple choice question database" specifically used for retrieval.

[0103] Step 210: Calculate the text statistics information corresponding to the text data of the problem to be solved, determine the similarity between the text data of the problem to be solved and the problems in the problem database based on the text statistics information, and select similar question text data corresponding to the image data of the problem to be solved from the problem database according to the question similarity.

[0104] In one embodiment, the software converts Xiaoming's question text into a mathematical representation (such as a TF-IDF vector), highlighting keywords such as "car," "incline," "stationary," "sliding down," and "speed." In the database, it calculates the vector for each question in the same way and calculates their similarity to Xiaoming's question vector. It finds the question with the highest similarity, whose clear stem is: "A car slides down an inclined plane from rest at the top of the plane. The inclined plane is 1m high. Neglecting friction, find the speed at the bottom." The software extracts this text as similar question text data.

[0105] Step 212: Generate multimodal problem-solving data based on similar problem text data and problem image data, and input the multimodal problem-solving data into the problem-solving model.

[0106] In one embodiment, the software generates multimodal problem-solving data according to a fixed format: [Instruction] Please refer to the solution image for similar problems. [Similar Problem] A car slides down an inclined plane from rest at the top. The inclined plane is 1m high. Neglecting friction, find the speed at the bottom. [Image]. This data package is then input into a visual language model (problem-solving model) trained with a large number of problems.

[0107] Step 214: Construct complete question text data based on similar question text data and question image data using the problem-solving model; solve the problem based on the complete question text data to obtain the answer text data corresponding to the question image data; generate and output the solution information corresponding to the question image data based on the complete question text data and answer text data.

[0108] In one embodiment, the model combines a clear, similar question (mentioning "the inclined plane is 1m high") with visual information from the original image to correct OCR errors and generate clear question text: "A toy car slides down an inclined plane from rest. If the inclined plane is 1m high and friction is ignored, its speed at the bottom is ____. A. greater than 4.5m / s B. less than 4.5m / s." Based on this clear question, the model calculates and obtains the answer as less than 4.5m / s. The model organizes the language and finally presents it to Xiaoming on the application interface: Title: (Clearly reconstructed title text) Analysis: "According to the law of conservation of mechanical energy, the gravitational potential energy of the cart is completely converted into kinetic energy: mgh = (1 / 2)mv². Substituting h = 1m and g ≈ 10m / s², we get v ≈ 4.47m / s, so the speed is less than 4.5m / s." Answer: B.

[0109] In the end-user's experience, Xiaoming photographed a somewhat blurry question. The software not only provided the correct answer B, but also restored the blurry question to clear text, along with a detailed physical explanation. Xiaoming understood the key points and solution method of the question. Throughout the process, the introduction of clear and similar questions was crucial in helping the system overcome image blur and accurately understand the question's meaning.

[0110] Corresponding to the above method embodiments, this specification also provides embodiments of an image-based problem-solving device. Figure 3 A schematic diagram of an image-based problem-solving device according to one embodiment of this specification is shown. Figure 3 As shown, the device includes: The parsing module 302 is configured to determine the image data of the problem to be solved, parse the image data of the problem to be solved, and obtain the text data of the problem to be solved corresponding to the image data of the problem to be solved; The determination module 304 is configured to determine similar question text data corresponding to the question image data based on the question text data to be solved; The input module 306 is configured to input the similar question text data and the question image data to be solved into the problem-solving model, wherein the problem-solving model is a fine-tuned visual language model; The output module 308 is configured to output the solution information corresponding to the image data of the problem to be solved through the problem-solving model.

[0111] Optionally, the parsing module 302 is further configured to parse the image data to be solved, determine a text region in the image data to be solved; perform character recognition on the text region to obtain the recognized text data corresponding to the text region; and determine the text data to be solved corresponding to the image data to be solved based on the recognized text data.

[0112] Optionally, the parsing module 302 is further configured to input the identified text data into a semantic recognition model, determine the semantic information corresponding to the identified text data through the semantic recognition model, sort the identified text data according to the semantic information, and generate the text data to be solved corresponding to the image data to be solved based on the sorting result.

[0113] Optionally, the determining module 304 is further configured to determine the question type corresponding to the question image data based on the question text data to be solved, and determine the question database corresponding to the question image data based on the question type; and filter the question database based on the question text data to be solved to obtain similar question text data corresponding to the question image data to be solved.

[0114] Optionally, the determining module 304 is further configured to calculate the text statistics information corresponding to the text data to be solved; determine the question similarity between the text data to be solved and the questions in the question database based on the text statistics information; and select similar question text data corresponding to the image data to be solved from the question database according to the question similarity.

[0115] Optionally, the input module 306 is further configured to generate multimodal problem-solving data based on the similar question text data and the problem-to-solve image data; and input the multimodal problem-solving data into the problem-solving model.

[0116] Optionally, the output module 308 is further configured to collect sample question images and obtain sample answers corresponding to the number of sample question images; determine similar question texts corresponding to the sample question images and construct a training sample set based on the sample question images, the sample answers, and the similar question texts; and use the training sample set to fine-tune the visual language model to obtain a problem-solving model.

[0117] The above is a schematic scheme of an image-based problem-solving device according to this embodiment. It should be noted that the technical solution of this image-based problem-solving device and the technical solution of the image-based problem-solving method described above belong to the same concept. For details not described in detail in the technical solution of the image-based problem-solving device, please refer to the description of the technical solution of the image-based problem-solving method described above.

[0118] Figure 4 A structural block diagram of a computing device 400 according to one embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0119] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0120] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0121] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0122] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image-based problem-solving method.

[0123] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image-based problem-solving method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image-based problem-solving method described above.

[0124] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the image-based problem-solving method described above.

[0125] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the image-based problem-solving method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image-based problem-solving method described above.

[0126] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described image-based problem-solving method.

[0127] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the image-based problem-solving method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the image-based problem-solving method described above.

[0128] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0129] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0130] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. An image-based problem-solving method, characterized in that, include: The image data of the problem to be solved is determined, and the image data of the problem to be solved is parsed to obtain the text data of the problem to be solved corresponding to the image data of the problem to be solved; Based on the text data of the problem to be solved, determine the similar text data of the problem to be solved corresponding to the image data of the problem to be solved; The similar question text data and the question image data to be solved are input into the problem-solving model, wherein the problem-solving model is a fine-tuned visual language model; The problem-solving model outputs the solution information corresponding to the image data of the problem to be solved.

2. The method according to claim 1, characterized in that, The image data of the problem to be solved is parsed to obtain the text data of the problem to be solved corresponding to the image data, including: The image data of the problem to be solved is parsed, and the text region is determined in the image data of the problem to be solved; Perform character recognition on the text region to obtain the recognized text data corresponding to the text region; The text data corresponding to the image data to be solved is determined based on the identified text data.

3. The method according to claim 2, characterized in that, Determining the text data corresponding to the image data to be solved based on the identified text data includes: The identified text data is input into a semantic recognition model, and the semantic recognition model determines the semantic information corresponding to the identified text data. The identified text data is sorted according to the semantic information, and the text data to be solved corresponding to the image data to be solved is generated based on the sorting result.

4. The method according to claim 1, characterized in that, Based on the text data of the problem to be solved, similar question text data corresponding to the image data of the problem to be solved are determined, including: The question type corresponding to the image data to be solved is determined based on the text data to be solved, and the question database corresponding to the image data to be solved is determined based on the question type; Based on the text data of the question to be solved, the question database is filtered to obtain similar question text data corresponding to the image data of the question to be solved.

5. The method according to claim 4, characterized in that, Based on the text data of the problem to be solved, the question database is filtered to obtain similar question text data corresponding to the image data of the problem to be solved, including: Calculate the text statistics corresponding to the text data to be solved; Based on the text statistical information, determine the question similarity between the text data to be solved and the questions in the question database; Based on the question similarity, similar question text data corresponding to the image data of the question to be solved is selected from the question database.

6. The method according to claim 1, characterized in that, Inputting the similar question text data and the question image data to be solved into the problem-solving model includes: Multimodal problem-solving data is generated based on the similar problem text data and the problem image data to be solved; The multimodal problem-solving data is input into the problem-solving model.

7. The method according to any one of claims 1-6, characterized in that, The training of the problem-solving model includes: Collect sample question images and obtain the sample answers corresponding to the number of sample question images; Identify the similar question texts corresponding to the sample question images, and construct a training sample set based on the sample question images, the sample answers, and the similar question texts; The visual language model is fine-tuned using the training sample set to obtain a problem-solving model.

8. The method according to any one of claims 1-6, characterized in that, The problem-solving model outputs the solution information corresponding to the image data to be solved, including: The problem-solving model constructs complete problem text data based on the similar problem text data and the problem image data to be solved; Solve the problem based on the complete question text data to obtain the answer text data corresponding to the image data of the problem to be solved; Based on the complete question text data and the answer text data, generate and output the solution information corresponding to the image data of the question to be solved.

9. An image-based problem-solving device, characterized in that, include: The parsing module is configured to determine the image data of the problem to be solved, parse the image data of the problem to be solved, and obtain the text data of the problem to be solved corresponding to the image data of the problem to be solved; The determination module is configured to determine similar question text data corresponding to the question image data based on the question text data to be solved; The input module is configured to input the similar question text data and the question image data to be solved into the problem-solving model, wherein the problem-solving model is a fine-tuned visual language model; The output module is configured to output the solution information corresponding to the image data of the problem to be solved through the problem-solving model.

10. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images