Multi-modal problem solving method and multi-modal problem solving system

Abstract

The embodiment of the invention provides a multi-modal problem solving method and a multi-modal problem solving system.The multi-modal problem solving method comprises the steps that an input image is obtained, and the input image comprises a problem text and a problem graph; analyzing the question text to obtain a question analysis result and a target graphic element; performing vector graphic marking on the target graphic element in the topic graphic to obtain a vector graphic marking result; and based on a vector graphic marking result, reasoning a question analysis result to obtain an answer text. Attention on image details is improved in a vector graphic marking mode, image perception precision is improved, a vector graphic marking result can serve as an intermediate representation in a processing process, interpretability of a problem solving process and understanding of a verification method on the image details are enhanced, and a problem analysis result is reasoned based on the vector graphic marking result, so that the accuracy of image perception is improved. Direct perception of possible errors can be avoided, and the problem solving accuracy is improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of large model technology, and in particular to a multimodal problem-solving method and a multimodal problem-solving system. Background Technology

[0002] With the rapid development of intelligent problem-solving technology, problem-solving tools have begun to shift from basic text recognition and matching to actively generating answers.

[0003] Currently, problem-solving methods, represented by intelligent question answering, combine website analysis, mouse simulation, optical character recognition (OCR) technology, and mathematical calculation formulas to efficiently extract valuable data information from complex online educational resources and provide answers to users' questions.

[0004] However, current problem-solving methods have limited ability to perceive image details when directly solving problems, easily overlooking key information. This leads to inaccurate identification of key elements, spatial relationships, and the number of objects in the image, resulting in a low accuracy rate of the final solution. There is an urgent need for a problem-solving method that can accurately identify the problem and output a high-accuracy answer. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a multimodal problem-solving method. One or more embodiments of this specification also relate to a multimodal problem-solving system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a multimodal problem-solving method is provided, comprising: The process involves: acquiring an input image, which includes question text and question graphics; analyzing the question text to obtain the question analysis results and target graphic elements; performing vector graphic marking on the target graphic elements in the question graphics to obtain vector graphic marking results; and inferring the answer text based on the vector graphic marking results.

[0007] According to a second aspect of the embodiments of this specification, a multimodal problem-solving system is provided, comprising: The image input module is used to acquire input images, which include question text and question graphics; the multimodal problem-solving module is used to analyze the question text to obtain the question analysis results and target graphic elements; to perform vector graphic marking on the target graphic elements in the question graphics to obtain vector graphic marking results; and to reason based on the vector graphic marking results to obtain the answer text.

[0008] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the multimodal problem-solving method described above.

[0009] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the multimodal problem-solving method described above.

[0010] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the multimodal problem-solving method described above.

[0011] The multimodal problem-solving method provided in one or more embodiments of this specification acquires an input image, which includes problem text and problem graphics; analyzes the problem text to obtain problem analysis results and target graphic elements; performs vector graphic marking on the target graphic elements in the problem graphics to obtain vector graphic marking results; and infers the problem analysis results based on the vector graphic marking results to obtain the answer text. Using vector graphic marking improves the focus on image details and enhances image perception accuracy. Furthermore, the vector graphic marking results can serve as intermediate representations in the processing, enhancing the interpretability of the problem-solving process and verifying the method's understanding of image details. Therefore, inferring the problem analysis results based on the vector graphic marking results can avoid potential errors from direct perception and improve problem-solving accuracy. In addition, this method does not rely on predefined detection categories, thus possessing strong universality and adaptability to the recognition of various types of problems. Attached Figure Description

[0012] Figure 1 A flowchart illustrating a multimodal problem-solving method provided in one embodiment of this specification; Figure 2 An input diagram for a multimodal problem-solving method provided in one embodiment of this specification; Figure 3 This is an output diagram of a multimodal problem-solving method provided in one embodiment of this specification; Figure 4 A flowchart illustrating yet another multimodal problem-solving method provided in one embodiment of this specification; Figure 5 A system architecture diagram of a multimodal problem-solving system provided in one embodiment of this specification; Figure 6This is a schematic diagram of the structure of a multimodal problem-solving device provided in one embodiment of this specification; Figure 7 This is a structural block diagram of a computing device provided for one embodiment of this specification. Detailed Implementation

[0013] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “said,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0016] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0018] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0019] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0020] Multimodal: This generally refers to data from multiple different sources or formats. These data sources can include text, images, audio, video, and even more complex data types. In machine learning, "multimodal learning" refers to models being designed and trained to simultaneously process and utilize these input data from different types of data.

[0021] Scalable Vector Graphics (SVG) is a vector graphics format that allows for lossless scaling and supports the drawing of various graphic elements, including rectangles, circles, lines, and text. SVG images are not composed of pixels, but rather of geometric shapes (such as points, lines, curves, polygons, and text) defined by mathematical formulas. This means that images can be infinitely enlarged or reduced without becoming blurry or pixelated, always maintaining sharp and clear edges. SVG images are essentially plain text files, using XML syntax to describe the image. They can be directly modified using a code editor or dynamically manipulated using JavaScript.

[0022] HyperText Markup Language (HTML): Defines the structure and content of a webpage using a set of predefined tags. HTML is more than just a markup language; it's one of the core technologies of the Web, used to create and design documents that can be viewed in a web browser.

[0023] Detail perception: refers to the model's ability to accurately identify and understand key information such as subtle features, geometric elements, and spatial relationships in an image.

[0024] Region of Interest (ROI): refers to a specific area in an image that requires special processing or labeling.

[0025] Open category detection: refers to detection models that can detect and identify object categories that were not seen during training, such as the DINO model (DINO, DIstillation with no labels).

[0026] YOLO (You Only Look Once) algorithm: A revolutionary real-time object detection algorithm that treats object detection as a single regression problem, directly resolving the object's category and location coordinates from image pixels. Renowned for its extremely high speed and accuracy, it is widely used in fields such as intelligent security, autonomous driving, industrial inspection, and medical image analysis, and also provides key technical support for educational technology (such as intelligent problem solving).

[0027] Lossy compression format (JPEG, Joint Photographic Experts Group): A widely used lossy compression standard and image format, with the extension .jpg or .jpeg. To significantly reduce file size, it discards some image data. The higher the compression level, the more data is discarded, resulting in a smaller file size, but the image quality degradation (potentially blurring, color banding, or "mosquito noise") is also more pronounced.

[0028] Portable Network Graphics (PNC): A bitmap image format that supports lossless compression and transparency. Designed to replace the then-patent-restricted GIF format, it has become one of the most widely used standard image formats on the internet. It employs a lossless data compression algorithm. This means that no information is lost during compression and decompression, and the image quality remains completely unchanged. It is ideal for scenarios requiring precise reproduction of the original image.

[0029] Graphics Interchange Format (GIF): A bitmap image format that uses the LZW lossless compression algorithm, ensuring no data loss during compression. A GIF file can contain multiple images (frames) that loop at set time intervals to create simple animation effects.

[0030] Device-Independent Bitmap (BMP / DIB): A raster image file format primarily used in Windows and OS / 2 operating systems. It is one of the most basic and original image formats on the Windows platform. While simple lossless compression using RLE (run-length encoding) is possible, the vast majority of BMP files are stored uncompressed. This means that it preserves the complete pixel data of the image.

[0031] LaTeX markup (LaTeX, Lamport TeX): is a high-quality document preparation system based on the TeX typesetting system, particularly suitable for generating scientific and academic documents containing a large number of mathematical formulas, complex tables, algorithms, and references.

[0032] Graphviz (Graph Visualization Software) is an open-source toolset. Its core function is to read text files written in the DOT language and render them into images in various formats (such as PNG, SVG, PDF, EPS, etc.). It uses sophisticated algorithms for automatic layout, meaning you only need to define nodes and edges, and Graphviz intelligently arranges their positions, avoiding overlap and making the graphs aesthetically pleasing and easy to read.

[0033] DOT language: A simple, declarative domain-specific language (DSL). Its syntax is intuitive, easy to learn and write. It describes the nodes, edges, and connections between them in a graph through text. It supports rich attributes to customize the appearance of nodes and edges (such as color, shape, labels, line styles, etc.).

[0034] OpenCV (Open Source Computer Vision Library) is an open-source software library for computer vision and machine learning. It provides implementations of hundreds of computer vision and machine learning algorithms and is widely used in academic research, industrial products, and commercial projects.

[0035] OpenGL (Open Graphics Library) is a cross-language, cross-platform professional graphics programming interface (API) used to render 2D and 3D vector graphics.

[0036] This specification provides a multimodal problem-solving method, and also relates to a multimodal problem-solving apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0037] Current multimodal large language models generally have low accuracy when answering questions that require precise understanding of image details. These types of questions include ray counting, brick counting, and dense object counting, requiring the model to accurately identify key information such as geometric elements, spatial relationships, and the number of objects in the image.

[0038] Existing technical solutions have the following problems: 1. When multimodal large language models directly solve problems, their ability to perceive image details is limited, and they are prone to missing key information; 2. Traditional object detection models (such as YOLO) have limited category coverage and cannot adapt to diverse question content; 3. Open category detection models (such as DINO) have high accuracy in detecting natural objects, but low accuracy in recognizing geometric elements and non-natural objects; 4. There is a lack of effective intermediate representation mechanisms, and the model cannot gradually analyze and verify its understanding of image details during the problem-solving process.

[0039] Therefore, this specification proposes the following solution to address the problem of insufficient image detail perception in existing multimodal large language models. It enhances the model's detail understanding ability and improves problem-solving accuracy through SVG tagging technology. By having the model first describe image details using SVG tags during the problem-solving process, and then performing inference calculations based on the tagging results, more accurate problem answers are achieved.

[0040] See Figure 1 , Figure 1 A flowchart of a multimodal problem-solving method provided in one embodiment of this specification includes the following steps: Step 102: Obtain the input image, which includes the question text and the question graphic.

[0041] The input image is an image containing information related to the problem. It serves as data for solving the problem and is input into the problem-solving tool to obtain the answer. The problem-related information can be the problem to be solved, auxiliary descriptions of the problem, or solutions to the problem, etc.

[0042] The input image can be an image modality, a video modality, or a single-frame image modality extracted from a video sequence, depending on the actual needs. This specification does not limit the specific choice in this regard. The input image can originate from a publicly available question corpus, be collected through specific channels, or be manually constructed. Specifically, a single-frame image can be extracted from video modal input images obtained from the aforementioned sources, and this is not limited here. Examples include publicly available exam question banks, question sets collected by staff, and question materials uploaded by users.

[0043] Problem-solving tools can be code tools, OCR recognition, question bank matching, NLP technology, knowledge graph indexing, pre-trained question analysis models or large language models, etc. The specific choice depends on the actual situation, and the embodiments in this specification do not limit this.

[0044] The problem text is textual data describing the problem in the input image. It is used to express the problem that needs to be solved, to provide auxiliary descriptions of the problem's content, or to suggest a solution. For example, "Count how many angles are in the image below?" presents the problem to be solved; "The angle of ∠2 is 45°" provides auxiliary descriptions of the problem's content; and "Try to find the pattern in the diagram" suggests a solution. Problem text typically includes text, numbers, and symbols, and often appears in combinations of these. In some cases, it may also include codes or encodings, as well as other semantic content. This specification does not limit this aspect in the embodiments. See also... Figure 2 , Figure 2 This is an input image of a multimodal problem-solving method provided in one embodiment of this specification. As can be seen, the upper left corner of the image is the problem text "Count how many angles are in the picture below?", and the center of the image is the problem graphic. Multiple angles are defined by points such as O, P, C1, C2, C3, and Q.

[0045] The question text can be obtained by parsing the input image. Optionally, there are several ways to parse the input image to obtain the question text. One option is to perform OCR recognition on the input image to obtain the question text. Another option is to extract text from the input image using a multimodal large language model. Yet another possible method is to obtain the question text by labeling the input image based on interactive machine labeling and human assistance; this is not limited here.

[0046] The problem graph is a graphical representation of the problem in the input image, used to provide supplementary descriptions of the problem's content, offer key information, or illustrate solutions. The types of problem graphs include, but are not limited to, geometric figures, charts and tables, function graphs, logic diagrams, etc., but this specification does not limit the specific types used in the embodiments.

[0047] The question image can also be obtained by parsing the input image. Optionally, there are several ways to parse the input image to obtain the question image. One option is to determine the connected pixel blocks in the input image based on connected component analysis, and then determine the question image based on the pixel blocks. Another option is to use a multimodal large language model to parse the input image to determine the question image. This embodiment of the specification does not limit this method.

[0048] Step 104: Analyze the question text to obtain the question analysis results and target graphic elements.

[0049] The question analysis results are an intermediate representation obtained by analyzing the question text. This representation may include the question type and solution approach, used to identify the question type, logical connections, and key points marked with vector graphics. This instructs the problem-solving tool to solve the problem based on the question type and / or solution approach, improving the accuracy of the tool's solution and the precision of the obtained answer text. The question analysis results can also be displayed as visual output. For developers, this increases the transparency of the problem-solving tool and the interpretability of the solution process, facilitating optimization. For users, it provides detailed problem-solving approach analysis, enabling parallel problem-solving and instruction, and improving the user experience.

[0050] Target graphic elements are graphic elements to be marked obtained from analyzing the question text. These elements indicate the valid content or necessary information for solving the problem. Target graphic elements include regions, points, lines, polylines, circles, semicircles and sectors, polygons, braces, text, angles, etc., and vary depending on the question's graphic. For example, for questions analyzing graphic angles, the target graphic elements are the angles and the line segments that make up the angles.

[0051] By parsing the question text, identifying the question analysis results, and determining the target graphic elements that need to be marked, this process combines semantic understanding and image feature recognition. It can obtain accurate processing guidance for the input image based on the textual description of the question text, thereby determining the question graphic and achieving the practical purpose of accurately marking the question graphic.

[0052] There are several ways to analyze the question text to obtain the question analysis results and target graphic elements. One possible method is to extract keywords from the input question text, obtain question keywords, match the question keywords with a preset question type knowledge base, determine the question type and corresponding question analysis strategy template based on the matching results, and analyze the question based on the question analysis strategy template to generate question analysis results and target graphic elements. Another optional approach is to extract the question text using regular expressions, input the extraction results and preset prompts into a pre-trained large language model, and obtain the question analysis results and target graphic elements based on the pre-trained large language model. In this approach, there is a correlation between the preset prompts and the extraction results. Alternatively, the question text can be input into a multimodal large language model, and the question text can be analyzed based on the pre-trained multimodal large language model to obtain the question analysis results and target graphic elements.

[0053] by Figure 2 For example, Figure 2 This is an input diagram for a multimodal problem-solving method provided in one embodiment of this specification. Figure 2 The question text and question image are input into a pre-trained large language model. Based on the question text, the large language model outputs the question analysis result: "This question requires calculating the number of angles. Besides the small angles formed by two adjacent rays, we also need to consider the large angles formed by adjacent small angles." Based on the question analysis result, it identifies the target graphic elements in the question image as multiple rays and angles formed by multiple rays. Figure 2 Multiple rays and the angle formed by the rays.

[0054] Step 106: Perform vector graphic marking on the target graphic elements in the question figure to obtain the vector graphic marking results.

[0055] Vector graphic markup is the operation of identifying target graphic elements in a question graphic within an input image. Vector graphic markup can be done using text, XML text, etc., and can highlight the location, name, or attributes of the target graphic element.

[0056] Therefore, by marking vector graphics during the processing of problem diagrams, the problem-solving process can be guided based on the highlighted vector graphics, thus guiding the model to make correct recognitions.

[0057] Vector graphic markup results are the markup results obtained by vector graphic marking of target graphic elements. They can be represented in the form of vector markup codes (such as SVG code), structured data containing information such as paths, coordinates, and attributes, or encoded binary vector formats—there are no limitations. As an intermediate representation for graphic content generation and processing, vector graphic markup uses mathematical formulas such as points, lines, curves, and polygons to describe graphics. Therefore, no matter how much magnification is applied, the image edges remain smooth and clear, without pixelation or blurring. It is particularly suitable for responsive design, preserving the geometric structure and editability of graphics, and facilitating rendering, editing, transmission, and interaction across different platforms, resolutions, and application scenarios.

[0058] SVG tags are text-based XML formats that can be directly written, modified, and embedded in web pages, images, or videos using code. They support style control via CSS and can also manipulate the DOM via JavaScript to achieve animations and respond to user interactions (such as clicks and hovers), making them ideal for building dynamic, interactive graphical interfaces.

[0059] Vector graphics markup results include text labels for regions, points, lines, polylines, circles, semicircles and sectors, polygons, braces, text, and angles. Vector graphics markup results can be multimodal, including text, graphics, or video. For example, text can be SVG scalable vector graphics, LaTeX, Graphviz+DOT, etc. Taking SVG scalable vector graphics as an example, the geometric composition of the graphic can be determined, and the position, size, color, and other attributes of these shapes can be described using mathematical formulas.

[0060] In SVG (Scalable Vector Graphics), each graphic element is defined by a set of instructions, such as a start point, an end point, and a radius (e.g., drawing a circle only requires knowing the center coordinates (cx, cy) and the radius (r)), and is represented using XML-based character or code tags. Each graphic element is represented by a specific tag (e.g., ...). <circle> 、 <rect>(etc.), and styles (such as color, border, etc.) can be set through attributes. Figure 2 Taking the graphic shown as an example, the SVG code generated for this graphic could be: <ai-svg roi="0.265,0.224 0.758,0.793"> <line id="1" x1="0.320" y1="0.717" x2="0.442" y2="0.312" caption="" / > <line id="2" x1="0.321" y1="0.722" x2="0.510" y2="0.371" caption="" / > <line id="3" x1="0.322" y1="0.727" x2="0.655" y2="0.553" caption="" / > <line id="4" x1="0.324" y1="0.734" x2="0.679" y2="0.658" caption="" / > <line id="5" x1="0.324" y1="0.732" x2="0.700" y2="0.727" caption="" / > < / ai-svg> " Among them: the "roi" attribute defines the range of the marked area (coordinates of the top left and bottom right corners), the "line" tag is used to mark the ray, the "x1", "y1", "x2", and "y2" attributes define the starting and ending coordinates of the ray, the "id" attribute assigns a unique identifier to each ray for easy subsequent calculations, and "caption" is a preset field used to describe the attributes of the generated SVG code.

[0061] There are several ways to perform vector graphic labeling on target graphic elements in the question image. One possible approach is to extract the target graphic elements from the question image through image segmentation contour extraction and then perform vector graphic labeling based on these target graphic elements. Another possible approach is to extract the target graphic elements from the question image using a pre-trained image recognition model and then perform vector graphic labeling based on these target graphic elements. A further possible implementation is to extract the coordinate positions of the target graphic elements through pixel recognition and then perform vector graphic labeling based on these coordinate positions.

[0062] By highlighting important parts, users' attention can be guided to focus on the most crucial information, avoiding distractions from unimportant details. For complex images or data charts, highlighting helps users grasp the core content more quickly, understand the main information the image is conveying, significantly improve the effectiveness of information delivery, simplify the information processing process, reduce cognitive load, make products or services more user-friendly, and increase user satisfaction and loyalty.

[0063] Step 108: Based on the vector graphics marking results, reason about the question analysis results to obtain the answer text.

[0064] The answer text is the textual information that parses and / or answers the question text and / or question graphics in the input image. The answer text is generated by problem-solving tools or models after understanding the question analysis results and vector graphic marking results, through methods such as information retrieval, logical reasoning, or content generation. Based on the answer text, users can understand the solution and reasoning process for the question text and / or question graphics; therefore, the answer text can guide and assist users in solving the question, or directly solve the question to obtain the solution result.

[0065] Vector graphics possess characteristics such as clear geometric structure, scalability without loss of quality, and editable primitives. By marking vector graphics (e.g., marking points, lines, angles, regions, function curves, etc.), geometric relationships, quantitative features, and spatial logic within the graphics can be accurately extracted. Combined with the question text (e.g., "Calculate the measure of ∠ABC" or "Calculate the area of ​​the shaded region"), the system can more accurately understand the question, achieving a closed loop from "image understanding" to "semantic reasoning," significantly improving the accuracy and intelligence level of automatic problem-solving.

[0066] Vector graphics are unaffected by resolution, lighting, noise, etc., and the labeling results are stable and reliable. Compared with bitmap-based OCR and object detection, vector labeling can more consistently identify graphic elements and reduce false recognition. Furthermore, this method is applicable to electronic exam papers, textbook illustrations, and other materials from different sources, demonstrating good cross-document generalization ability.

[0067] In one optional implementation, the answer text is obtained by reasoning about the question analysis results based on the vector graphic marking results.

[0068] In another optional implementation, the answer text is obtained by reasoning from the question analysis results based on the vector graphic marking results. This can be done by reasoning from the question analysis results based on both the vector graphic marking results and the question text.

[0069] Integrating vector graphic annotation technology into multimodal problem-solving models can significantly improve the ability to perceive image details. This fills the gap in existing multimodal large language models' insufficient ability to perceive image details, providing a new technical solution for solving problems requiring precise understanding of image details. By using vector graphic annotations as an intermediate representation, the model can more accurately perceive image details and perform precise calculations based on the annotation results, greatly improving the accuracy of solving complex problems.

[0070] There are several ways to deduce the answer text based on the vector graphic marking results and the question analysis results. One possible method is to use computer vision to recognize the vector graphic marking results to obtain the marking recognition results, and then deduce the answer text based on the marking recognition results and the question analysis results.

[0071] Another possible approach is to obtain a pre-trained multimodal large language model for problem-solving, analyze the vector graphics labeling results to obtain labeling result analysis, and infer the problem analysis results based on the labeling result analysis to obtain the answer text and output it.

[0072] The model generates answer text based on vector graphic markers. By using vector graphic markers as intermediate representations, the model can accurately locate graphic details, thereby improving the accuracy of reasoning. Specifically, it assists the reasoning and answer generation processes based on visual information, providing highly relevant and targeted image-based auxiliary content to guide and assist the reasoning process towards the correct path, avoiding model illusions, and generating more accurate answer text. This achieves efficient, accurate, and interpretable intelligent problem-solving.

[0073] by Figure 2 Taking the content as an example, the analysis results of the question can be inferred based on the vector graphic markings to obtain the reasoning text: "We can see that there are $5$ rays originating from the same vertex. We can calculate the number of angles by combining these rays. - There are: $5-1=4$ single angles. - There are: $3$ angles composed of $2$ single angles. - There are: $2$ angles composed of $3$ single angles. - There are: $1$ angles composed of $4$ single angles. Therefore, there are a total of: $4+3+2+1=10$ angles." Based on the reasoning text, the answer text "$10$" can be obtained.

[0074] The solution described in this specification involves acquiring an input image, which includes question text and a question graphic; analyzing the question text to obtain the question analysis result and target graphic elements; performing vector graphic annotation on the target graphic elements in the question graphic to obtain vector graphic annotation results; and inferring the answer text based on the vector graphic annotation results. Using vector graphic annotation improves the focus on image details and enhances image perception accuracy. Furthermore, the vector graphic annotation results can serve as intermediate representations in the processing, enhancing the interpretability of the problem-solving process and verifying the problem-solving tool's understanding of image details. Therefore, inferring the question analysis results based on the vector graphic annotation results can avoid potential errors from direct perception and improve problem-solving accuracy. In addition, this method does not rely on predefined detection categories, thus possessing strong universality and adaptability to the recognition of various types of questions.

[0075] Compared with traditional methods, the embodiments in this specification have the following technical advantages: 1. Enhanced Detail Perception: Vector graphic labeling forces the model to focus on image details, improving perception accuracy. 2. Intermediate Representation Validation: Vector graphic labeling results serve as intermediate representations, validating the model's understanding of image details. 3. Improved Computational Accuracy: Calculations are based on vector graphic labeling results, avoiding errors that may arise from direct perception. 4. Greater Adaptability: It does not rely on predefined detection categories and can adapt to various question types. 5. Low Training Data Requirements: No need for large amounts of labeled data; leverages the general capabilities of large language models.

[0076] In one optional embodiment of this specification, after reasoning from the question analysis results based on the vector graphics marking results to obtain the answer text, the method further includes: The system integrates the question analysis results, vector graphics marking results, and answer text to obtain text output results. Based on the vector graphics marking results, it renders the target graphic elements on the input image to obtain image output results.

[0077] Integration involves associating, fusing, and verifying the consistency of question analysis results, vector graphic annotation results, and answer text. Integration can align the granularity between question analysis results, vector graphic annotation results, and answer text, thereby further optimizing the answer text. It can also display the analysis and problem-solving process to users based on the integration results, providing strong interpretability and a better user experience.

[0078] There are several possible ways to integrate the question analysis results, vector graphic marking results, and answer text to obtain the text output result. One possible approach is to align and verify the information from the question analysis results, vector graphic marking results, and answer text, and then organize the logical order of the three to obtain the text output result.

[0079] Another option is to construct and visualize the reasoning chain based on the question analysis results, vector graphic annotation results, and answer text to generate the text output. Yet another option is to align the integrated question analysis results, vector graphic annotation results, and answer text based on a pre-trained large language model to obtain the text output.

[0080] Rendering is the process of converting data or models (such as vector graphics markers, 3D scene descriptions, text instructions, etc.) into visualized images or screens. This process involves calculating visual attributes such as color, lighting, shadows, texture, and geometric deformation, ultimately generating a visually perceptible result on a screen, paper, or other display medium. In the image output obtained by rendering target graphic elements, the rendered content clearly highlights the key graphic content of the input image. This allows for a visual presentation of these key graphic contents to the user, guiding them to understand the content more quickly, efficiently, and accurately, thus enhancing the user experience.

[0081] by Figure 2 For example, integration could involve combining the problem analysis results from the above steps: "This problem requires calculating the number of angles. In addition to the smaller angles formed by two adjacent rays, we also need to consider the larger angles formed by adjacent smaller angles." and the vector graphics marking results: <ai-svg roi="0.265,0.2240.758,0.793"> <line id="1" x1="0.320" y1="0.717" x2="0.442" y2="0.312" caption="" / > <line id="2" x1="0.321" y1="0.722" x2="0.510" y2="0.371" caption="" / > <line id="3" x1="0.322" y1="0.727" x2="0.655" y2="0.553" caption="" / > <line id="4" x1="0.324" y1="0.734" x2="0.679" y2="0.658" caption="" / > <line id="5" x1="0.324" y1="0.732" x2="0.700" y2="0.727" caption="" / > < / ai-svg> The answer text "$10$" was obtained based on the reasoning text. The answer "$10$" was integrated to obtain the text output result: "### Problem Analysis Result: This problem requires calculating the number of angles. In addition to the small angles formed by two adjacent rays, it is also necessary to consider the large angles formed by adjacent small angles."

[0082] ### Vector Graphic Labeling Results: <ai-svg roi="0.265,0.224 0.758,0.793"> <line id="1" x1="0.320" y1="0.717" x2="0.442" y2="0.312" caption="" / > <line id="2" x1="0.321" y1="0.722" x2="0.510" y2="0.371" caption="" / > <line id="3" x1="0.322" y1="0.727" x2="0.655" y2="0.553" caption="" / > <line id="4" x1="0.324" y1="0.734" x2="0.679" y2="0.658" caption="" / > <line id="5" x1="0.324" y1="0.732" x2="0.700" y2="0.727" caption="" / > < / ai-svg> ## Answer: $10$.

[0083] Rendering can be based on SVG code, generating SVG tags (i.e., vector graphics tags) on the image for further annotation, forming graphic content that replaces or overlays the original image. The rendering result is shown below. Figure 3 , Figure 3 The output diagram of a multimodal problem-solving method provided in one embodiment of this specification shows that the rendering result is overlaid on the original multiple rays in the form of dashed lines, forming annotations for the rays and angles.

[0084] Based on vector graphics annotation results, there are several ways to render target graphic elements on the input image to obtain the image output. One option is to call an open graphics library interface to render the target graphic elements on the input image based on the vector graphics annotation results. Another option is to call a pre-trained large language model to render the target graphic elements on the input image based on the vector graphics annotation results. Yet another option is to call a client-side script-based interactive processing module to render the target graphic elements on the input image to obtain the image output.

[0085] By integrating the problem analysis results, vector graphic marking results, and answer text, the solution implemented in this specification obtains text output results. Based on the vector graphic marking results, the target graphic elements are rendered on the input image to obtain image output results. This approach provides detailed image and text output, offering intuitive and comprehensive information display, greatly enhancing the effectiveness of information delivery, providing users with more comprehensive problem-solving strategies and answer text, meeting users' problem-solving needs, effectively enhancing user experience, and increasing user engagement.

[0086] In one optional embodiment of this specification, the above-mentioned reasoning based on the vector graphic marking results to obtain the answer text includes: The analysis results of the questions are analyzed to generate a reasoning chain; Based on the results of vector graphic marking, a reasoning thought chain is executed to obtain the reasoning thought chain output and the answer text; Integrate the question analysis results, vector graphics marking results, and answer text to obtain the text output results, including: The system integrates the results of question analysis, vector graphic marking, reasoning chain output, and answer text to obtain the text output result.

[0087] A reasoning chain is a sequence of interpretable thought processes that involves understanding a problem and progressively transforming raw information. Starting with understanding the problem input, the reasoning chain proceeds through a series of logically coherent and clearly defined intermediate reasoning steps (such as concept identification, relationship analysis, rule application, computational derivation, or hypothesis verification) to gradually transform raw information into processed intermediate data, ultimately leading to a conclusion and generating a traceable and interpretable answer text.

[0088] The text output is a comprehensive document or data structure integrating the problem analysis results, vector graphic annotation results, reasoning chain output, and the final answer text. The text output comprehensively reflects the entire process and basis from the original problem image to the final answer, providing a clear and detailed solution path. Based on the vector graphic annotation results, the reasoning chain is executed to obtain the reasoning chain output and answer text. By combining highly attention-grabbing vector graphic annotation results with a strongly guided reasoning chain, the system can achieve the synergy of accurately perceiving key visual elements and rigorously executing logical deductions. This results in a highly accurate, completely transparent, traceable, and diagnostically valuable reasoning process and answer, significantly improving the reliability, interpretability, and practicality of the intelligent problem-solving system.

[0089] Still using the above Figure 2 For example, according to Figure 2 The reasoning chain generated by the questions can be as follows: "We can see that there are 5 rays originating from the same vertex. We can calculate the number of angles by combining these rays."

[0090] - There are 5-1=4 individual corners.

[0091] - There are 3 angles composed of 2 single angles. - There are 2 angles composed of 3 single angles. - There is 1 angle composed of 4 single angles. Therefore, there are a total of 4 + 3 + 2 + 1 = 10 angles. There are several ways to analyze the question results and generate a reasoning chain. One option is to match the question analysis results with a thought chain template library to obtain a thought chain template. The thought chain template includes the problem-solving approach corresponding to the question and the markers to be filled in for the question content. Based on the question analysis results, the markers to be filled in are replaced to generate the reasoning chain. Another option is to use a pre-trained large language model to analyze the question analysis results and generate a reasoning chain.

[0092] There are several possible ways to integrate the question analysis results, vector graphic annotation results, reasoning chain output, and answer text to obtain the text output. One possible approach is to align and validate the question analysis results, vector graphic annotation results, reasoning chain output, and answer text, and then establish the logical relationships between them to obtain the text output. Another possible approach is to integrate the question analysis results, vector graphic annotation results, reasoning chain output, and answer text based on a pre-trained large language model to obtain the text output.

[0093] In one possible implementation, the generated text output may not include vector graphic markers. The final text output consists of the question analysis results, the reasoning chain output, and the answer text. This simplifies the output by hiding the vector graphic markers that the user is not interested in, emphasizing key content, and providing the user with a concise and clear visual experience. In other words, the text output is obtained by integrating the question analysis results, the reasoning chain output, and the answer text.

[0094] For example, using a pre-trained large language model to analyze the problem analysis results and generate a reasoning chain could be achieved by inputting the problem analysis results and prompts into the pre-trained large language model. The prompts could be something like, "You are a math problem-solving assistant. Please reason step-by-step to solve the problem based on the following problem analysis results, and output a clear reasoning chain: [Problem Analysis Results] Problem type: Isosceles triangle angle calculation; Given: Vertex angle = 80°; Solve: The degree measures of the two base angles. Hint: The two base angles of an isosceles triangle are equal, and the sum of the interior angles of the triangle is 180°. Please output the reasoning process step by step, with each step containing only one logical deduction." The large language model analyzes the problem based on the prompts. The analysis results are then processed to generate a reasoning chain, such as: "This problem involves finding the angles of an isosceles triangle. Given the vertex angle, we need to find the two base angles. Knowledge application: According to the fundamental theorem of plane geometry, the sum of the three interior angles of any triangle is 180°. Property application: An isosceles triangle has two equal base angles, let each base angle be x degrees. Equation construction: Based on the angle sum relationship, we can set up the equation: x + x + 80° = 180°. Algebraic solution: Simplifying, we get 2x = 100°, solving for x, we get x = 50°. Result verification: 50° + 50° + 80° = 180°, which conforms to the triangle angle sum theorem. Conclusion: The two base angles of this isosceles triangle are both 50°."

[0095] The generated text output is as follows Figure 4 "### Problem Analysis: This problem requires calculating the number of angles. Besides the smaller angles formed by two adjacent rays, we also need to consider the larger angles formed by adjacent smaller angles. We can see that there are $5$ rays originating from the same vertex. We can calculate the number of angles by combining these rays. - There are 5-1=4 single angles. - There are 3 angles formed by 2 single angles. - There are 2 angles formed by 3 single angles. - There is 1 angle formed by 4 single angles. Therefore, there are a total of $4+3+2+1=10 angles."

[0096] Answer: $10. The solution implemented in this specification analyzes the problem analysis results to generate a reasoning chain. Based on the corresponding vector graphic marker results, the corresponding reasoning chain is executed to obtain the reasoning chain output and answer text. By integrating the corresponding problem analysis results, the corresponding vector graphic marker results, the corresponding reasoning chain output, and the corresponding answer text, a text output result is obtained. This approach can accurately perceive key visual elements while coordinating a logically rigorous reasoning chain to produce a highly accurate, completely transparent, traceable, and diagnostically valuable reasoning process and answer, significantly improving the reliability, interpretability, and practicality of the intelligent problem-solving system.

[0097] In one optional embodiment of this specification, the above-mentioned vector graphic marking result includes vector graphic marking code; performing vector graphic marking on the target graphic elements in the title graphic to obtain the vector graphic marking result includes: Based on the preset coding rules for vector graphic elements, code generation is performed on the target graphic elements in the question graphic to obtain vector graphic markup codes; Based on the vector graphics marking results, the target graphic elements are rendered on the input image to obtain the image output results, including: Execute vector graphics markup code to overlay and render target graphic elements on the input image to obtain an enhanced rendering image; From the rendered and enhanced image, the image region containing the target graphic element is cropped to obtain the image output result.

[0098] The coding rules for vector graphic elements are a standardized, structured coding convention followed when describing geometric elements in an image or scene using a vector graphics markup language (such as SVG). These rules explicitly specify how to select the tagging language corresponding to a particular graphic type (e.g., using...). <line>This involves defining how to represent line segments, their geometric properties (such as start and end coordinates), visual styles (such as color and line width), and semantic information (such as identifiers, labels, and types) to ensure that vector tags are consistent, readable, resolvable, and interoperable.

[0099] Following the coding rules of vector graphics elements, unstructured visual information can be transformed into standardized data that is machine-readable, human-understandable, system-calculateable, and platform-compatible. This is an important part of realizing the intelligent image understanding, automated analysis, and interpretable AI of the multimodal problem-solving method in this manual.

[0100] For example, using the SVG vector graphics markup language, supported markup elements include regions, points, lines, polylines, circles, semicircles and sectors, polygons, braces, text, and angles. Among them, a region is used to mark or fill a specific area; a point marks a point at a specified location; a line marks a straight line segment; a polyline marks a polyline connecting multiple points; a circle marks a circle; semicircles and sectors (sector-by-points) mark semicircular or sector-shaped regions; a polygon marks any polygon; a brace marks a brace symbol; text adds text descriptions at specified locations; and an angle marks an angle.

[0101] Based on preset coding rules for vector graphics elements, code generation can be performed on target graphic elements in the question image to obtain vector graphic marker codes. One possible approach is to process the question image using contour extraction from image segmentation to determine the target graphic elements, and then generate codes for these elements according to the preset coding rules. Another approach is to use a pre-trained image recognition model to determine the target graphic elements, and then generate codes for these elements according to the preset coding rules. A further possible implementation involves determining the coordinate positions of the target graphic elements based on pixel identification, and then generating codes for these elements based on the coordinate positions and the preset coding rules.

[0102] Overlay rendering is a graphics processing method that superimposes multiple images, layers, or rendering results according to a certain order and rules, based on rendering. In the embodiments of this specification, overlay rendering can overlay vector graphics generated according to vector graphic markup codes onto the original input image, problem graphic, or target graphic element to obtain an enhanced rendering image. This enhances the attention of problem-solving tools / models, creating a user-readable enhanced rendering image on the user's end, thereby guiding user attention and assisting in problem-solving, providing a better user experience. Based on the above... Figure 2 , Figure 3 The example shown illustrates how overlay rendering can be performed as follows: Figure 3 The dashed lines in the image are superimposed on the edges of each corner of the original image to represent vector graphics.

[0103] Accordingly, a rendered enhanced image is the final image obtained by combining and applying special effects to multiple images, layers, or rendering results in a specific order and according to rules. Special effects can include transparency, blending modes, blur, shadows, etc. Rendered enhanced images can enhance visual expressiveness, provide richer details, or achieve specific effects. In the multimodal problem-solving method of this specification, rendered enhanced images can visually present the marked results to the user, clarify the key points of the problem, provide assistance to the user in solving the problem, and achieve a good user experience.

[0104] There are several ways to execute vector graphics annotation code to overlay and render target graphic elements onto the input image to obtain an enhanced image. One possible approach is to use a graphics processing library to execute vector graphics annotation code and overlay and render the target graphic elements onto the input image to obtain the enhanced image. Another possible approach is to input the vector graphics annotation code into a pre-trained image generation model, and then use the image generation model to overlay and render the target graphic elements onto the input image to obtain the enhanced image.

[0105] After overlay rendering, to further enhance the overlay rendering effect, the image region containing the target graphic element can be cropped from the rendered enhanced image to obtain the image output result. There are several ways to achieve this. One possible approach is to obtain the boundary coordinates and size of the image region containing the target graphic element based on the overlay rendering process, determine the image region containing the target graphic element based on the boundary coordinates and size, and then crop the image region to obtain the image output result. Another possible approach is to use a large language model to identify the rendered image region in the rendered enhanced image, crop the image region containing the target graphic element, and obtain the image output result.

[0106] For example, the rendered enhanced image and the prompt can be input together into a large language model with visual understanding capabilities. The prompt could be, "Please analyze the image content, identify and locate the geometric region that needs to be highlighted in the image. The target shape is: an isosceles triangle and its altitude. Please return the bounding box coordinates (x_min, y_min, x_max, y_max) of the shape and describe the graphic elements it contains." This will obtain the image region where the target graphic element is located, and then crop the image region containing the target graphic element to obtain the image output result.

[0107] The solution implemented in this specification involves generating code for the target graphic element in the problem image according to preset code rules for vector graphic elements, thereby obtaining vector graphic markup code; executing the vector graphic markup code to overlay and render the target graphic element on the input image, thereby obtaining a rendered enhanced image; cropping the image area containing the target graphic element from the rendered enhanced image to obtain the image output result. Based on enhancing the attention of the problem-solving tool / problem-solving model, a user-readable rendered enhanced image is formed on the user's end, thereby guiding the user's attention and assisting in problem-solving, providing the user with a good user experience.

[0108] In one optional embodiment of this specification, the question text is analyzed to obtain the question analysis results and target graphic elements, including: Based on preset question type rules, the question text is analyzed to obtain question analysis results and target graphic elements.

[0109] It should be noted that question type rules refer to a series of pre-defined analysis rules based on different question types (such as multiple choice, fill-in-the-blank, and problem-solving questions). These rules guide the system on how to parse the question text, extract key information, and determine the target graphic elements that need to be marked. These rules combine question type characteristics, semantic structure, and visual requirements to ensure that the system can accurately understand the question's intent and generate corresponding processing instructions.

[0110] For example, the preset question type rules may include ray counting questions, brick counting questions, dense object counting questions, shape classification questions, etc. Among them, ray counting questions can accurately calculate the number of angles by marking all rays; brick counting questions can accurately calculate the number of missing bricks by marking the positions of missing bricks; dense object counting questions can accurately count the number of objects of each type by marking various geometric shapes; and shape classification questions can classify and count objects by marking different categories of shapes.

[0111] For example, regarding Figure 2 The question in the text can be identified as a ray-angle counting question by using a large language model. Then, the question text can be analyzed using the ray-angle counting question rule to obtain the question analysis result: "This question requires calculating the number of angles. In addition to the small angle formed by two adjacent rays, it is also necessary to consider the large angle formed by adjacent small angles." and the target graphic elements: multiple rays and angles formed by multiple rays.

[0112] In addition, the question types may also include questions on graphic symmetry recognition, area calculation, path planning, logic diagram analysis, color filling, ray angle counting, brick counting, dense object counting, graphic classification, etc., depending on actual needs. This specification does not limit these specific types of questions in the embodiments.

[0113] By applying the solutions in the embodiments of this specification, the question text is structured and parsed through preset question type rules (such as keywords, sentence patterns, or semantic logic) to identify its examination intent (such as counting, classification, or calculation). This can transform the abstract question analysis task into a specific visual target, thereby determining the graphic elements that need to be labeled by the model and realizing the mapping from natural language instructions to executable image annotation tasks.

[0114] To ensure the effective execution of the processing, the input image can be quality checked to ensure that the image can be used normally. In one optional embodiment of this specification, after acquiring the input image, the process includes: Perform image size checks on the input image; And / or, Perform image format checks on the input image.

[0115] Image size checking involves verifying and checking the length and width of the input image. This ensures the input image's size and / or aspect ratio are suitable for recognition processing, avoiding extreme sizes and / or proportions. Image format checking involves checking the file format of the input image to ensure it can be recognized and labeled. Image size refers to the width and height of the image, and the unit can be pixels. Common image sizes include: Standard square size: such as 1024*1024 pixels, suitable for applications where landscape or portrait orientation is not required. Vertical rectangle size: such as 720*1280 pixels, commonly used for vertical layouts in mobile phone screens and application design. Horizontal rectangle size: such as 1280*720 pixels, suitable for desktop monitors, television screens, and web banners requiring horizontal display. In practical applications, most image sizes and aspect ratios can be accommodated using the solutions in the embodiments of this specification; for example, a 100*1000 pixel image format is also compatible. The above examples are for illustrative purposes only and are not intended to limit the embodiments of this specification.

[0116] Image format refers to the file format for storing images. Common image formats include JPEG, PNG, GIF, BMP, SVG, etc.

[0117] For example, the system requires the uploaded answer sheet image to have a resolution of 2100×2970 (the pixel size of A4 paper at 300dpi). If the user uploads an image of 1080×1920, the system will prompt "Image size is too small, please retake the photo to ensure clarity and recognition," to prevent the filled-in area from being inaccurately segmented due to the image being too small. Alternatively, the system only supports JPG, PNG, and PDF (after conversion to image) formats. If the user uploads a WebP or HEIC format image, the system will prompt "Unsupported image format, please upload a JPG or PNG file." In practical applications, the image size and image format are not limited to a specific pixel size and can be selected according to the situation. The above example is only a description of an embodiment of this specification and is not intended to limit the embodiments of this specification.

[0118] In the embodiments of this specification, by performing image size checks and / or image format checks on the input image, it is ensured that the input image not only meets the size requirements so that subsequent recognition processing can proceed smoothly, but also that the image file format is correct, ensuring that the image can be accurately read, recognized and labeled by the system.

[0119] In one optional embodiment, the question text is analyzed to obtain the question analysis results and target graphic elements, including: Using a large language model, the question text is analyzed to obtain the question analysis results and target graphic elements; Vector graphic marking is performed on the target graphic elements in the question image to obtain the vector graphic marking results, including: Using a large language model, vector graphics are used to mark the target graphic elements in the question graphics to obtain vector graphics marking results. Based on the vector graphics marking results, reasoning is performed on the question analysis results to obtain the answer text, including: Using a large language model and based on vector graphics marking results, inference is performed on the question analysis results to obtain the answer text.

[0120] The Large Language Model (MLM) is a pre-trained natural language processing model with both natural language understanding and visual understanding capabilities. Based on deep learning technology and trained on massive amounts of text, the MLM can understand the grammatical structure, contextual relationships, and deep semantics of natural language. It can deeply comprehend natural language and attempt to generate coherent, context-relevant answers based on input content, logical reasoning, and domain context. In this specification, the MLM has the ability to process text and image features, and can perform reasoning based on these features to determine the question analysis results and target graphic elements, and output the answer text.

[0121] In practical applications, the above steps can be performed based on prompts and a large language model: acquiring the input image; analyzing the question text using prompts and the large language model to obtain the question analysis results and target graphic elements; vector graphics marking of the target graphic elements in the question image using prompts and the large language model to obtain vector graphics marking results; and reasoning based on the question analysis results using prompts and the large language model to obtain the answer text. By guiding the large language model through prompts, the system sequentially completes question text analysis, target graphic element recognition, vector graphics marking generation, and reasoning based on text and image information, achieving end-to-end understanding and solution of text-image questions. By fully leveraging the advantages of the large language model in semantic parsing, structured generation, and logical reasoning, the system improves the accuracy of the answer text solution and the interpretability of the problem-solving reasoning, significantly enhancing the automated processing capabilities and reliability of answer generation in complex visual-language tasks.

[0122] With its powerful semantic understanding and reasoning capabilities, the large language model can accurately parse the question text and extract the target graphic elements and in-depth analysis results. Compared with traditional methods, it is more intelligent, flexible and robust, significantly improving the automation level and user experience of the graphic understanding system.

[0123] Vector graphic labeling results can enhance the attention of large language models to target graphic elements, guide the large language model in solving problems, instruct the large language model to perform selective processing based on target graphic elements, improve the recognition accuracy of large language models for geometric elements and natural graphics, as well as their ability to analyze spatial relationships, and ultimately improve the accuracy of large language models in identifying and processing questions.

[0124] The solution implemented in this specification analyzes the question text using a large language model to obtain the question analysis results and target graphic elements. It then uses the large language model to perform vector graphic marking on the target graphic elements in the question graphics, obtaining vector graphic marking results. Based on these vector graphic marking results, the large language model is used to reason about the question analysis results to obtain the answer text. This significantly improves the system's ability to understand complex text-based questions and its accuracy in solving them. It also enhances the interpretability of the reasoning process and the structured output capability of the results, achieving an intelligent closed loop from perception to cognition. This provides an efficient, flexible, and scalable technical solution for automated problem-solving, intelligent tutoring, and interactive educational applications.

[0125] In one optional embodiment of this specification, before using a large language model to analyze the question text and obtain the question analysis results and target graphical elements, the following steps are also included: Using a large language model, the question text and question image are extracted from the input image.

[0126] Relying on the ability of large language models to recognize natural language and images, it can automatically identify and separate the question text and question graphics from the input image, greatly improving processing efficiency.

[0127] The solution implemented in this specification uses a large language model to extract the question text and question graphics from the input image. This not only achieves high-precision and automated separation of text and image information and semantic understanding, but also accurately establishes the mapping relationship between text descriptions and graphic elements. This provides a reliable data foundation for subsequent vector labeling, logical reasoning, and answer generation, significantly improving the automation level, understanding depth, and practical application value of the intelligent problem-solving system.

[0128] The following is in conjunction with the appendix Figure 4 Taking the application of the multimodal problem-solving method provided in this manual to ray counting problems as an example, this paper further explains the multimodal problem-solving method. Figure 4 A flowchart of another multimodal problem-solving method provided in one embodiment of this specification, specifically including the following steps: Step 402: Obtain the input ray angle problem image, wherein the input image includes the ray angle problem and the ray angle diagram.

[0129] The system receives ray-angle calculation problem images as input, requiring detail perception. It performs basic quality checks on the images (e.g., image size, format) to ensure they are usable (no complex image preprocessing is required). Input images include ray-angle calculation problems in text format and ray-angle diagrams in image format.

[0130] Step 404: Analyze the ray counting problem image to obtain the counting problem analysis results and target graphic elements.

[0131] The multimodal large language model is used to analyze the ray-based angle counting problem, determine the problem type and the details that need to be marked, select the corresponding analysis method based on the problem type, analyze how to process the ray-based angle counting problem, obtain the angle counting problem analysis results and reasoning chain, and determine the target graphic elements that need to be marked by vector graphics. In the ray-based angle counting problem, the target graphic elements can be the rays that make up each angle.

[0132] Step 406: Perform vector graphic marking on the target graphic elements in the ray corner graphic to obtain the vector graphic marking results.

[0133] Using SVG tags to annotate key details (such as rays, bricks, geometric shapes, etc.) in a ray-angle diagram, vector graphics annotation results are obtained. The vector graphics annotation results show the rays contained in the ray-angle diagram and their positions.

[0134] Step 408: Based on the vector graphics marking results, reason about the results of the angle-counting problem analysis to obtain the answer text.

[0135] The multimodal large language model uses vector graphics marking results as the analysis object, performs reasoning calculations based on the analysis results of the counting and reasoning thought chain, and generates complete solution details and answers as answer text.

[0136] The SVG rendering and answer summary module extracts SVG tag code from the solution of the multimodal large language model, executes the SVG code, renders the tagging results onto the original image, and crops the rendered results to generate a new visual tagging image. Together with text parsing and answer summary, it outputs clearer results.

[0137] The scheme implemented in this specification involves acquiring an input ray-based angle counting problem image, where the input image includes the ray-based angle counting problem and ray-based angle graphics; analyzing the ray-based angle counting problem image to obtain the angle counting problem analysis results and target graphic elements; performing vector graphic marking on the target graphic elements in the ray-based angle graphics to obtain vector graphic marking results; and inferring the angle counting problem analysis results based on the vector graphic marking results to obtain the answer text. Analyzing the ray-based angle counting problem effectively processes the content of the ray-based angle counting problem and the ray-based angle graphics. Performing vector graphic marking on the obtained target graphic elements enables a collaborative understanding of the text and graphics content while strengthening the identification and focus of key areas, improving the perceptual accuracy when analyzing and reasoning about the problem. Inferring the angle counting problem analysis results based on the vector graphic marking results results in a higher accuracy rate for the obtained answer text.

[0138] The following is in conjunction with the appendix Figure 5 This document further explains the system architecture diagram corresponding to the multimodal problem-solving system provided in this manual. See [link / reference]. Figure 5 , Figure 5 A system architecture diagram of a multimodal problem-solving system provided in one embodiment of this specification is provided, which specifically includes the following modules: The image input module 502 is used to acquire an input image, which includes the question text and the question graphic.

[0139] The multimodal problem-solving module 504 is used to analyze the problem text, obtain the problem analysis results and target graphic elements; to perform vector graphic marking on the target graphic elements in the problem graphics, and obtain vector graphic marking results; and to reason based on the vector graphic marking results to obtain the answer text.

[0140] The answer summary module 506 is used to integrate the question analysis results, vector graphics marking results, and answer text to obtain the text output results; The vector graphics rendering module 508 is used to render target graphic elements on the input image based on the vector graphics marking results, and obtain the image output results.

[0141] The method described in this specification can be used to analyze the question text, effectively process the content of the question text and question graphics, and perform vector graphic marking based on the obtained target graphic elements. This enables a collaborative understanding of the text and graphics content while strengthening the identification and focus of key areas, improving the perceptual accuracy when analyzing and reasoning about the question. By reasoning about the question analysis results based on the vector graphic marking results, the correctness of the obtained answer text is higher.

[0142] Corresponding to the above method embodiments, this specification also provides embodiments of a multimodal problem-solving device. Figure 6 This is a schematic diagram of a multimodal problem-solving device provided in one embodiment of this specification. Figure 6 As shown, the device includes: The acquisition module 602 is configured to acquire an input image, wherein the input image includes the question text and the question graphic.

[0143] Analysis module 604 is configured to analyze the question text to obtain the question analysis results and target graphic elements.

[0144] The marking module 606 is configured to perform vector graphic marking on the target graphic elements in the question graphic to obtain the vector graphic marking result.

[0145] The reasoning module 608 is configured to reason about the question analysis results based on the vector graphic marker results to obtain the answer text.

[0146] Optionally, the multimodal problem-solving device also includes an integration module configured to integrate the problem analysis results, vector graphic marking results, and answer text to obtain text output results, and to render target graphic elements on the input image based on the vector graphic marking results to obtain image output results.

[0147] Optionally, the reasoning module 608 is further configured to parse the results of the question analysis, generate a reasoning thought chain, execute the reasoning thought chain based on the vector graphic marking results, and obtain the reasoning thought chain output and the answer text.

[0148] Optionally, the integration module is further configured to integrate the question analysis results, vector graphic marking results, reasoning chain output, and answer text to obtain text output results.

[0149] Optionally, the marking module 606 is further configured to generate codes for target graphic elements in the title graphic according to preset code rules for vector graphic elements, thereby obtaining vector graphic mark codes.

[0150] Optionally, the integration module is further configured to execute vector graphics markup code, overlay rendering of the target graphic elements on the input image to obtain a rendered enhanced image; and crop the image region containing the target graphic elements from the rendered enhanced image to obtain the image output result.

[0151] Optionally, the analysis module 604 is further configured to analyze the question text according to preset question type rules to obtain question analysis results and target graphic elements.

[0152] Optionally, the multimodal problem-solving apparatus further includes a checking module configured to check the image size of the input image; and / or to check the image format of the input image.

[0153] Optionally, the analysis module 604 is further configured to use a large language model to analyze the question text and obtain the question analysis results and target graphic elements.

[0154] Optionally, the tagging module 606 is further configured to use a large language model to perform vector graphic tagging on the target graphic elements in the question graphic to obtain vector graphic tagging results.

[0155] Optionally, the reasoning module 608 is further configured to use a large language model to reason about the question analysis results based on the vector graphics marking results, and obtain the answer text.

[0156] Optionally, the multimodal problem-solving apparatus also includes an extraction module configured to extract problem text and problem graphics from the input image using a large language model.

[0157] The scheme implemented in this specification involves acquiring an input image, which includes question text and a question graphic; analyzing the question text to obtain question analysis results and target graphic elements; performing vector graphic marking on the target graphic elements in the question graphic to obtain vector graphic marking results; and inferring the question analysis results based on the vector graphic marking results to obtain the answer text. Analyzing the question text effectively processes the content of both the question text and the question graphic. Performing vector graphic marking on the obtained target graphic elements enables a collaborative understanding of the text and graphic content while strengthening the identification and focus of key areas, thus improving the perceptual accuracy during question analysis and reasoning. Inferring the question analysis results based on the vector graphic marking results results in a higher accuracy rate for the obtained answer text.

[0158] The above is a schematic scheme of a multimodal problem-solving device according to this embodiment. It should be noted that the technical solution of this multimodal problem-solving device and the technical solution of the multimodal problem-solving method described above belong to the same concept. For details not described in detail in the technical solution of the multimodal problem-solving device, please refer to the description of the technical solution of the multimodal problem-solving method described above.

[0159] Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0160] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0161] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0162] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0163] The processor 720 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the aforementioned multimodal problem-solving method.

[0164] The above is a schematic representation of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the aforementioned multimodal problem-solving method belong to the same concept. Details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the aforementioned multimodal problem-solving method.

[0165] This specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the multimodal problem-solving method described above. It should be noted that the technical solution of this storage medium and the technical solution of the multimodal problem-solving method described above belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the multimodal problem-solving method described above.

[0166] This specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the multimodal problem-solving method described above. It should be noted that the technical solution of this computer program product and the technical solution of the multimodal problem-solving method described above belong to the same concept. Details not described in detail in the technical solution of the computer program product can be found in the description of the technical solution of the multimodal problem-solving method described above.

[0167] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0168] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0169] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0170] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0171] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.< / line> < / rect> < / circle>

Claims

1. A multimodal problem-solving method, characterized in that, include: Obtain the input image, wherein the input image includes the question text and the question graphic; The question text is analyzed to obtain the question analysis results and target graphic elements; The target graphic elements in the title graphic are marked with vector graphics to obtain vector graphics marking results; Based on the vector graphic marking results, the analysis results of the questions are used to deduce the answer text.

2. The method according to claim 1, characterized in that, After obtaining the answer text by reasoning from the question analysis results based on the vector graphics marking results, the process further includes: By integrating the question analysis results, the vector graphic marking results, and the answer text, a text output result is obtained. Based on the vector graphic marking results, the target graphic element is rendered on the input image to obtain an image output result.

3. The method according to claim 2, characterized in that, The process of reasoning based on the vector graphic marker results to obtain the answer text includes: The analysis results of the question are analyzed to generate a reasoning chain; Based on the vector graphic marking results, the reasoning thought chain is executed to obtain the reasoning thought chain output and the answer text; The process of integrating the question analysis results, the vector graphic marking results, and the answer text to obtain the text output results includes: By integrating the question analysis results, the vector graphic marking results, the reasoning chain output, and the answer text, a text output result is obtained.

4. The method according to claim 2, characterized in that, The vector graphic marking result includes vector graphic marking code; the step of performing vector graphic marking on the target graphic elements in the title graphic to obtain the vector graphic marking result includes: According to the preset coding rules for vector graphic elements, code is generated for the target graphic elements in the title graphic to obtain vector graphic markup codes; The step of rendering the target graphic elements on the input image based on the vector graphic marking results to obtain image output results includes: Execute the vector graphics markup code to overlay and render the target graphic elements on the input image to obtain an enhanced rendering image; From the rendered and enhanced image, the image region containing the target graphic element is cropped to obtain the image output result.

5. The method according to claim 1, characterized in that, The analysis of the question text to obtain the question analysis results and target graphic elements includes: Based on preset question type rules, the question text is analyzed to obtain question analysis results and target graphic elements.

6. The method according to claim 1, characterized in that, After acquiring the input image, the process also includes: Perform an image size check on the input image; And / or, The input image is checked for image format.

7. The method according to any one of claims 1-6, characterized in that, The analysis of the question text to obtain the question analysis results and target graphic elements includes: Using a large language model, the question text is analyzed to obtain the question analysis results and target graphic elements; The step of performing vector graphic marking on the target graphic elements in the title graphic to obtain vector graphic marking results includes: Using the large language model, vector graphics marking is performed on the target graphic elements in the question graphic to obtain vector graphics marking results; The process of reasoning based on the vector graphic marker results to obtain the answer text includes: Using the large language model and based on the vector graphics marking results, reasoning is performed on the question analysis results to obtain the answer text.

8. The method according to claim 7, characterized in that, Before using a large language model to analyze the question text and obtain the question analysis results and target graphic elements, the process also includes: Using a large language model, the question text and the question image are extracted from the input image.

9. A multimodal problem-solving system, characterized in that, Includes an image input module and a multimodal problem-solving module; The image input module is used to acquire an input image, wherein the input image includes the question text and the question graphic; The multimodal problem-solving module is used to analyze the question text to obtain the question analysis results and target graphic elements; to perform vector graphic marking on the target graphic elements in the question graphics to obtain vector graphic marking results; and to perform reasoning on the question analysis results based on the vector graphic marking results to obtain the answer text.

10. The system according to claim 9, characterized in that, It also includes an answer summary module and a vector graphics rendering module; The answer summary module is used to integrate the question analysis results, the vector graphics marking results, and the answer text to obtain the text output results; The vector graphics rendering module is used to render the target graphic elements on the input image based on the vector graphics marking results, and obtain the image output results.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The system stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

13. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images