Large model reasoning auxiliary method with imaging function
By visualizing the inference problem of the big model and combining the image and text information of the multimodal large language model, the problem of opacity in the inference process of the big model is solved, the inference ability and interpretability of complex tasks are improved, and the inference ability and interpretability of complex tasks are achieved. Widely used in different fields and issues.
Patent Information
- Application Number
- CN202410876691.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-02
AI Technical Summary
The complexity and opacity of large models make their reasoning process difficult to intuitively understand, and the prior art is not effective in dealing with tasks that require concrete reasoning, especially ignoring visual information.
By embodying abstract inference problems into images, using large language models to accept text descriptions as input, and using self-generated images as enhanced inference information, combining multimodal large language models for joint inference between images and text, generating a graph A thought chain combined with text to explain the reasoning process.
Improve the inference ability of large models in complex tasks, reduce the introduction of prior knowledge, improve the universality and interpretability of methods, enable users to understand and evaluate inference results more intuitively, and enhance the trust of results. , and significantly reduce the cost of adapting to new problems.
Smart Images

Figure CN119026686A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and specifically is a concrete large-model reasoning auxiliary method. Background Art
[0002] In recent years, artificial intelligence technology and related applications have developed by leaps and bounds, and deep learning technology has played a vital role in promoting this. As an important branch of artificial intelligence, deep learning achieves intelligent processing of complex problems by simulating the operation of human brain neural networks. Its core is to build a complex multi-layer neural network model, which combines low-level features to form more abstract high-level representations, thereby achieving automatic learning and understanding of input data.
[0003] With the improvement of computing power and the surge in data volume, deep learning technology has gradually evolved from processing simple tasks to solving more complex and challenging problems. In order to solve complex problems through large amounts of data training, large models have emerged, and their computing power and accuracy far exceed those of traditional small-scale models. With the rapid development of artificial intelligence technology, large neural network models (large models) are increasingly widely used in various fields. However, the complexity and opacity of large models pose challenges to the understanding and application of their reasoning process.
[0004] Existing large model reasoning technologies include thought chain technology, self-consistency technology, and a series of derived technologies, which aim to enhance the reasoning ability of the model by decomposing multiple intermediate "thinking" steps in the problem-solving process. Although these methods have achieved certain results in text reasoning tasks, they are not ideal when dealing with tasks that require concrete reasoning, such as geometry problems or chess problems. The main reason is that the "thinking" steps of these methods are limited to text representation and ignore visual information. There are also some visualization tools such as TensorBoard and Netron, which are used to display the architecture and parameters of neural network models. However, these tools mainly focus on the visualization of the model training stage, and the support for the reasoning stage is relatively limited. Due to the large number of parameters and complex structure, the reasoning process of large models is often difficult to understand intuitively. Traditional visualization methods are difficult to fully and accurately display the reasoning process of large models. Therefore, it is necessary to propose a concrete large model reasoning auxiliary method that combines visual information and text information for comprehensive reasoning. Summary of the invention
[0005] In order to solve the problem that the reasoning process of the above-mentioned large model is often difficult to intuitively understand due to the large number of parameters and complex structure, the purpose of the present invention is to provide a concrete large model reasoning assistance method, which visualizes the abstract reasoning problem into an image, allows the large language model to accept the text description of the problem as input, and uses self-generated images as an additional information channel to enhance reasoning, which can reduce the introduction of prior knowledge and thus improve the universality of the method.
[0006] In order to achieve the above object, the technical solution of the present invention is as follows: a concrete large model reasoning auxiliary method, comprising the following steps:
[0007] Step 1: Enter the question: Use the text description of the original question As input, the problem is initially understood and analyzed through a large language model to understand the needs and background of the problem;
[0008] Step 2, planning stage: identify key meta information required in the stage and extract metadata for subsequent reasoning stage. Metadata includes selecting drawing tools and creating initial images.
[0009] Step 3, iterative reasoning stage: gradually update the image according to the instructions given by the language model, and generate a thinking chain combining images and texts by iterative reasoning on the image;
[0010] Step 4, joint reasoning stage: The large model reasoning algorithm combines the final drawn image with the original question and iteratively generated thought chain and presents it to the multimodal large language model. The multimodal large language model combines the image and text information to interpret and reason about the entire thinking process.
[0011] Step 5: Output results: Based on the results of the joint reasoning of the multimodal large language model, output the original question The final answer.
[0012] Further, in step 1, the original question Compatible with large voice models.
[0013] Further, in step 2, extracting metadata includes the following steps:
[0014] S1, from Select the drawing tool T that is suitable for visualizing the problem;
[0015] S2, the number of iterative reasoning steps n;
[0016] S3. Generate hints P for each iterative reasoning step r ;
[0017] S4. Generate a prompt word or code C for initializing the image 0 .
[0018] Further, in step 2, after importing the prompt word template, use the selected drawing tool T and initialization code C 0 Generate the initial image g 0 .
[0019] Further, in step 3, the iterative reasoning phase includes the following steps:
[0020] S1. In each iteration, according to the current image g t , original question Thinking Track Z t and the hint P of the iteration step r , generate new thinking content z t and the code or prompt word C used to update the image t ;
[0021] S2. Use the drawing tool T and the new code or prompt word C t Update image g t to g t+1 ;
[0022] S3. Put new thinking content into t Add to Thought Track Z t In the formation of Z t+1 ;
[0023] S4. Repeat the above process according to the set number of iterations n until the specified number of iterations is reached or the stopping condition is met.
[0024] Furthermore, in step 4, the joint reasoning phase includes the following steps:
[0025] S1. After the iterative reasoning is completed, the final image g n 、Thinking track Z n and the original question Submit together to the multimodal large language model;
[0026] S2, a multimodal large language model combines image and text information to explain and reason about the entire thinking process;
[0027] S3. In the reasoning process, multimodal chain reasoning prompt technology is used to design a set of prompt words to jointly interpret the image and text content.
[0028] Furthermore, in step 2 and step 3, in the planning phase and the iterative reasoning phase, the structured information in JSON format is output by controlling the model.
[0029] Furthermore, in step 2, in the planning stage, few-sample learning is combined: input and output templates are provided as few-sample data, and the few-sample data is used to plan the large language model.
[0030] Furthermore, in step three, during the iterative reasoning phase, the input and output samples generated in the previous iterative step n are used for few-shot learning, maintaining the reasoning and drawing process.
[0031] Furthermore, in step 2, the drawing tools T include two categories. The first category is the graphics rendering tools that rely on logic programming to render images, including Python Turtle and Matplotlib; the second category is the image generation models that directly generate images based on text prompts, including DALLE3, StableDiffusion, and ControlNet.
[0032] The basic scheme has the following beneficial effects: 1. The present invention improves the reasoning ability of the large language model when dealing with complex tasks by introducing a concrete reasoning auxiliary method. By visualizing abstract reasoning problems into images, the concrete large model reasoning method can reduce the introduction of prior knowledge, thereby improving the universality of the method. This means that it can be applied to different fields and problems without the need to customize specific solutions for each problem. The image and thinking trajectory are gradually updated by iterative reasoning to ensure the consistency and stability of the reasoning process. Each step of reasoning is based on the result of the previous step, thereby building a complete thinking chain. The reasoning process can be visualized into an image, so that users can more intuitively understand and evaluate the results of reasoning. This is particularly important for non-professional users because they may not have the ability to deeply understand complex reasoning processes. By combining a multimodal large language model, it is also possible to jointly interpret and reason image and text information, further improving the interpretability of reasoning. This enables users to understand the basis and process of reasoning more clearly, thereby enhancing their trust in the results. It has strong versatility and flexibility, and it can be widely used in various problems that require concrete solutions.
[0033] 2. The present invention shows significant advantages by innovatively combining small sample learning methods with large language models. Small sample learning is a cutting-edge machine learning strategy designed specifically for situations where labeled data is scarce. In real-world application scenarios, obtaining a large amount of high-quality labeled data is often a time-consuming and costly task. Therefore, the small sample learning method has become the key to solving this problem with its efficient data utilization capabilities. The core of the present invention is that by introducing small sample learning, the large language model can quickly understand and learn the solution to a new problem by relying only on a very small number of similar examples. This strategy greatly reduces the cost of adapting to new problems and improves the generalization ability of the model. For large language models, contextual learning ability is one of its most core capabilities, which enables the model to infer reasonable answers or perform corresponding tasks based on given contextual information. Through small sample learning, the model can refer to similar examples when generating text, thereby avoiding the generation of content that is inconsistent with the facts. This greatly improves the quality and credibility of the text generated by the model.
[0034] 3. Even when facing a completely new task that has never been seen before, the large language model of the present invention can quickly adapt and find a solution to the problem through a small number of examples. The large language model can quickly understand and learn the solution to a new problem by relying only on a small number of similar examples. This powerful ability enables the model to adapt to new tasks in a short time without the need for a large amount of annotated data for retraining. This not only greatly shortens the model deployment cycle, but also significantly reduces the deployment cost, allowing the model to be put into practical application more quickly. When generating text, the large language model sometimes produces content that is inconsistent with the facts, the so-called "hallucination phenomenon". The present invention can make full use of limited annotated data to allow the large language model to learn the solution to the problem more accurately during training. This means that in the testing phase, the model can more accurately understand the context of the problem and generate text that is consistent with the facts. This not only improves the accuracy of text solution, but also enhances the practicality of the model in practical applications.
[0035] 4. In the planning and iterative reasoning steps of the present invention, it is very important to generate structured information. Since the output of the language model is text, it is difficult to output it naturally and let the program parse it to obtain structured information. In order to overcome this difficulty, the concrete reasoning method uses the prompt word method to guide the model to generate a specific type of output. They can be text, tags or other forms of input, aiming to activate specific knowledge or structure inside the model. The model is controlled to generate output in JSON format. When the model receives these prompt words, it will understand and follow these instructions and adjust its output to JSON format. Specifically, these prompt words may include instructing the model to start a new JSON object, add a key-value pair, end an object or array, etc. This output format is very friendly for subsequent parsing work. The program can easily parse JSON data, extract the required information, and use it for various tasks, such as data visualization, decision making, etc. Further improve the quality and accuracy of the model output. By designing appropriate prompt words, the model is guided to focus on the key information in the input and encouraged to generate more accurate and useful output, making subsequent parsing simpler. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 A comparison chart of the large model reasoning auxiliary method visualized in an embodiment of the present invention and the traditional reasoning technology.
[0037] Figure 2 The present invention is a workflow diagram of an example of a large model reasoning auxiliary method for solving a geometric intersection reasoning problem that is visualized in an embodiment of the present invention.
[0038] Figure 3 The diagram is an application diagram of the large model reasoning auxiliary method visualized in the embodiment of the present invention on four different tasks.
[0039] Figure 4 This is a graph showing the performance of the large model reasoning auxiliary method visualized in an embodiment of the present invention in the task of solving geometric intersections with different numbers of graphics.
[0040] Figure 5 This is an overall performance diagram of the large model reasoning auxiliary method visualized in an embodiment of the present invention in the task of solving geometric intersections.
[0041] Figure 6 This is an overall performance diagram of the large-model reasoning auxiliary method visualized in an embodiment of the present invention on the Sudoku task.
[0042] Figure 7 This is a diagram showing the overall performance of the large model reasoning auxiliary method in the time series prediction task visualized in an embodiment of the present invention.
[0043] Figure 8This is an overall performance diagram of the large model reasoning auxiliary method for the traveling salesman problem visualized in an embodiment of the present invention.
[0044] Fig. 9 This is a representation diagram of the image drawing module in the large model reasoning auxiliary method that is concretized in an embodiment of the present invention.
[0045] Fig.10 This is an interpretable diagram of the large model reasoning auxiliary method visualized in the embodiment of the present invention on the time series prediction task.
[0046] Fig.11 This is an interpretable graph of the large model reasoning auxiliary method visualized in an embodiment of the present invention displayed on the traveling salesman problem. DETAILED DESCRIPTION
[0047] The following is further described in detail through specific implementation methods:
[0048] The embodiment is basically as shown in the attached Figure 1 -Attached Figure 2 As shown: A concrete large model reasoning auxiliary method includes the following steps:
[0049] Step 1: Enter the question: Use the text description of the original question As input, the large language model is used to initially understand and analyze the problem, understand the needs and background of the problem, and the original problem Compatible with large speech models;
[0050] Step 2, planning stage: identify the key meta information required in the stage, extract metadata for the subsequent reasoning stage, metadata includes selecting drawing tools and creating initial images, combining few-sample learning to provide input and output templates as few-sample data, using few-sample data to plan for the large language model, and outputting structured information in JSON format by controlling the model; extracting metadata includes the following steps:
[0051] S1, from Select the drawing tool T that is suitable for visualizing the problem;
[0052] S2, the number of iterative reasoning steps n;
[0053] S3. Generate hints P for each iterative reasoning step r ;
[0054] S4. Generate a prompt word or code C for initializing the image 0 ;
[0055] S5. After entering the prompt word template, use the selected drawing tool T and initialization code C 0 Generate the initial image g 0 .
[0056] Step 3, iterative reasoning stage: gradually update the image according to the instructions given by the language model, generate a thought chain combining images and texts by iterative reasoning on the image, use the input and output samples generated in the previous iterative step n for few-shot learning, and maintain the reasoning and drawing process; iterative reasoning includes the following steps:
[0057] S1. In each iteration, according to the current image g t , original question Thinking Track Z t and the hint P of the iteration step r , generate new thinking content z t and the code or prompt word C used to update the image t ;
[0058] S2. Use the drawing tool T and the new code or prompt word C t Update image g t to g t+1 ;
[0059] S3. Put new thinking content into t Add to Thought Track Z t In the formation of Z t+1 ;
[0060] S4. Repeat the above process according to the set number of iterations n until the specified number of iterations is reached or the stopping condition is met.
[0061] Step 4, joint reasoning stage: The large model reasoning algorithm combines the final drawn image with the original question and iteratively generated thought chain and presents it to the multimodal large language model. The multimodal large language model combines the image and text information to interpret and reason about the entire thinking process. Joint reasoning includes the following steps:
[0062] S1. After the iterative reasoning is completed, the final image g n 、Thinking track Z n and the original question Submit together to the multimodal large language model;
[0063] S2, a multimodal large language model combines image and text information to explain and reason about the entire thinking process;
[0064] S3. In the reasoning process, multimodal chain reasoning prompt technology is used to design a set of prompt words to jointly interpret the image and text content.
[0065] Step 5: Output results: Based on the results of the joint reasoning of the multimodal large language model, output the original question The final answer.
[0066] The specific implementation process is as follows: As input, the exploration space is expanded to To enhance the reasoning ability of the multimodal large language model. It is then implemented by algorithms in three stages: planning, iterative reasoning, and joint reasoning. The task of the planning stage is to identify the key meta-information required for subsequent stages, including selecting drawing tools, creating initial images, etc. In the iterative reasoning stage, the algorithm will gradually update the image according to the instructions given by the language model. In order to maintain a coherent reasoning trajectory, each iteration will generate an immediate thought. The thought here refers to a string that expresses the thinking content of this step in text. Finally, in the joint reasoning stage, the algorithm combines the final drawn image with the original question and the iteratively generated thought chain and presents it to the multimodal large language model to obtain the final answer.
[0067] In the mind tree, s t Represents the original problem plus the expanded tree structure. Initial state s 0 is set to That is, a set consisting only of the original questions. In each subsequent step, the state is updated as follows: in Representation in language space The arbitrarily large model reasoning auxiliary method proposed in.
[0068] This technology will explore the space from purely linguistic space Expanding to multimodal space in The pixel space representing the image. As mentioned above, in certain specific problems that require concrete information as an aid, using visual results as a space for thinking exploration will significantly enhance its reasoning ability. The core challenge of this idea is to develop a method that can be used in the joint space. An algorithm that effectively explores the problem and maximizes the success rate of problem solving.
[0069] In the first planning stage, the algorithm will accept the text question as input and extract metadata for the subsequent reasoning stage through analysis of the large language model. The information extracted in this stage includes: (1) (1) Select a drawing tool T suitable for problem visualization; (2) the number of iterative reasoning steps n; (3) the prompt P for each iterative reasoning step r ; and (4) using T to prepare the initialization code or prompt word C of the initial image 0 . This step can be formalized as shown in the following equation:
[0070]
[0071] g0 ←T(C 0 )
[0072] in, Represents the language model with prompt engineering involved in this stage. After the prompt word template is introduced, is the initial image.
[0073] By putting the tools for drawing images, how to use them, and the usage scenarios into the prompts of the large language model in the form of contextual learning, and using the contextual learning ability of the large language model, it can automatically learn to adapt different drawing tools to different problems. This technology provides two types of tools for the large language model. The first type is graphics rendering tools that rely on logic programming to render images, including Python Turtle and Matplotlib. The second type is image generation models that directly generate images based on text prompts, including cutting-edge text-to-image models such as DALLE3, StableDiffusion, and ControlNet. Therefore, the tool library can be defined as
[0074] In the subsequent iterative reasoning stage, the algorithm generates a chain of thought combining images and text by iterative reasoning on the image. t Will be generated with thinking z t (i.e. the textual interpretation of the current state) is updated synchronously. This process is described by the following equation:
[0075]
[0076] Z t+1 ←Z t ∪{z t}
[0077] g t+1 ←T(C t ),t∈[1,n]
[0078] Here, C t Represents a code snippet or prompt word used to guide image update, Z t is a thought chain initialized to an empty set. Note that whether iterative updates are needed depends on the nature of the task itself. If the first-stage algorithm sets n = 0, the process will skip this stage.
[0079] In the final joint reasoning stage, the final image g is obtained after the iterative reasoning. n and thinking track Z n, together with the original question, it will be submitted to the multimodal large language model. The multimodal large language model can understand images and texts at the same time and provide a summary answer for all the thinking processes. In this step, the concrete large model reasoning method uses the multimodal chain reasoning prompt technology, that is, a set of prompt words is designed so that the large model will jointly interpret the image and text content before giving the final answer to enhance its reasoning ability. This process can be described by the following equation:
[0080]
[0081] in, It represents a large language model that uses multimodal thought chain prompts. is the final answer.
[0082] The above process is a three-stage implementation of concrete large-model reasoning. This method not only minimizes the introduction of prior knowledge to improve the universality of the method, but also can visualize abstract reasoning problems and improve the ability of large models to solve problems.
[0083] At the same time, this method can also be combined with the few-shot learning method. Few-shot learning is a powerful strategy that can solve problems by using a very small number of similar examples and leveraging the contextual learning capabilities of a large language model. This method not only allows the large language model to quickly understand and learn how to solve a new problem, but also alleviates the hallucination phenomenon of the large model and effectively improves the accuracy of the solution. Few-shot learning and the concrete reasoning method are orthogonal, and the two can be used together. By making some simple modifications in the concrete reasoning method, few-shot learning can be effectively integrated into the concrete large model reasoning.
[0084] For a concrete reasoning method that applies few-shot learning, its design ideas can be discussed in three stages. The planning process in the first stage is a static process and plays a crucial role in whether the final problem can be successfully solved. Therefore, in this stage, few-shot learning can be used to provide a series of input and output templates, where the input is the problem description and the output is the metadata corresponding to the problem description. These samples are artificially fabricated. In the second stage of iterative reasoning, the few-shot learning method can also be used. The algorithm will use the n input and output samples obtained from the previous n-step reasoning to perform n-sample learning to maintain stability during dynamic reasoning and drawing. Note that when the number of previous reasoning steps is less than n rounds, only the data of all previous steps is used, and no samples are used in the first round of iterative reasoning. Finally, in the joint reasoning stage, the algorithm does not need any samples. The entire stage is reasoned in a zero-sample manner, and the process is completely guided by the prompt words.
[0085] In terms of specific implementation details, it is very important to generate structured information in the planning and iterative reasoning steps. Since the output of the language model is text, it is difficult to naturally output and let the program parse to obtain structured information. In order to overcome this difficulty, the concrete reasoning method uses the prompt word method to control the model to generate JSON format output, making subsequent parsing easier. The concrete large model reasoning method is a highly versatile auxiliary reasoning algorithm that can be widely used for problems that require concrete solutions.
[0086] Experimental application and verification
[0087] This experiment is a concrete application case of the large model reasoning auxiliary method on four problems, including: Intersection Count, Sudoku Puzzle, Time Series Prediction, and Traveling Salesman Problem TSP. The task description and input and output samples are as follows: Figure 3 The following application cases are based on the "gpt-4-vision-preview" model, which is the most advanced multimodal language model with vision capabilities in the current field. The default sampling temperature for all cases is set to 0.0.
[0088] 1. Application of the task of geometric intersection number problem
[0089] Various geometric shapes specified by descriptive language are used to calculate the number of intersections between sentences. Given an input description: "There is a line segment from (-2.5,-1) to (-0.5,-1), and a circle with a radius of 1 and a center at (-1.5,-1), the correct output should be "2", indicating that there are 2 intersections between the circle and the line segment. This task demonstrates the accuracy of the model in understanding geometric space. This problem requires the algorithm to have spatial imagination and be able to understand the shapes in Euclidean space based on abstract text. Therefore, concrete assistance is very much needed, and the concrete large model reasoning method provides a new approach to this problem.
[0090] Through comparative experiments, this paper verifies the performance of the concrete large model reasoning method and three large language model-based algorithms in solving geometric intersection problems. These benchmark algorithms include: standard input and output prompt method, thinking chain prompt method and its self-consistent thinking chain method. At the same time, we also introduce three algorithms specifically for solving geometric problems: Inter-GPS, G-LLaVA (only using its text mode) and LLM-SymPy, the latter of which combines mathematical equation solving tools to enhance its ability to calculate intersections, such as Figure 4 and 5 shown.
[0091] Experimental results show that specialized geometric problem solvers such as Inter-GPS and G-LLaVA do not perform well in text input scenarios, mainly because they were originally trained on bimodal datasets and have limited effectiveness in text input scenarios. LLM-SymPy establishes a strong benchmark among large language model-based methods by introducing external mathematical tools for problem solving. The accuracy of the two methods, standard prompts and thought chain prompts, is only 8.5% and 10.0%, respectively. Even in the self-consistent thought chain method using 20 samplings, the accuracy is only slightly improved by 1.5%.
[0092] In contrast, the concrete large model reasoning method described in the present invention significantly improves the accuracy to 17.0% when providing one example. In particular, in complex scenarios involving three or more shapes, the performance of concrete large model reasoning far exceeds that of other large model-based methods. This result shows that concrete large model reasoning demonstrates its significant advantages and great potential in improving problem-solving accuracy and handling complex situations by using visual information to solve complex reasoning problems.
[0093] 2. Task application of Sudoku problems
[0094] The Sudoku problem is a relatively complex logical reasoning problem designed to assess performance in logical reasoning ability. The model receives a Sudoku example described in natural language, where cells filled with numbers are represented by numeric characters and empty cells are represented by dots. The model's task is to give the next action, which is a command in the format of "xy number", where x and y specify the cell coordinates on the board, and the number is the value to be placed in the cell.
[0095] In the embodiment of the present invention, for the Sudoku problem-solving task, a variety of methods are used for comparative experiments to verify their effectiveness. The first is the standard prompt method, which solves the problem by providing a single example with the previous state and the last instruction. Secondly, the self-consistent method is also introduced, which enhances the consistency of the results by sampling k answers and taking the majority vote as the final answer. For a comprehensive evaluation, it is also compared with the mind tree method.
[0096] The experimental results are as follows Figure 6As shown, Corret Rate represents the accuracy rate, and Collision Rate represents the ratio of conflicts between instructions and the Sudoku board. This indicator is used to evaluate the most basic ability to abide by the rules. It can be seen that the standard prompt method performs poorly in successfully solving Sudoku problems, with a success rate of only 8.6% and a high violation rate. This phenomenon reveals the strict requirements of Sudoku for logical reasoning and rule understanding, which poses a huge challenge to language models that rely solely on prompt engineering. Significant performance improvements can be observed by introducing thinking chains and self-consistent thinking chain methods, which enhance the reasoning coherence of language models by providing references to previous steps. In addition, the thinking tree-based algorithm performed the best, with a success rate of 22.6%, because it adopts a tree-structured thinking process.
[0097] However, the concrete large model reasoning method performed best among all the methods, with an accuracy rate of up to 35.5%. The experiment also found that the concrete large model reasoning method can significantly reduce the violation rate, which shows that by integrating the image modality, the language model can be further assisted to understand the rules of the game more effectively. This case proves that combining the concrete large model reasoning method can significantly improve the ability to solve complex problems, and demonstrates its significant advantages in improving the accuracy of solving logic games and compliance with rules.
[0098] 3. Task application of time series prediction problem
[0099] The language model is used as a time series predictor with zero-shot learning capabilities. The model's task is to predict the next n values in the sequence given a series of data points. This task verifies the model's ability to perform numerical analysis and forecasting. In the experiment, we introduced a variety of contrasting algorithms, including: the standard prompt word method, which predicts the next sequence in a single input-output step; the large model autoregressive algorithm, which uses the language model to predict the next n values in an autoregressive manner; and the large model autoregressive algorithm combined with the self-consistent technique, which generates k predictions and calculates their average as the final prediction.
[0100] In addition, the experiment also selected the LLM TIME algorithm as one of the most advanced comparison methods, which introduced data rescaling strategies and word segmentation techniques to improve numerical accuracy. For example, the sequence "8.05, 1, 35" will be segmented into "805, 100, 3500". At the same time, a traditional statistical method, simple moving average, was also used as a comparison benchmark for this task.
[0101] according to Figure 7The experimental results show that the simple moving average, as a traditional and effective time series prediction method, has a mean absolute error (MAE) of 611, which exceeds most language model-based methods. The integration of autoregression and sampling strategies has been proven to be effective. The large model autoregression and self-consistent method significantly reduced the MAE from 767 to 675, showing a significant improvement compared to the standard prompt method. In addition, the LLM TIME algorithm even slightly outperforms the simple moving average by introducing preprocessing techniques for language model processing sequence data.
[0102] On this basis, this experiment introduces a new method that combines a concrete large model reasoning method with LLM TIME preprocessing technology. The experimental results show that this combined method surpasses all the comparative algorithms and demonstrates its excellent performance and wide applicability in time series forecasting. This proves that the present invention is not only innovative in theory, but also has obvious advantages and benefits in practical applications.
[0103] 4. Task Application of the Traveling Salesman Problem
[0104] The traveling salesman problem is a classic NP-hard problem, whose input includes a series of coordinates and the output is the full permutation of the city indexes that the traveling salesman has passed through. This task highlights the performance of the model in dealing with complex optimization problems. In this experiment, we introduced several baseline algorithms, which are divided into two categories: the first category is the traditional traveling salesman problem solver, including Concorde, an exact traveling salesman problem solver using branch and bound search, the nearest neighbor algorithm based on the greedy strategy to select the nearest city, an efficient insertion method, namely the fast insertion method, and a random baseline that randomly selects the path.
[0105] The second category is methods based on large language models, including the standard prompt method that directly generates a list of city indexes; the thinking chain that adds a "step-by-step thinking" instruction to the standard prompt; and the large model autoregressive method that generates city indexes by autoregression and the large model autoregressive and self-consistent method, which samples the next city multiple times and selects the most frequent option. All methods based on large language models are set with 3 learning samples. If the output of the model violates the constraints of the traveling salesman problem (for example, duplicate cities are generated), the algorithm will randomly select a city to continue solving.
[0106] The experimental results are as follows Figure 8As shown in the figure, the results show that the performance of the concrete large model reasoning method surpasses all large model-based algorithms when dealing with the 10-city traveling salesman problem, and is close to the performance of the fast insertion method, with a performance advantage of up to 6.0% in percentage. In addition, when the number of cities increases to 20, the performance of the large model-based algorithm decreases significantly, which is most likely due to the increased complexity of the traveling salesman problem, which poses a challenge to the large model that has not been trained specifically. However, the superiority of the concrete large model reasoning method is more significant, indicating that it is capable of handling higher complexity.
[0107] In addition, the drawing module of the concrete large model reasoning method can also be separated out to analyze the performance of the drawing module and the gain to the final effect. In the present invention, the integrity of the image obtained by the drawing module is first evaluated. The so-called integrity refers to whether the generated image contains all the information of the original problem. In order to evaluate more objectively, this experiment hired human annotators to independently evaluate a set of generated images, and adopted the majority of the results as the final evaluation criteria.
[0108] Fig. 9 The experimental results show that the drawing module exhibits high integrity in different tasks, indicating its stability. In particular, in time series prediction and traveling salesman problem, the module achieved 100% and 98.0% integrity, respectively. In addition, the present invention also examines the impact of correcting inaccurate images on the overall performance of concrete large model reasoning. By manually correcting errors in the image and re-evaluating the performance, significant room for improvement was observed. Experimental data show that in intersection counting and Sudoku tasks, performance improvements of 5.8% and 5.1% were shown, respectively. These results demonstrate the beneficial effect of accurate images on problem solving.
[0109] Finally, in order to better understand the internal mechanism of concrete large model reasoning, this paper conducts two experimental studies to deeply explore the improvement effect of concrete large model reasoning on the solution task. Fig.10 As shown in the figure, for the time series prediction task, the effect of concrete large model reasoning on the task is demonstrated by comparing the sequence represented only by text and the image representation generated by using vision as an enhancement. In the case of only text problem description, even if the sequence shows an obvious periodic pattern, the GPT-4V model has difficulty identifying this periodicity.
[0110] In contrast, when the model was presented with images, it was able to easily identify the periodicity of the sequence. This study suggests that reifying large model reasoning can provide a more efficient and clearer information format, thereby enhancing the model's reasoning ability on prediction tasks. The second study was on the traveling salesman problem, and the results are as follows Fig.11Figure 1. Visual comparison of two solutions to the same instance of the traveling salesman problem, one generated by standard prompting and the other generated by reasoning with the reified large model. The comparison of the two solutions shows that the standard prompting produces many crossing paths, which can be shown to be suboptimal in Euclidean space.
[0111] In contrast, the solution paths of the concrete large model reasoning are less crossed. This improvement is likely due to the clear visualization of the visited paths by the concrete large model reasoning, which provides an effective basis for the model to exclude poor path selection. In summary, the concrete large model reasoning provides effective concrete assistance for complex reasoning problems from different aspects, which can steadily improve the reasoning ability of the large model.
[0112] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0113] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.
Claims
1. A concrete large model reasoning auxiliary method, characterized by: The following steps are involved: Step 1: Enter the question: Use the text description of the original question As input, the problem is initially understood and analyzed through a large language model to understand the needs and background of the problem; Step 2, planning stage: identify key meta information required in the stage and extract metadata for subsequent reasoning stage. Metadata includes selecting drawing tools and creating initial images. Step 3, iterative reasoning stage: gradually update the image according to the instructions given by the language model, and generate a thinking chain combining images and texts by iterative reasoning on the image; Step 4, joint reasoning stage: The large model reasoning algorithm combines the final drawn image with the original question and iteratively generated thought chain and presents it to the multimodal large language model. The multimodal large language model combines the image and text information to interpret and reason about the entire thinking process. Step 5: Output results: Based on the results of the joint reasoning of the multimodal large language model, output the original question The final answer.
2. The method for assisting large model reasoning according to claim 1, characterized in that: In step 1, the original question Compatible with large voice models.
3. The method for assisting large-scale reasoning according to claim 2 is characterized in that: In step 2, extracting metadata includes the following steps: S1, from Select the drawing tool T that is suitable for visualizing the problem; S2, the number of iterative reasoning steps, S3. Generate hints P for each iterative reasoning step r ; S4. Generate a prompt word or code c0 for initializing the image.
4. The method for assisting large-scale reasoning according to claim 3 is characterized in that: In step 2, after the prompt word template is brought in, the selected drawing tool T and initialization code C0 are used to generate the initial image g0.
5. The method for assisting reasoning with a large model according to claim 4, characterized in that: In step 3, the iterative reasoning phase includes the following steps: S1. In each iteration, according to the current image g t , original question Thinking Track Z t and the hint P of the iteration step r , generate new thinking content z t and the code or prompt word c for updating the image t ; S2. Use the drawing tool T and the new code or prompt word c t Update image g t to g t+1 ; S3. Put new thinking content into t Add to Thought Track Z t In the formation of Z t+1 ; S4. Repeat the above process according to the set number of iterations n until the specified number of iterations is reached or the stopping condition is met.
6. The method for assisting reasoning with a large model according to claim 5, characterized in that: In step 4, the joint reasoning phase includes the following steps: S1. After the iterative reasoning is completed, the final image g n 、Thinking track Z n and the original question Submit together to the multimodal large language model; S2, a multimodal large language model combines image and text information to explain and reason about the entire thinking process; S3. In the reasoning process, multimodal chain reasoning prompt technology is used to design a set of prompt words to jointly interpret the image and text content.
7. The method for assisting reasoning with a large model according to claim 6, characterized in that: In step 2 and step 3, during the planning phase and iterative reasoning phase, the control model outputs structured information in JSON format.
8. The method for assisting reasoning with a large model according to claim 7, characterized in that: In step 2, during the planning phase, few-shot learning is combined: input and output templates are provided as few-shot data, and the few-shot data are used to plan the large language model.
9. The method for assisting reasoning with a large model according to claim 8, characterized in that: In step 3, during the iterative reasoning phase, few-shot learning is performed using the input and output examples generated in the previous iterative step n, maintaining the reasoning and drawing process.
10. The method for assisting reasoning with a large model according to claim 9, characterized in that: In step 2, the drawing tools T include two categories. The first category is the graphics rendering tools that rely on logic programming to render images, including PythonTurtle and Matplotlib; The second category is image generation models that generate images directly based on text prompts, including DALLE3, StableDiffusion, and ControlNet.
Citation Information
Patent Citations
Multi-modal large language model training method and system based on multi-modal encoder
CN117218498A
Multi-modal large language model training method, electronic equipment and storage medium
CN117409431A
Relationship extraction method based on multi-modal large language model
CN117972121A
Hybrid context nerve-symbol collaborative reasoning method driven by large language model
CN118170913A
Bimodal forged information detection method based on large language model
CN118171235A
Cited By
Text task processing control method and device based on large language model
CN119623649A
Multi-agent collaborative multi-modal reasoning method and device, electronic equipment and product
CN120725166A
Trajectory prediction system and method based on multi-modal thinking chain
CN120873507A
Multi-modal problem solving method, multi-modal problem solving system, computing device and medium
CN121279457A
