An engineering problem solution roadmap automatic construction method based on a large language model

Abstract

This invention discloses an automatic roadmap construction method for engineering problem solving based on a large language model. The invention constructs a skill point repository dataset and an expert-verified roadmap dataset. An automatic roadmap construction model is built based on these datasets. A pre-defined complex engineering problem is input into the automatic roadmap construction model. An initialization agent generates an initial roadmap. A knowledge agent enhances the initial roadmap by combining internal and external knowledge. A logical critique agent and a granular critique agent critique the logical structure and task node decomposition granularity of the roadmap, respectively. A revision agent improves the roadmap based on the critique results. The improved roadmap is evaluated, and if it reaches a preset passing score, a structured roadmap for solving the complex engineering problem is output. This invention effectively addresses the shortcomings of existing technologies in cultivating the ability to solve complex engineering problems.

Description

Technical Field

[0001] This invention relates to the field of large language models, and more particularly to an automatic method for constructing engineering problem-solving roadmaps based on large language models. Background Technology

[0002] In recent years, a widespread digital education boom has swept the world. The Third World Congress on Higher Education focused on the digital transformation of education, emphasizing shaping the future of global higher education. Breakthroughs in large language models have led to "AI + Education" becoming a global focus, with countries launching "Artificial Intelligence + Education" strategies. The spirit of the Third Plenary Session of the 20th CPC Central Committee clearly pointed out the need to "unswervingly promote the digitalization of education and give full play to the leading role of higher education." Minister Huai Jinpeng pointed out at the World Digital Education Conference that artificial intelligence should drive the transformation of the education industry and empower educational actions. Vice Minister Wu Yan pointed out at the 7th Digital China Construction Summit that a demonstration action for the application of large-scale artificial intelligence models in the education system will be launched. The successful convening of the World MOOC and Online Education Conference has also brought a period of opportunity for digital transformation to the whole society, marking the beginning of the era of smart education.

[0003] In today's education field, cultivating students' ability to solve complex engineering problems has become increasingly important and has received widespread attention. For beginners, quickly solving a complex engineering problem often presents the following three challenges:

[0004] 1. Numerous related knowledge points make it difficult to form a systematic and progressive knowledge framework. For a complex engineering problem, learners need to master a large number of related knowledge points, but these points are often scattered across different textbooks, papers, and online resources, with complex connections and interdependencies between them. For example, "Public sentiment analysis of trending events on Weibo" involves knowledge points from multiple fields such as natural language processing, sentiment analysis, data mining, and machine learning, each with its own subfields and knowledge points. Beginners find it difficult to systematically organize and master these knowledge points in a short period, thus hindering the formation of a systematic and progressive knowledge framework. This leads to low learning efficiency and even frustration. Expert-level roadmaps can provide clear and effective guidance for beginners, helping them find the right direction and solve complex engineering problems step by step.

[0005] 2. Online learning resources are disorganized, making it difficult to locate high-quality, relevant content. Even after beginners have effective guidance and know the direction of their learning, they often wonder: How should I learn this knowledge? Are there any readily available high-quality learning resources? It's easy to see that the internet, as a platform connecting global knowledge, has accumulated a massive amount of learning resources. However, while this vast information repository is rich in resources, its quality varies greatly. Beginners struggle to quickly filter out truly valuable learning resources, and some low-quality or even misleading resources may waste their valuable time and lead to misunderstandings. Therefore, a system that can intelligently filter and recommend high-quality learning resources is particularly important.

[0006] 3. Expert guidance is costly, time-consuming, and difficult to personalize. While expert guidance can provide significant assistance to learners, it faces several limitations in practice. First, expert resources are scarce, and their time is expensive, making one-on-one guidance difficult for each learner. Second, expert guidance is often constrained by time and location, making it impossible to respond promptly to learners' needs, resulting in poor timeliness. Furthermore, each learner's background and needs differ, making it difficult for expert guidance to be fully personalized and meet each learner's unique requirements. These factors limit the widespread application of expert guidance in cultivating problem-solving skills in complex engineering problems.

[0007] Therefore, realizing an automated, low-cost, efficient, and personalized intelligent system to provide expert-level guidance for beginners in solving complex engineering problems has become a topic of great interest in the current education field.

[0008] On the other hand, since OpenAI released its first large-scale dialogue language model, Chat GPT-3.5, in November 2022, AI has rapidly become the focus of attention. Its applications in various fields have been continuously explored and expanded, and the ability of large language models to handle complex tasks has been gradually improved in this process. These models can not only understand complex language structures, but also generate coherent responses based on context, bringing unprecedented opportunities for change to the field of education.

[0009] Against the backdrop of digital transformation in education, designing and implementing intelligent systems by leveraging the powerful text understanding and content generation capabilities of large language models is expected to become an effective way to solve the challenge of cultivating problem-solving skills for complex engineering problems.

[0010] The essence of the automated construction task of engineering problem-solving roadmaps is the generation of structured content. The main technical implementation solutions of the relevant patents that have been found are as follows:

[0011] A method and system for automatically constructing mind maps of review literature:

[0012] This invention discloses a method and system for automatically constructing mind maps of review literature, relating to the field of data processing technology. The method includes: parsing the target review literature, extracting information on the main text, references, and hierarchical objects, and outputting structured text; analyzing hierarchical objects using a large language model to extract macro-level semantic information; interacting with a literature database to match and extract references, and parsing to obtain original reference information; inputting the main text and original reference information into the large language model to parse micro-level semantic information; associating and fusing micro- and macro-level semantic information to obtain mind map construction information; and importing into a mind map construction tool to automatically generate the literature mind map. This achieves the technical effect of deep semantic recognition and improved mind map quality.

[0013] Flowchart generation method, apparatus, electronic device and computer-readable storage medium

[0014] This patent provides a flowchart generation method, apparatus, electronic device, and computer-readable storage medium. The method includes: acquiring natural language text and intermediate language type information of the flowchart; performing a first encoding process on the natural language text to obtain natural language encoding information; performing a second encoding process on the intermediate language type information to obtain intermediate language type encoding information; jointly processing the natural language encoding information and the intermediate language type encoding information to obtain joint language encoding information; decoding the joint language encoding information to obtain intermediate language information; and performing image drawing processing on the intermediate language information to obtain the target flowchart. Through the above technical solution, the flowchart generation process can be made simpler and faster, thereby greatly promoting the rapid development of financial business.

[0015] The disadvantages of existing technologies are as follows:

[0016] (1) Not suitable for automated roadmap construction tasks. Existing methods are merely structured content generation methods, but they have limitations when it comes to automating the construction of roadmaps for complex engineering problems. They focus more on literature reviews or flowchart generation and are not directly optimized for the specific needs of engineering problem-solving roadmaps. Therefore, when applied to real-world complex engineering problem-solving scenarios, they may not be able to effectively provide systematic and progressive guidance for building knowledge networks.

[0017] (2) Lack of personalization and intelligence. Although existing technologies utilize large language models for data processing and content generation, they still fall short in providing personalized guidance. Different learners have different basic knowledge and learning needs, and existing technologies have failed to fully consider these individual differences, thus failing to provide learners with tailored learning paths and resource recommendations, thereby affecting learning outcomes and experience.

[0018] (3) Neglecting learner interaction and feedback mechanisms. Effective interaction between learners and intelligent systems is crucial in cultivating the ability to solve complex engineering problems. However, existing technologies often neglect this, lacking real-time feedback and adjustment mechanisms. This means that the system cannot dynamically adjust the learning path and resource recommendations based on the learner's real-time feedback and progress, thus limiting the system's flexibility and adaptability. Summary of the Invention

[0019] The present invention aims to at least partially solve one of the technical problems in the related art.

[0020] This invention proposes an automatic roadmap construction method for engineering problem solving based on a large language model. For complex engineering problems, it can automatically construct a clearly structured and logically coherent roadmap, providing learners with a systematic and progressive knowledge framework, thus solving the dilemma faced by beginners in organizing knowledge points when encountering complex engineering problems. By combining a knowledge agent with an external knowledge base (skill point repository dataset), the initial roadmap is enhanced with knowledge, supplementing missing key knowledge points to ensure the generated roadmap is comprehensive and accurate, avoiding problems of insufficient knowledge and unreasonable logical structure. A multi-index evaluator is employed to rigorously evaluate the roadmap from four dimensions: logical structure, task decomposition granularity, topic relevance, and content completeness. Only roadmaps that meet all the "passing scores" of the indicators are output, ensuring the high quality of the generated roadmap.

[0021] Another objective of this invention is to propose an automatic roadmap construction device for engineering problem solving based on a large language model.

[0022] To achieve the above objectives, this invention proposes an automatic construction method for engineering problem-solving roadmaps based on large language models, comprising:

[0023] Construct a skills point repository dataset and an expert validation roadmap dataset respectively;

[0024] An automatic roadmap construction model is built based on a skills point repository dataset and an expert-validated roadmap dataset; wherein, the automatic roadmap construction model consists of multiple professional agents based on a large language model and a multi-metric evaluator;

[0025] The system automatically builds a model by inputting a pre-defined complex engineering problem into the roadmap. An initialization agent generates an initial roadmap, and a knowledge agent enhances the initial roadmap by combining internal and external knowledge. A critique-improvement-evaluation loop is then performed. In each loop, the logical critique agent and the granular critique agent critique the logical structure of the roadmap and the decomposition granularity of the task nodes, respectively. The revision agent improves the roadmap based on the critique results. Finally, a multi-index evaluator evaluates the improved roadmap. If it reaches the preset pass score, a structured roadmap for solving the complex engineering problem is output.

[0026] The automatic construction method for engineering problem-solving roadmaps based on large language models in this invention may also have the following additional technical features:

[0027] In one embodiment of the present invention, after generating a structured roadmap for solving complex engineering problems, the method further includes:

[0028] The leaf nodes in the structured roadmap are input into the node resource mounting model, and network resource retrieval is performed based on the tasks of the leaf nodes.

[0029] Based on the large language model and index calculation, all retrieved resources are comprehensively scored, and resources with scores higher than a preset threshold are retained.

[0030] All resources in the structured roadmap are examined through duplicate content detection, usability testing, and a multi-source, multi-format fusion strategy.

[0031] In one embodiment of the present invention, constructing a skill point repository dataset includes:

[0032] Use the MinerU tool to convert the PDF file of the paper into plain text Markdown format;

[0033] Key content of the paper was extracted based on Markdown format, and initial skill points were generated using a large language model.

[0034] The automatically extracted skill points are corrected and improved through manual review and verification, and a vector retrieval database is built using vector models and the Chroma vector database to construct a skill point repository dataset.

[0035] In one embodiment of the present invention, constructing an expert verification roadmap dataset includes:

[0036] Extract a core engineering problem for each paper;

[0037] We use a large language model to generate an initial roadmap for each engineering problem, and use Markdown's multi-level headings to represent the structure and hierarchical relationship of the roadmap.

[0038] The roadmap was adjusted based on expert experience to build an expert-validated roadmap dataset.

[0039] In one embodiment of the present invention, the use of Markdown's multi-level headings to represent the structure and hierarchical relationship of the roadmap includes:

[0040] Treat each skill point or skill module as a node;

[0041] Use Markdown's first, second, or third level headings to represent nodes at different levels. Each node, in addition to its name, carries a level label to uniquely identify its position in the roadmap.

[0042] In one embodiment of the present invention, the multi-metric evaluator is implemented based on LLM and verifies the roadmap based on a scoring system containing four evaluation metrics, including evaluating the logical coherence of the roadmap, evaluating the rationality of the task decomposition granularity, evaluating the relevance of the roadmap to the input engineering problem, and evaluating the richness and completeness of the roadmap content.

[0043] In one embodiment of the present invention, the method further includes:

[0044] Three state-of-the-art LLM models were selected as multi-metric evaluators. Each model was used to score the revised roadmap independently, and the average score of the three models was used as the final score of the roadmap.

[0045] Passing scores for four evaluation metrics were set based on experimental results conducted on an expert-validated roadmap dataset.

[0046] The roadmap that scores above the preset passing score for all four evaluation metrics is output through the multi-metric evaluator; otherwise, the roadmap is returned and re-entered into the critique-improvement-evaluation loop until it passes the evaluation or reaches the maximum number of loops.

[0047] In one embodiment of the present invention, all resources of the structured roadmap are examined through a mechanism of duplicate content detection, usability verification, and multi-source, multi-form fusion, including:

[0048] Availability verification: The Requests library is used to send a HEAD request to each resource URL. If the returned status code is not 200, it indicates that the link to the current resource is invalid and it is removed directly; and a browser simulation engine based on the Selenium library is used to automatically detect the actual rendering effect of the page.

[0049] Duplicate content detection: The similarity between any two resources is calculated based on the SimHash algorithm, and two resources with a distance of less than a preset threshold are considered duplicate content;

[0050] Multi-source and multi-format fusion strategy: In terms of resource sources, each task node is required to contain high-quality resources from at least two different platforms; in terms of resource types, each node is required to contain both video and text resources; when the retained network resources do not meet the multi-source and multi-format fusion strategy, the quality scoring threshold is allowed to be reduced by a preset threshold step to re-select resources until the strategy requirements are met.

[0051] In one embodiment of the present invention, the similarity between any two resources is calculated based on the SimHash algorithm, and two resources with a distance less than a preset threshold are considered as duplicate content, including:

[0052] The name and description of each network resource are concatenated to form the text to be processed, and n feature keywords are obtained through text cleaning and word segmentation using the Jieba open-source library.

[0053] The signature is obtained by aligning and weighting each feature keyword according to its frequency of occurrence. Then, a vector weighting is performed on each feature keyword to obtain a signature. That is, for each bit of the signature, if it is 1, the hash is multiplied positively by the weight, and if it is 0, the hash is multiplied negatively by the weight to obtain the vector of each feature word.

[0054] The SimHash fingerprint signature of the text is obtained by summing the vectors of all feature words and reducing their dimensionality. The similarity is judged by calculating the Hamming distance between the fingerprint signatures of any two network resources. If the distance is lower than a preset threshold, the content is considered to be duplicated. The resource with the highest quality score will be automatically retained, and the remaining duplicate resources will be removed.

[0055] To achieve the above objectives, another aspect of the present invention proposes an automatic roadmap construction device for engineering problem solving based on a large language model, comprising:

[0056] The automatic roadmap generation module is used to input a description of the complex engineering problem to be solved and to perform basic format validation on the input. Based on the engineering problem input by the user, it automatically calls the automatic roadmap construction algorithm to generate a corresponding knowledge roadmap for solving the problem and displays it in a tree structure. The knowledge roadmap contains multiple levels of nodes, each node represents a specific knowledge point or task, and there are clear logical relationships between the nodes.

[0057] The learning resource recommendation module is used to automatically call the node resource mounting algorithm for each leaf node in the roadmap, retrieve and filter high-quality learning resources related to the node task from the Internet, and mount the learning resources to the corresponding node.

[0058] The personalized editing module allows users to flexibly edit the nodes in the generated roadmap according to their own learning progress and existing knowledge.

[0059] The user interaction module is used to provide a user interface and provide feedback on user information.

[0060] This invention presents an automatic roadmap construction method and apparatus for engineering problem solving based on a large language model. By combining a knowledge agent with an external knowledge base, the initial roadmap is enhanced with knowledge, supplementing missing key knowledge points to ensure the generated roadmap is comprehensive and accurate, avoiding problems of insufficient knowledge and unreasonable logical structure. A multi-index evaluator is employed to rigorously evaluate the roadmap from four dimensions: logical structure, task decomposition granularity, topic relevance, and content completeness. Only roadmaps that meet all the "passing scores" are output, guaranteeing high-quality generated roadmaps. It meets the personalized needs of different users for learning paths. It ensures high-quality and diverse resources to satisfy different user learning preferences, improving learning effectiveness and satisfaction. It achieves intelligent dynamic adjustment and optimization. It can dynamically adjust learning paths and resource recommendations based on real-time user feedback and progress, enhancing the system's flexibility and adaptability, and solving the problem of lacking real-time feedback and adjustment mechanisms in existing technologies.

[0061] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0062] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0063] Figure 1 This is a flowchart of an automatic construction method for engineering problem-solving roadmap based on a large language model according to an embodiment of the present invention;

[0064] Figure 2 This is a framework diagram of an automatic roadmap construction algorithm according to an embodiment of the present invention;

[0065] Figure 3 This is a front-end framework diagram of an automatic roadmap construction system according to an embodiment of the present invention;

[0066] Figure 4 This is a backend framework diagram of the roadmap automatic construction system according to an embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram illustrating the construction of a skill point repository dataset according to an embodiment of the present invention;

[0068] Figure 6 This is a flowchart illustrating the construction process of an expert verification roadmap dataset according to an embodiment of the present invention.

[0069] Figure 7This is a flowchart of the node resource mounting algorithm according to an embodiment of the present invention;

[0070] Figure 8 This is a flowchart of the SimHash algorithm according to an embodiment of the present invention;

[0071] Figure 9 This is a structural diagram of an automatic roadmap construction system for engineering problem solving based on a large language model, according to an embodiment of the present invention. Detailed Implementation

[0072] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0073] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0074] The following description, with reference to the accompanying drawings, outlines an automatic method and apparatus for constructing engineering problem-solving roadmaps based on a large language model, according to embodiments of the present invention.

[0075] Figure 1 This is a flowchart of an automatic construction method for engineering problem-solving roadmaps based on a large language model according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes:

[0076] S1, construct the skill point repository dataset and the expert verification roadmap dataset respectively;

[0077] S2, an automatic roadmap construction model is built based on a skill point repository dataset and an expert-verified roadmap dataset; wherein, the automatic roadmap construction model consists of multiple professional agents based on a large language model and a multi-metric evaluator;

[0078] S3 automatically constructs a model by inputting a pre-defined complex engineering problem into the roadmap. An initial roadmap is generated by the initialization agent, and the knowledge agent performs knowledge enhancement on the initial roadmap by combining internal and external knowledge. A critique-improvement-evaluation loop is then performed. In each loop, the logical critique agent and the granular critique agent critique the logical structure of the roadmap and the decomposition granularity of the task nodes, respectively. The revision agent improves the roadmap based on the critique results. Finally, the multi-index evaluator evaluates the improved roadmap. If it reaches the preset pass score, a structured roadmap for solving the complex engineering problem is output.

[0079] like Figure 2 , Figure 3 and 4 The diagrams shown represent the framework of the automatic roadmap construction algorithm, the front-end framework of the automatic roadmap construction system, and the back-end framework of the automatic roadmap construction system. The specific implementation involves the following steps:

[0080] First, this invention constructs two datasets in total: a skill point repository dataset and an expert verification roadmap dataset.

[0081] Understandably, the skill point repository dataset primarily focuses on organizing and summarizing various skill points closely related to solving complex engineering problems. These skill points should cover a wide range of engineering fields. The purpose of collecting these skill points is to provide the model with sufficient knowledge reserves in the knowledge enhancement step of the automatic roadmap construction algorithm, so that each generated roadmap has comprehensive guidance.

[0082] The main steps in constructing the skill point repository dataset are as follows: First, the PDF files of master's and doctoral dissertations are converted into plain text Markdown format using the MinerU tool; then, key content of the dissertations is extracted, including the introduction, methodology, and conclusion; next, initial skill points are generated using a large language model; then, the automatically extracted skill points are corrected and improved through manual review and verification to ensure the accuracy, comprehensiveness, and uniqueness of each skill point; finally, a vector retrieval database is constructed using a vector model and the Chroma vector database. Figure 5 As shown.

[0083] Understandably, expert-validated roadmap datasets primarily focus on providing expert-level reference roadmaps for complex engineering problems with research value. These reference roadmaps are considered the optimal roadmaps for solving the corresponding engineering problems and are mainly used to verify the quality of the roadmaps built by the model. Specifically, the generation effect is evaluated by calculating the differences between the roadmap built by the model and the reference roadmap under certain metrics.

[0084] The main steps in constructing the expert-validated roadmap dataset are as follows: First, a core engineering problem is extracted for each paper; then, an initial solution roadmap is generated for each engineering problem using a large language model; finally, domain experts adjust the roadmap to improve its quality. Regarding the roadmap representation, this invention uses Markdown's multi-level headings to clearly represent the roadmap's structure and hierarchical relationships. Specifically, this invention treats each skill point or skill module as a node, using Markdown's first, second, or third-level headings to represent nodes at different levels. Each node, in addition to its name, carries a hierarchy label (e.g., 1, 2, 3) to uniquely identify its position in the roadmap. This representation method is simple, intuitive, and facilitates subsequent data processing and model analysis. Figure 6 As shown.

[0085] Furthermore, for a complex engineering problem, the automatic roadmap construction algorithm will build a well-structured and accurate roadmap to solve the engineering problem.

[0086] Figure 2 The detailed process of the automatic roadmap construction algorithm is shown. The algorithm framework consists of five professional agents (I, K, L, G, R) based on a large language model and a multi-index evaluator. They work together to achieve the automatic construction of roadmaps for solving complex engineering problems.

[0087] Table 1 illustrates the steps of the automatic roadmap construction algorithm: When a complex engineering problem is input into the system, an initial roadmap is first generated by the initialization agent I. Then, the knowledge agent K performs knowledge enhancement on the initial roadmap by combining internal and external knowledge. Then, a "critique-improvement-evaluation" loop is performed. In each loop, two critique agents (logical critique agent L and granular critique agent G) critique the logical structure of the roadmap and the decomposition granularity of the task nodes, respectively. Then, the revision agent R improves the roadmap based on the critique results. Finally, a multi-index evaluator evaluates the improved roadmap. If the preset pass score of the evaluator is reached, it is allowed to be output as the final result; otherwise, the next round of the loop begins.

[0088] Table 1

[0089]

[0090]

[0091] The I agent is responsible for receiving complex engineering problems as input and transforming them into an initial, coarse roadmap. This initial roadmap may have shortcomings such as insufficient knowledge, unreasonable logical structure, unreasonable subtask granularity, and weak thematic relevance, but it provides a basic framework for subsequent agent processing. Its responsibilities can be represented by Equation 2-1.

[0092] y initial ←I(x roadmap Equation (2-1)

[0093] Agent K is responsible for enhancing the initial roadmap by combining external knowledge bases with its own knowledge reserves. The details of constructing the skill point repository dataset, as described earlier, will serve as an external knowledge base for Agent K. Agent K can identify and supplement key knowledge points missing from the initial roadmap, ensuring the comprehensiveness of the roadmap's knowledge. Its responsibilities can be represented by Equation 2-2.

[0094] y knowledge ←K(y initial (knowledge) Equation (2-2)

[0095] L-agent focuses on evaluating the logical structure of the roadmap. This invention defines two logical relationships between tasks: Father-son Logic and Sibling Logic. The former requires that child nodes are direct refinements or execution steps of the parent node's task, while the latter requires that child node tasks under the same parent node have a parallel and progressive relationship. L-agent checks whether the nodes constituting the above relationships meet the above logical requirements, identifies problematic node pairs, briefly explains the reasons, and provides improvement suggestions. Formulas 2-3 represent its responsibilities.

[0096] LC t ←L(y t Equation (2-3)

[0097] The G agent focuses on evaluating the granularity of task decomposition in the roadmap. This invention defines two unreasonable task decomposition granularities: Too Detailed and Too Brief. Too Detailed means the node has been broken down into too many trivial subtasks; Too Brief means the node's content is too rich and requires further decomposition. The G agent analyzes the decomposition of each node, identifies nodes with unreasonable granularity, and provides appropriate adjustment suggestions. Equations 2-4 represent its responsibilities.

[0098] GC t ←G(y t Equation (2-4)

[0099] Agent R is responsible for revising the initial roadmap based on the improvement suggestions from the two critical agents. It receives feedback from agents L and G, and integrates these suggestions to adjust the roadmap. Agent R's improvement work is central to improving roadmap quality, and it may employ various correction strategies, including adjusting the order of task nodes, adding or deleting subtasks, and modifying task descriptions. It's important to note that because the feedback from agents L and G may contradict each other, Agent R will not blindly accept all improvement suggestions, but will only adopt reasonable, actionable, and consistent ones. Its responsibilities can be represented by Formula 2-5.

[0100] y t+1 ←R(y t ,LC t GC t Equation (2-5)

[0101] This invention designs a multi-metric evaluator, implemented based on LLM, focusing on evaluating the quality of generated roadmaps from three dimensions. This invention develops a scoring system containing four evaluation metrics to verify the quality of the roadmaps:

[0102] Logic Structure (LS): Evaluates the logical coherence of the roadmap.

[0103] Granularity Degree (GD): Evaluates the reasonableness of the task decomposition granularity.

[0104] Topic Relevance (TR): Evaluates the topic relevance of the roadmap to the input engineering problem.

[0105] Completeness (CO): Assess the richness and completeness of the roadmap's content.

[0106] This invention selects three state-of-the-art LLM models (Qwen Max, DeepSeek V3, and Gemini2.5FlashPro) as the implementation of the evaluator. Each model will independently score the revised roadmap (out of 10 points), and the average score of the three models will be used as the final score of the roadmap.

[0107] Then, this invention heuristically sets a "passing score" for four indicators by observing the results of a large number of experiments conducted on an expert-validated roadmap dataset. Only roadmaps with scores exceeding the "passing score" in all four indicators are approved and output by the evaluator. Otherwise, the roadmap is returned and re-enters the "critique-improve-evaluate" loop until it passes the evaluation or reaches the maximum number of loops. Its responsibilities can be represented by formulas 2-6.

[0108] E t←E(y t+1 Equation (2-6)

[0109] The node resource mounting algorithm is an evolution of the roadmap automatic construction algorithm. For each leaf node of the roadmap, the node resource mounting algorithm attempts to add appropriate network learning resources to it.

[0110] Figure 7 The detailed process of the node resource mounting algorithm is shown. The algorithm consists of three steps: First, network resource retrieval is performed based on the task of the task node. Then, all retrieved resources are comprehensively scored using a large language model and index calculation, and only resources with scores higher than a preset threshold are retained. Finally, all resources in the roadmap are checked through mechanisms such as duplicate content detection, availability verification, and multi-source and multi-form fusion strategies to ensure their quality.

[0111] To ensure the diversity and comprehensiveness of learning resources, this invention selected several well-known resource websites for searching. Specifically, the search scope of this invention covers Bilibili, Zhihu, Baidu Encyclopedia, and CSDN.

[0112] The resource retrieval methods of this node's resource mounting algorithm mainly include two types: platform-specific API retrieval and search engine API-based general retrieval. This is primarily because different platforms have varying degrees of openness in their API interfaces.

[0113] Platform-Specific API Retrieval. This method refers to directly calling the dedicated API interfaces provided by each platform for information retrieval. Its advantage lies in direct access to the platform's database, thereby obtaining richer and more complete content information. Typically, this method offers faster retrieval speeds and returns results in a standardized format, facilitating subsequent parsing. However, its limitation lies in its reliance on the openness of the platform's API and interface permissions. For the Bilibili platform, this algorithm employs an open-source API retrieval repository. This repository integrates Bilibili's video, article, and other resource retrieval interfaces, supporting content retrieval through keywords, UP (uploader) names, and other methods.

[0114] This method utilizes a universal search engine API. It involves searching by inputting keywords followed by the content platform's suffix. The advantage of this method is its cross-platform search capability, allowing access to a wider range of information without being limited by specific platform APIs or permissions. However, the results often contain a large amount of irrelevant and redundant information and are frequently structurally irregular, requiring further filtering and processing. In this algorithm, we use Microsoft's Bing Search API to perform a universal search of learning resources on platforms such as Zhihu, Baidu Baike, and CSDN. By setting reasonable keyword combinations and filtering conditions, we aim to maximize the accuracy and relevance of the search results.

[0115] The primary purpose of quality scoring filtering is to select high-quality learning resources from the retrieved online resources. Research has found that evaluating the quality of learning resources typically involves multiple dimensions, including but not limited to the accuracy and completeness of the content, topic relevance, timeliness, and user reviews. This algorithm uses a combination of automatic scoring via a large language model and indicator calculation to assess the quality of online resources, thereby selecting high-quality learning resources.

[0116] This invention designs a comprehensive quality assurance mechanism to ensure that the learning resources ultimately mounted on the roadmap are rich, diverse, and of high quality. It mainly includes three aspects: usability verification, duplicate content detection, and multi-source, multi-format strategies.

[0117] Availability verification. Availability verification ensures that learning resources are actually accessible. This algorithm uses the Requests library to send a HEAD request to each resource URL. If the returned status code is not 200, it means the resource link is invalid and is directly removed. However, some resource links are reachable, but the website will display messages such as "resource not found." Therefore, a browser simulation engine based on the Selenium library is also used to automatically detect the actual page rendering effect. When abnormal prompts such as "This content is temporarily unavailable" or "404 Not Found" are detected, the system will also automatically remove the resource.

[0118] Duplicate content detection. Duplicate content detection aims to avoid highly similar content appearing in learning resources from the same node, ensuring the diversity of learning resources. This algorithm calculates the similarity between any two resources based on the SimHash algorithm, and considers two resources with a distance less than 0.05 as duplicate content. Figure 8 The flowchart for calculating text fingerprints using the SimHash algorithm is shown.

[0119] Specifically, for each network resource, its name and description are concatenated to form the text to be processed. Then, text cleaning (removing stop words, special characters, etc.) and word segmentation using the Jieba open-source library are used to obtain n feature keywords. These keywords are then weighted according to their frequency of occurrence. Each feature keyword is then hashed to obtain a signature. Vector weighting is then performed: for each bit of the signature, if it's 1, the hash is multiplied positively by the weight; if it's 0, the hash is multiplied negatively by the weight, resulting in a vector for each feature keyword. Finally, the vectors of all feature keywords are summed and their dimensionality reduced (to 1 if greater than 0, otherwise to 0) to obtain the SimHash fingerprint signature of the text. The similarity between any two network resource fingerprint signatures is then determined by calculating the Hamming distance. If the distance is less than 0.05, the content is considered duplicated, and the system automatically retains the resource with the highest quality score (or, if quality scores are the same, the resource with higher timeliness is selected), while the remaining duplicate resources are discarded. It is worth noting that the above detection process can not only detect resources within the same platform, but also perform cross-platform detection, effectively avoiding resource redundancy issues caused by the same teaching videos and reprinted articles appearing on different platforms.

[0120] A multi-source, multi-format fusion strategy is employed. This strategy ensures the diversity of learning resources through a dual constraint mechanism: regarding resource sources, each task node must contain high-quality resources from at least two different platforms; regarding resource types, each node must simultaneously contain video resources and text resources (tutorials, technical blogs, etc.). When the network resources retained after the aforementioned algorithmic steps do not meet this strategy, the quality scoring threshold is lowered in increments of 0.05 for resource re-selection until the strategy requirements are met. This strategy aims to ensure that learners can acquire knowledge from multiple perspectives and through various formats, thereby improving learning effectiveness and satisfaction.

[0121] The automatic construction method for engineering problem-solving roadmaps based on a large language model according to embodiments of the present invention can automatically construct a well-structured and logically coherent solution roadmap for complex engineering problems. It enhances the initial roadmap by combining knowledge agents with an external knowledge base, supplementing missing key knowledge points. A multi-index evaluator is employed to rigorously evaluate the roadmap from four dimensions: logical structure, task decomposition granularity, topic relevance, and content completeness, generating a high-quality roadmap. The system allows for personalized editing of the generated roadmaps to meet the individualized learning path needs of different users. Suitable online learning resources are added to each leaf node to ensure high quality and diversity, satisfying different user learning preferences and improving learning effectiveness and satisfaction. The system generates roadmaps that meet requirements, enabling intelligent dynamic adjustment and optimization. It can dynamically adjust learning paths and resource recommendations based on real-time user feedback and progress, enhancing the system's flexibility and adaptability and solving the problem of lacking real-time feedback and adjustment mechanisms in existing technologies. Automated roadmap construction and learning resource recommendations reduce learners' time and effort in finding learning paths and resources, lowering learning costs. Through quality scoring filtering, duplicate content detection, and multi-source, multi-format fusion strategies, it quickly selects high-quality, diverse learning resources, improving the efficiency of learning resource utilization.

[0122] To achieve the above embodiments, such as Figure 9 As shown, this embodiment also provides an automatic construction device 10 for engineering problem-solving roadmaps based on large language models, including:

[0123] The automatic roadmap generation module 100 is used to input a description of the complex engineering problem to be solved and to perform basic format validation on the input content. Based on the engineering problem input by the user, it automatically calls the automatic roadmap construction algorithm to generate a corresponding knowledge roadmap for solving the problem and displays it in a tree structure. The knowledge roadmap contains multiple levels of nodes, each node represents a specific knowledge point or task, and there are clear logical relationships between the nodes.

[0124] The learning resource recommendation module 200 is used to automatically call the node resource mounting algorithm for each leaf node in the roadmap, retrieve and filter high-quality learning resources related to the node task from the Internet, and mount the learning resources to the corresponding node.

[0125] The personalized editing module 300 is used to flexibly edit the nodes in the generated roadmap according to one's own learning progress and existing knowledge.

[0126] User interaction module 400 is used to provide a user interaction interface and provide feedback on user information.

[0127] Specifically, the automatic roadmap construction application system is an intelligent application system that integrates the automatic roadmap construction algorithm and the automatic node resource mounting algorithm of this invention. This application system provides learners with functions such as automated roadmap construction and personalized secondary editing of roadmaps, thus maximizing the value of this invention.

[0128] This system uses Vue as the front-end framework and Spring Boot as the back-end framework, and also utilizes technologies such as MySQL database, MinIO object storage tool, and Redis memory caching for development and deployment.

[0129] The roadmap automatic construction application system needs to have the following functional requirements:

[0130] The system includes an automatic roadmap generation module (module 100). This module should provide a clear and concise input interface, allowing users to input a description of the complex engineering problem to be solved. Text input is supported, and basic format validation is performed to ensure the validity of the input. Based on the user-inputted engineering problem, the system should automatically invoke an automatic roadmap construction algorithm to quickly generate a corresponding knowledge roadmap for solving the problem. The generated roadmap should be displayed in a clear tree structure, containing multiple levels of nodes. Each node represents a specific knowledge point or task, and the nodes should have clear logical relationships, such as the detailed relationship between parent and child nodes, and the parallel relationship between sibling nodes, intuitively presenting the steps and knowledge framework for solving complex engineering problems. During the roadmap generation process, the system should be able to display the generation progress in real time, such as through a progress bar, to allow users to understand the current status and avoid anxiety caused by waiting.

[0131] The learning resource recommendation module has 200 modules. For each leaf node in the roadmap, the system automatically invokes a node resource mounting algorithm to retrieve and filter high-quality learning resources related to the task at that node from the internet and mount them onto the corresponding node. The recommended resources should cover various types, such as video tutorials, technical blogs, and documentation, to meet the learning preferences of different users. Users can further filter the recommended learning resources according to their needs, such as by resource type, to quickly locate the resources that best suit their requirements and improve the utilization rate of learning resources.

[0132] The personalized editing module 300 allows users to flexibly edit nodes in the generated roadmap based on their learning progress and existing knowledge. This includes adding new nodes, deleting unnecessary nodes, and modifying node names and descriptions to meet the personalized adjustment needs of different users' learning paths. Users can also adjust the order of nodes according to their learning habits and comprehension levels, such as changing the order of parent-child and sibling nodes through dragging and dropping, thereby optimizing the learning path to better suit their individual learning logic. Users can also edit the learning resources attached to nodes, such as adding new high-quality resource links, deleting inappropriate resources, and modifying resource descriptions to enrich and optimize the learning resource library and better support learning.

[0133] User Interaction Module 400. The system should provide a simple, intuitive, and easy-to-use user interface, adopting a modern design style and rationally arranging various functional modules to enable users to quickly get started and become familiar with the operation process. The interface should have good visual effects and interactive experience, such as reasonable color matching, clear font display, and smooth animation effects, to enhance user comfort and satisfaction. During user operations, the system should provide timely and accurate interactive prompts, such as clear error messages when users enter incorrect information, corresponding success feedback when operations are successful, and confirmation prompts when operations may produce irreversible results, to help users better understand and use system functions and avoid misoperations.

[0134] The automatic roadmap construction device for engineering problem solving based on a large language model according to embodiments of the present invention can automatically construct a well-structured and logically coherent roadmap for complex engineering problems. It enhances the initial roadmap by combining knowledge agents with an external knowledge base, supplementing missing key knowledge points. A multi-index evaluator is employed to rigorously evaluate the roadmap from four dimensions: logical structure, task decomposition granularity, topic relevance, and content completeness, generating a high-quality roadmap. The system allows for personalized editing of the generated roadmaps to meet the individualized learning path needs of different users. Suitable online learning resources are added to each leaf node to ensure high quality and diversity, satisfying different user learning preferences and improving learning effectiveness and satisfaction. The system generates roadmaps that meet requirements, enabling intelligent dynamic adjustment and optimization. It can dynamically adjust learning paths and resource recommendations based on real-time user feedback and progress, enhancing the system's flexibility and adaptability and solving the problem of lacking real-time feedback and adjustment mechanisms in existing technologies. Automated roadmap construction and learning resource recommendations reduce learners' time and effort in finding learning paths and resources, lowering learning costs. Through quality scoring filtering, duplicate content detection, and multi-source, multi-format fusion strategies, it quickly selects high-quality, diverse learning resources, improving the efficiency of learning resource utilization.

[0135] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0136] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A large language model-based engineering problem solution roadmap automatic construction method, characterized in that, include: Construct a skills point repository dataset and an expert validation roadmap dataset respectively; An automatic roadmap construction model is built based on a skills point repository dataset and an expert-validated roadmap dataset; wherein, the automatic roadmap construction model consists of multiple professional agents based on a large language model and a multi-metric evaluator; The system automatically builds a model by inputting a pre-defined complex engineering problem into the roadmap. An initialization agent generates an initial roadmap, and a knowledge agent enhances the initial roadmap by combining internal and external knowledge. A critique-improvement-evaluation cycle is then performed. In each cycle, the logical critique agent and the granular critique agent critique the logical structure of the roadmap and the granularity of the task nodes, respectively. The logical critique agent checks the logical relationships between nodes in the roadmap based on pre-defined parent-child and sibling logic to determine whether the nodes satisfy the refinement relationship or execution step relationship between parent and child nodes, and whether the child nodes under the same parent node satisfy the parallel progressive relationship. The granular critique agent evaluates the granularity of each task node in the roadmap based on pre-defined overly fine and overly coarse granularity standards to identify whether a node has been decomposed into too many trivial subtasks or whether the node content is too rich and needs further decomposition. The revision agent improves the roadmap based on the critique results. Finally, a multi-index evaluator evaluates the improved roadmap. If it reaches a pre-defined pass score, a structured roadmap for solving the complex engineering problem is output. After generating a structured roadmap for solving complex engineering problems, the leaf nodes in the structured roadmap are input into the node resource mounting model, and network resource retrieval is performed based on the tasks of the leaf nodes. All retrieved resources are comprehensively scored based on the large language model and index calculation, and resources with scores higher than a preset threshold are retained. All resources in the structured roadmap are checked through duplicate content detection, usability verification, and multi-source multi-form fusion strategy mechanisms.

2. The method of claim 1, wherein, Construct a skills point repository dataset, including: Use the MinerU tool to convert the PDF file of the paper into plain text Markdown format; Key content of the paper was extracted based on Markdown format, and initial skill points were generated using a large language model. The automatically extracted skill points are corrected and improved through manual review and verification, and a vector retrieval database is built using vector models and the Chroma vector database to construct a skill point repository dataset.

3. The method of claim 1, wherein, Construct an expert-validated roadmap dataset, including: Extract a core engineering problem for each paper; We use a large language model to generate an initial roadmap for each engineering problem, and use Markdown's multi-level headings to represent the structure and hierarchical relationship of the roadmap. The roadmap was adjusted based on expert experience to build an expert-validated roadmap dataset.

4. The method of claim 3, wherein, The multi-level headings used in Markdown to represent the structure and hierarchy of the roadmap include: Treat each skill point or skill module as a node; Use Markdown's first, second, or third level headings to represent nodes at different levels. Each node, in addition to its name, carries a level label to uniquely identify its position in the roadmap.

5. The method of claim 4, wherein, The multi-metric evaluator is implemented based on LLM and validates the roadmap based on a scoring system containing four evaluation metrics, including evaluating the logical coherence of the roadmap, the rationality of the task decomposition granularity, the relevance of the roadmap to the input engineering problem, and the richness and completeness of the roadmap content.

6. The method of claim 5, wherein, The method further includes: Three state-of-the-art LLM models were selected as multi-metric evaluators. Each model was used to score the revised roadmap independently, and the average score of the three models was used as the final score of the roadmap. Passing scores for four evaluation metrics were set based on experimental results conducted on an expert-validated roadmap dataset. The roadmap that scores above the preset passing score for all four evaluation metrics is output through the multi-metric evaluator; otherwise, the roadmap is returned and re-entered into the critique-improvement-evaluation loop until it passes the evaluation or reaches the maximum number of loops.

7. The method according to claim 1, characterized in that, All resources in the structured roadmap are examined through duplicate content detection, usability checks, and a multi-source, multi-format fusion strategy, including: Availability verification: The Requests library is used to send a HEAD request to each resource URL. If the returned status code is not 200, it indicates that the link to the current resource is invalid and it is removed directly; and a browser simulation engine based on the Selenium library is used to automatically detect the actual rendering effect of the page. Duplicate content detection: The similarity between any two resources is calculated based on the SimHash algorithm, and two resources with a distance of less than a preset threshold are considered duplicate content; Multi-source and multi-format fusion strategy: In terms of resource sources, each task node is required to contain high-quality resources from at least two different platforms; in terms of resource types, each node is required to contain both video and text resources; when the retained network resources do not meet the multi-source and multi-format fusion strategy, the quality scoring threshold is allowed to be reduced by a preset threshold step to re-select resources until the strategy requirements are met.

8. The method of claim 7, wherein, The similarity between any two resources is calculated using the SimHash algorithm, and resources with a distance less than a preset threshold are considered duplicate content, including: The name and description of each network resource are concatenated to form the text to be processed, and n feature keywords are obtained through text cleaning and word segmentation using the Jieba open-source library. The signature is obtained by aligning and weighting each feature keyword according to its frequency of occurrence. Then, a vector weighting is performed on each feature keyword to obtain a signature. That is, for each bit of the signature, if it is 1, the hash is multiplied positively by the weight, and if it is 0, the hash is multiplied negatively by the weight to obtain the vector of each feature word. The SimHash fingerprint signature of the text is obtained by summing the vectors of all feature words and reducing their dimensionality. The similarity is judged by calculating the Hamming distance between the fingerprint signatures of any two network resources. If the distance is lower than a preset threshold, the content is considered to be duplicated. The resource with the highest quality score will be automatically retained, and the remaining duplicate resources will be removed.

9. The system according to any one of claims 1-8, wherein the system is a large language model-based engineering problem solution roadmap automatic construction system. include: The automatic roadmap generation module is used to input a description of the complex engineering problem to be solved and to perform basic format validation on the input. Based on the engineering problem input by the user, it automatically calls the automatic roadmap construction algorithm to generate a corresponding knowledge roadmap for solving the problem and displays it in a tree structure. The knowledge roadmap contains multiple levels of nodes, each node represents a specific knowledge point or task, and there are clear logical relationships between the nodes. The learning resource recommendation module is used to automatically call the node resource mounting algorithm for each leaf node in the roadmap, retrieve and filter high-quality learning resources related to the node task from the Internet, and mount the learning resources to the corresponding node. The personalized editing module allows users to flexibly edit the nodes in the generated roadmap according to their own learning progress and existing knowledge. The user interaction module is used to provide a user interface and provide feedback on user information.

Images