Data perception method and system of embedded large language model cue word optimizer

Abstract

The invention discloses a data perception method and system of an embedded big language model cue word optimizer, and belongs to the technical field of data acquisition in big language model application. Obtaining a result according to the initial intention through a first neural network crawler Agent embedded with a cue word optimizer, and inputting the result into the optimizer; calculating each score based on a multi-dimensional evaluation system, judging according to a preset threshold value, and iteratively optimizing the cue word through a self-adaptive strategy selection algorithm until the cue word meets the standard; splicing the optimized corresponding dynamic URL list with a pre-stored static list to generate a to-be-processed set; and traversing and processing the set through a second neural network crawler Agent of a non-embedded optimizer, obtaining target webpage content, storing the target webpage content in a standardized format, and updating an optimization strategy selection result to the rule base. According to the method, automatic evaluation and optimization of the cue word are realized, and the quality, integrity and efficiency of data acquisition are improved.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology in large language model applications, and in particular to a data perception method, system, device and storage medium for embedding a large language model prompt word optimizer. Background Technology

[0002] With the rise of large language modeling (LAM) technology, especially in vertical applications, its integration with RAG technology can significantly improve the intelligence level and decision-making efficiency of specific industry tasks. Therefore, the need for rapidly building efficient domain knowledge bases using RAG is urgent. Besides existing local static data, a large amount of real-time data in cyberspace is of great value; however, the efficiency of rapidly and intelligently acquiring this data remains an issue. Traditional data acquisition methods include various approaches, such as relying on predefined rules and patterns for identification and extraction, and simulating browser environments, but these are not efficient or flexible enough to quickly meet the demands of complex cyberspace environments and artificial intelligence knowledge bases.

[0003] Furthermore, in the application of large language models, prompt words directly affect the model's understanding and output quality. Precise, clear, and specific prompt words can guide the model to more accurately capture user intent, thereby generating more relevant and high-quality content. Therefore, optimizing prompt words is essential. On the one hand, it can improve the system's response speed and efficiency, reduce misunderstandings and repetitive work, and enhance user experience and satisfaction. On the other hand, it helps reduce model bias and errors, ensuring the reliability and consistency of generated content. However, how to effectively optimize prompt words to improve system performance remains a challenge.

[0004] Currently, although large language models possess powerful semantic understanding capabilities, directly applying them to data perception in open network environments still faces two major challenges: First, the construction of prompt words highly relies on expert experience and its effectiveness is unstable, leading to a deviation between the model's retrieval intent and the user's actual needs; second, network data is extremely noisy, and the raw results generated by large models (such as URL lists) lack quantitative evaluation and closed-loop optimization mechanisms, directly using them as data sources will affect the quality of subsequent RAG knowledge bases. Therefore, how to construct an intelligent data perception framework that is deeply adapted to the capabilities of large models and possesses self-optimization capabilities has become an urgent technical challenge to be solved in the field.

[0005] Traditional data acquisition methods (such as rule-based pattern matching and static web crawling) are essentially at the grammatical level, lacking the ability to understand the semantics of users' deep intentions. They cannot generate high-quality prompts and structured data sources that meet the complex reasoning needs of large language models, making it difficult to directly serve intelligent applications based on large language models. Therefore, it is crucial to propose an efficient data acquisition method that facilitates the establishment of artificial intelligence knowledge bases. Summary of the Invention

[0006] The present invention aims to at least partially solve one of the technical problems in the related art.

[0007] To address this, this invention proposes a data-aware method that embeds a large language model prompt word optimizer. A neural network crawler agent with the embedded optimizer acquires results based on initial intent and performs multi-dimensional evaluation. Then, it iteratively optimizes the prompt words using an adaptive strategy until they meet the standards. Subsequently, the optimized dynamic links and static lists are concatenated into a set to be processed, which is then traversed and processed by another crawler agent to acquire, standardize, and store the target content. Simultaneously, the optimization strategy is updated to the rule base. This achieves automated optimization of the prompt word and data acquisition process, improving the accuracy and completeness of data acquisition.

[0008] Another objective of this invention is to provide a data-aware device that embeds a large language model prompt word optimizer.

[0009] The third objective of this invention is to provide a computer device.

[0010] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0011] To achieve the above objectives, this invention proposes a data-aware method for embedding a large language model cue word optimizer, comprising: S1, based on the data, the initial prompt word is input to obtain the intent. The initial result is obtained through a first neural network crawler Agent based on a large language model and embedded with a prompt word optimizer, and the initial result is input into the prompt word optimizer. S2, based on the multi-dimensional evaluation system of the prompt word optimizer, calculates the relevance score, accuracy score, completeness score and efficiency score, and determines whether optimization is needed according to the preset threshold. If so, the prompt words are iteratively optimized by an adaptive strategy selection algorithm until the evaluation result meets the preset standard. S3, concatenate the dynamic URL list corresponding to the optimized prompt words with the static URL list pre-stored in the network space to generate a set of URLs to be processed; S4 uses a large language model without an embedded prompt word optimizer to traverse the set of URLs to be processed, obtain the target web page content and store it in a standardized format, and update the rule base with the strategy selection results during the optimization process.

[0012] The data-aware method for embedding a large language model prompt word optimizer according to an embodiment of the present invention may also have the following additional technical features: In one embodiment of the present invention, the step of obtaining an initial prompt word based on the data acquisition intent, obtaining an initial result through a first neural network crawler Agent based on a large language model and embedded with a prompt word optimizer, and inputting the initial result into the prompt word optimizer includes: S11, the initial prompt words include specific domain-specific words and time-range-specific words; S12, when making an API call, set "Authorization": "Bearer" in the request header. <token>The "X-Respond-With": "no-content" parameter controls the response format.

[0013] In one embodiment of the present invention, the multi-dimensional evaluation system based on the prompt word optimizer calculates a relevance score, an accuracy score, a completeness score, and an efficiency score, determines whether optimization is needed based on a preset threshold, and if so, iteratively optimizes the prompt words using an adaptive strategy selection algorithm until the evaluation result meets the preset standard, including: S21, the formula for calculating the relevance score RS is: RS = 0.4*KWM + 0.3*ITM + 0.3*CTX; KWM is the ratio of the number of keyword occurrences in the response to the total number of expected keywords, ITM is the intent matching score based on a five-level scale, and CTX is the ratio of 1 minus the number of context deviations to the total number of paragraphs in the response; the formula for calculating the accuracy score AS is: AS = 0.5 * FC + 0.3 * CC + 0.2 * LC; FC is the factual accuracy rate, CC is the calculation accuracy rate, and LC is the logical consistency rate; the formula for calculating the completeness score CS is: CS = 0.4 * RR + 0.4 * CR + 0.2 * DR; RR is the demand response rate, CR is the coverage rate, and DR is the depth ratio; the formula for calculating the efficiency score ES is: ES = 0.5 * TE + 0.5 * LE; TE is the time efficiency, and LE is the length efficiency. S22, the adaptive strategy selection algorithm calculates the scores of four optimization strategies, including instruction clarity strategy score IC_SCORE, context supplementation strategy score CS_SCORE, constraint reinforcement strategy score CE_SCORE, and structure reorganization strategy score SR_SCORE, and selects the corresponding optimization strategy to optimize the prompt words based on the score of each strategy and the preset conditions.

[0014] In one embodiment of the present invention, the second neural network crawler agent, which uses a non-embedded prompt word optimizer, traverses the set of URLs to be processed, obtains the target webpage content and stores it in a standardized format, and updates the rule base with the strategy selection results during the optimization process, including: S41, the standardized format is Markdown; the Markdown format includes heading, body, and metadata fields; S42, the unique identifier is generated by parsing keywords in the URL and combining them with underscores.

[0015] In one embodiment of the present invention, it further includes: S5 performs time-series monitoring of specific target links in the static URL list, dynamically replaces the keyword part in the URL according to the preset update rules, and generates an updated static URL list. S6 concatenates the updated static URL list with the dynamic URL list to form a mixed URL set containing historical and latest data.

[0016] To achieve the above objectives, another aspect of the present invention proposes a data-aware device that embeds a large language model prompt word optimizer, comprising: The intent parsing module is used to obtain the initial prompt words for the intent input based on the data. It obtains the initial results through a first neural network crawler Agent based on a large language model and embedded with a prompt word optimizer, and inputs the initial results into the prompt word optimizer. The evaluation calculation module is used to calculate the relevance score, accuracy score, completeness score and efficiency score based on the multi-dimensional evaluation system of the prompt word optimizer. It determines whether optimization is needed based on the preset threshold. If so, it uses an adaptive strategy to select an algorithm to iteratively optimize the prompt words until the evaluation result meets the preset standard. The data source aggregation module is used to concatenate the dynamic URL list corresponding to the optimized prompt words with the static URL list pre-stored in the network space to generate a set of URLs to be processed. The large language model acquisition and storage module is used to traverse the set of URLs to be processed through a second neural network crawler agent without embedded prompt word optimizer, obtain the content of the target web pages and store it in a standardized format, and update the rule base with the strategy selection results during the optimization process.

[0017] In one embodiment of the present invention, it further includes: The time-series monitoring module is used to monitor specific target links in the static URL list in a time-series manner, dynamically replace the keyword part in the URL according to the preset update rules, and generate an updated static URL list. The secondary concatenation module is used to concatenate the updated static URL list with the dynamic URL list to form a mixed URL set containing historical and latest data.

[0018] This invention discloses a data-aware method and apparatus for embedding a large language model prompt word optimizer. By introducing a dynamic prompt word optimization mechanism that integrates multi-dimensional evaluation and adaptive strategies, it effectively solves the core problems in existing technologies where prompt word construction relies on human experience and data acquisition quality and efficiency are difficult to guarantee. It achieves closed-loop automation of the entire process from intent parsing, automatic evaluation, strategy optimization to data collection and storage, significantly improving the relevance, accuracy, and completeness of the acquired data. Through continuous iteration of optimization strategies and self-updating of the rule base, the system's adaptability, robustness, and long-term maintenance efficiency in dealing with diverse online data sources are enhanced. This invention provides a high-quality, high-timeliness automated data supply and prompt word optimization framework for RAG applications of large language models.

[0019] To achieve the above objectives, a third aspect of this application provides a computer device comprising a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing a data-aware method for embedding a large language model prompt word optimizer as described in the first aspect embodiment.

[0020] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements a data-aware method for embedding a large language model prompt word optimizer as described in the first aspect embodiment.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a data-aware method for embedding a large language model prompt word optimizer according to an embodiment of the present invention; Figure 2 This is a data-aware system architecture diagram of an embedded large language model prompt word optimizer according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the working mechanism of a dual-channel embedded prompt word optimizer, which is a data-aware method for embedding a large language model prompt word optimizer according to an embodiment of the present invention. Figure 4 This is a logical diagram of the computation method of the prompt word optimizer in a data-aware method for embedding a large language model prompt word optimizer according to an embodiment of the present invention; Figure 5 This is an index network structure diagram of a data perception method for embedding a large language model prompt word optimizer according to an embodiment of the present invention; Figure 6 This is a flowchart illustrating another data-aware method for embedding a large language model prompt word optimizer according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a data perception device for embedding a large language model prompt word optimizer according to an embodiment of the present invention; Figure 8 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] The following description, with reference to the accompanying drawings, describes a data-aware method, system, device, and storage medium for embedding a large language model prompt word optimizer according to an embodiment of the present invention.

[0026] The core idea of ​​this invention is to achieve intelligent and precise acquisition of cyberspace information by constructing a closed-loop data perception system with an embedded prompt word optimizer. The system first inputs initial prompt words based on the data acquisition intent and uses an agent with the embedded optimizer to perform preliminary retrieval and result feedback. Then, it measures the feedback results through a multi-dimensional quantitative evaluation system covering relevance, accuracy, completeness, and efficiency, and triggers an adaptive strategy selection algorithm based on preset thresholds to iteratively optimize the prompt words until they meet the standards. The optimized prompt words drive the generation of a dynamic target link list, which is then merged with a pre-stored static resource list to form a unified set of data sources to be processed. Finally, an agent without the embedded optimizer traverses and collects data from this set, storing the acquired content in a standardized format and simultaneously depositing the optimization strategy into a rule base, thus completing a closed-loop process from intent parsing, dynamic optimization, multi-source integration to automated collection and knowledge accumulation. This method effectively overcomes the limitations of traditional data acquisition relying on fixed rules and human experience, significantly improving the targeting, quality, and self-evolutionary capability of information capture.

[0027] Example 1 To achieve the above invention, embodiments of the present invention provide a data-aware method for embedding a large language model prompt word optimizer, such as... Figure 1 As shown, it includes: S1: Based on the data, the initial prompt word is obtained by acquiring the intent. The initial result is obtained by a first neural network crawler Agent based on a large language model and embedded with a prompt word optimizer, and the initial result is input into the prompt word optimizer.

[0028] Specifically, this step begins with the user or system module generating initial prompts based on the target data topic (such as energy transition and climate change). For example, the prompt phrase could be: "Query the latest information on energy transition and climate change." This prompt phrase is then fed into a neural network crawler agent embedded with a prompt phrase optimizer. This agent is typically built on a Large Language Model (LLM) and possesses semantic understanding and information retrieval capabilities. The agent identifies keywords, intent, and contextual information in the prompt phrase through a semantic parsing module, and actively retrieves a list of relevant URLs from the cyberspace using pre-defined search strategies (such as semantic similarity matching and time-sensitive ranking). The initial results returned include metadata such as title, URL source, and summary description, which are used for subsequent prompt phrase optimization and evaluation.

[0029] Furthermore, the prompt optimizer quantifies the initial results across multiple dimensions. First, it filters out obviously irrelevant results using Relevance (RS). Then, it assesses the breadth of information collection using Completeness (CS) and comprehensively weighs Efficiency (ES) and Credibility metrics. If the overall score (especially RS) falls below a threshold (e.g., RS < 0.8), the prompt optimization process is triggered to generate more precise search instructions for the next round of data acquisition, thus ensuring the quality and relevance of input information at the source of data acquisition. For example, Keyword Match (KWM) is calculated as the ratio of the number of keyword occurrences in the response to the total number of expected keywords; Intent Match (ITM) uses a five-point scale; and Contextual Coherence (CTX) is calculated as the ratio of the number of contextual deviations to the total number of paragraphs. These metrics collectively constitute the Relevance Score (RS): .like If the value is less than 0.8, the prompt word optimization process will be triggered.

[0030] Specifically, this step is applicable to scenarios that require proactively acquiring domain-specific information from cyberspace, such as building RAG knowledge bases, industry data analysis, and policy tracking. By using a neural network crawler agent embedded with a prompt word optimizer, the system can quickly respond to user intent and obtain a list of URLs highly relevant to the target, providing a high-quality input source for subsequent deep data processing.

[0031] Specifically, by guiding the crawler agent with intelligent prompts, the targeting and efficiency of data acquisition are significantly improved. Compared to traditional keyword- or rule-based crawling methods, this invention, through semantic understanding and dynamic optimization mechanisms, can more accurately match user intent, reduce the acquisition of invalid data, and improve the overall system's response speed and data quality.

[0032] Furthermore, S1 includes: S11, the initial prompt words include specific domain qualifiers and time range qualifiers.

[0033] Specifically, prompt words serve as input to a neural network crawler agent that incorporates a prompt word optimizer, driving it to proactively search for dynamic information sources in the internet space that highly match the user's intent. In some implementations, prompt words need to meet certain structured requirements, such as containing explicit keyword combinations, time-limited words, and target content types (e.g., policy, technology, economic impact), to ensure that the crawler agent can accurately locate target data.

[0034] Furthermore, the neural network crawler agent uses a natural language processing (NLP) model to semantically analyze the prompt words and extract a set of keywords. and time range The crawler agent performs matching and retrieval on the target webpage based on semantic similarity algorithms (such as cosine similarity or BERT embedding matching). The crawler agent returns an initial list of results containing the title, URL, summary description, and timestamp, for example...<url_1> ,<url_2> , ...,<url_n> Each URL corresponds to a web page resource related to the intent of the prompt word.

[0035] Furthermore, Keyword Matching (KWM), Intent Matching (ITM), and Contextual Coherence (CTX) together form the core computational foundation of the Relevance Score (RS), directly quantifying the strength of the association between the response content and the query intent in terms of keyword coverage, semantic fit, and logical coherence. Simultaneously, these three indicators provide crucial support for the evaluation of other dimensions: by ensuring accurate mention of core terms, semantic consistency with intent, and logical self-consistency of content, they lay a reliable contextual foundation for fact-checking and logical consistency judgment in the Accuracy Score (AS); their measurement of keyword coverage and intent responsiveness indirectly reflects the breadth of topics and depth of demand response focused on in the Completeness Score (CS); in addition, through rapid and automated preliminary screening of results, they effectively filter out low-quality content, avoiding the waste of subsequent in-depth analysis resources, thereby significantly improving the efficiency of the overall evaluation process and indirectly optimizing the Efficiency Score (ES). Therefore, KWM, ITM, and CTX do not act in isolation, but rather form an organic synergy and linkage among the four dimensions of relevance, accuracy, completeness, and efficiency, jointly ensuring the systematic nature, reliability, and overall effectiveness of the evaluation system.

[0036] Specifically, this step is suitable for scenarios requiring rapid access to real-time information in specific fields (such as energy transition and climate change), such as policy research, market analysis, or academic literature retrieval. By limiting the time frame (e.g., the latest developments by 2025), the system can prioritize capturing web content that is timely and frequently updated, thereby enhancing the data's cutting-edge nature and reference value.

[0037] Specifically, this step significantly improves the accuracy and efficiency of data acquisition through structured prompt input and a semantically driven crawling mechanism. Compared to traditional keyword-based search engines, this invention, by embedding the semantic understanding capabilities of a prompt optimizer, can more accurately identify user intent, reduce interference from noisy data, and provide a high-quality initial information source for subsequent prompt optimization and data processing.

[0038] S12, when making an API call, set "Authorization" in the request header: "Bearer <token>The "X-Respond-With": "no-content" parameter controls the response format.

[0039] Specifically, this step is widely used in first neural network crawler agents that embed prompt word optimizers to proactively acquire dynamic network information links that match user intent.

[0040] Specifically, the system communicates with the remote API service via HTTP requests, where "Authorization": "Bearer <token>"Used for authentication, ensuring the request has legitimate access rights. This token is typically issued by the API service provider via the OAuth 2.0 protocol and has an expiration date and access scope restrictions." "X-Respond-With": "no-content" is a custom request header parameter used to control the API response format, instructing the server to return only status codes and metadata, without returning the actual content body (such as HTML or JSON data). This mechanism is particularly important in large-scale data collection scenarios, significantly reducing network transmission overhead and improving response speed.

[0041] Furthermore, Authorization: Employs the Bearer Token mechanism. The token length is typically a 32-64 bit Base64 encoded string, with a validity period of 1-24 hours, depending on the API service configuration. X-Respond-With: This parameter is a string with a value of "no-content". It controls the response format and is semantically equivalent to the HTTP status code 204 (No Content). However, in this system, it is used as a custom header field to achieve finer-grained control.

[0042] Specifically, this step is mainly applied to the first neural network crawler agent, which embeds a prompt word optimizer, when performing active data acquisition tasks. For example, it generates a dynamic URL list based on the optimized prompt words and verifies the availability of URLs in batches or obtains their metadata (such as title, description, update time, etc.) through API interfaces. In actual deployment, the system can initiate multiple API requests in parallel. By setting reasonable request intervals (such as 500ms~2s) and concurrency (such as 10~50 concurrent requests), it ensures that stable connections and response efficiency can be maintained even under high throughput.

[0043] Specifically, by setting the aforementioned request header parameters, the system can achieve precise control over the API response content, thereby improving the efficiency and accuracy of data acquisition. On the one hand, the "no-content" response mode reduces unnecessary data transmission, lowering bandwidth consumption and server load; on the other hand, combined with the Bearer Token mechanism, the system can ensure the security and legitimacy of data access, preventing unauthorized access or abuse by web crawlers. This step, as the foundation for the system's interaction with external data sources, provides high-quality, structured input for subsequent URL concatenation, data processing, and knowledge base construction, and is a crucial supporting link for realizing a dual-channel data-aware architecture.

[0044] S2 calculates relevance score, accuracy score, completeness score and efficiency score based on the multi-dimensional evaluation system of the prompt word optimizer. It determines whether optimization is needed based on the preset threshold. If so, it iteratively optimizes the prompt words through an adaptive strategy selection algorithm until the evaluation result meets the preset standard.

[0045] Specifically, this step is the core control logic of the entire data perception system, ensuring that the output of the prompt words meets preset standards in terms of semantic matching, information accuracy, content completeness, and response efficiency, thereby improving the quality and efficiency of subsequent data crawling and knowledge base construction. The evaluation system is specifically designed for the characteristics of content generated by large language models. For example, "intent matching" requires using another LLM (Language Model) as an evaluator to perform deep semantic similarity calculation based on the user's original intent, rather than simple keyword matching. "Contextual coherence" assesses common logical jumps or topic deviations in long text generation by large models. This evaluation system constitutes a quantitative feedback loop for the quality of large model generation, forming the basis for its continuous optimization.

[0046] Furthermore, the system first receives initial prompt word input and obtains preliminary response results through a first neural network crawler agent that embeds a prompt word optimizer. Subsequently, the system performs a multi-dimensional evaluation of the response content, with the specific calculation formula as follows: Relevance score: ; Accuracy rating: ; Integrity score: ; Efficiency rating: ; Wherein, KWM is the ratio of the number of keyword occurrences in the response to the expected total number of keywords; ITM is the five-point scale score {0.2, 0.4, 0.6, 0.8, 1.0}; CTX is 1 - (number of context deviations / total number of paragraphs in the response); FC is the factual accuracy rate, factual accuracy rate = (total number of statements - number of incorrect statements) / total number of statements; CC is the calculation accuracy rate, calculation accuracy rate = (number of correct calculation steps) / total number of calculation steps; LC is the logical consistency rate, logical consistency = 1 - (number of logical contradictions / total number of arguments); RR is the demand response rate, demand response rate = number of responded demand points / total number of demand points; CR is the coverage rate, coverage rate = breadth of topics covered by the response / expected breadth of topics {0.2, 0.4, 0.6, 0.8, 1.0}; DR is the depth ratio, depth ratio = actual depth level / expected depth level; TE is the time efficiency, time efficiency = target response time / actual response time; LE For length efficiency, the formula is: Length Efficiency = Target Token Count / Actual Token Count. The system determines whether to optimize the suggestion words based on preset thresholds (RS < 0.8, AS < 0.7, CS < 0.75, ES < 0.85).

[0047] Specifically, the "First Neural Network Crawler Agent" embedded in the prompt word optimizer is, at its core, a crawler agent based on neural network deep learning technology. It not only possesses basic text generation capabilities but is also endowed with specific functions such as task planning, web page information understanding, and API calls. After receiving natural language prompt words, this agent can perform the following operations: 1) semantic parsing and intent decomposition; 2) simulating human search logic to construct query strategies; 3) calling external search tools or directly parsing web information; 4) performing preliminary semantic filtering and structured output on the obtained links and summaries. Essentially, it is a large-scale intelligent model oriented towards data-aware tasks.

[0048] Furthermore, if the evaluation results do not meet the standards, the system will use an adaptive strategy selection algorithm to select the optimal optimization strategy based on the optimization strategy scores (such as IC_SCORE, CS_SCORE, CE_SCORE, SR_SCORE) for iterative optimization. Specifically, the Instruction Clarification strategy (IC_SCORE = 0.6 * (1 - ITM) + 0.4 * (1 - FC)) is applied when IC_SCORE > 0.2 and is the highest-scoring strategy; the Context Supplementation strategy (CS_SCORE = 0.5 * (1 - CTX) + 0.3 * (1 - FC) + 0.2 * (1 - RR)) is applied when CS_SCORE > 0.2 and is the highest-scoring strategy; the Constraint Enhancement strategy (CE_SCORE = 0.4 * (1 - LC) + 0.3 * (1 - CR) + 0.3 * (1 - LE)) is applied when CE_SCORE > 0.2 and is the highest-scoring strategy; and the Structure Reorganization strategy (SR_SCORE = 0.4 * (1 - DR) + 0.3 * (1 - RR) + 0.3 * (1 - TE) Application conditions: SR_SCORE>0.2 and is the highest score strategy. For example, if IC_SCORE is the highest, the instruction clarification strategy is adopted, which improves the executability and accuracy of prompt words by clarifying action verbs, decomposing complex instructions, and eliminating ambiguous words.

[0049] Specifically, this step is widely applicable in practical scenarios that require acquiring structured or unstructured data from cyberspace, such as industry report analysis, policy tracking, and market intelligence gathering. Through multi-dimensional evaluation and adaptive optimization, the system can significantly improve the accuracy and efficiency of data acquisition, reduce invalid crawling and redundant processing, thereby providing reliable data support for building a high-quality RAG knowledge base.

[0050] Furthermore, S2 includes: S21, the formula for calculating the relevance score RS is: RS = 0.4*KWM + 0.3*ITM + 0.3*CTX; KWM is the ratio of the number of keyword occurrences in the response to the total number of expected keywords, ITM is the intent matching score based on a five-point scale, and CTX is the ratio of 1 minus the number of context deviations to the total number of paragraphs in the response; the formula for calculating the accuracy score AS is: AS = 0.5 * FC + 0.3 * CC + 0.2 * LC; FC is the factual accuracy rate, CC is the calculation accuracy rate, and LC is the logical consistency rate; the formula for calculating the completeness score CS is: CS = 0.4 * RR + 0.4 * CR + 0.2 * DR; RR is the demand response rate, CR is the coverage rate, and DR is the depth ratio; the formula for calculating the efficiency score ES is: ES = 0.5 * TE + 0.5 * LE; TE is the time efficiency, and LE is the length efficiency.

[0051] Specifically, in the embodiments of the present invention, the correlation score The calculation formula is: .

[0052] Specifically, this formula quantifies the degree of matching between the model's response content and the user's intent, and is one of the core metrics in the prompt word optimizer evaluation module. At the technical implementation level, this scoring mechanism comprehensively evaluates the response through three key dimensions: Keyword Matching (KWM), Intent Matching (ITM), and Contextual Coherence (CTX), corresponding to the response content's performance in terms of keyword coverage, intent understanding, and contextual logical consistency, respectively.

[0053] Furthermore, keyword matching degree The calculation method is the ratio of the number of times the expected keyword appears in the response to the total number of expected keywords, that is: .

[0054] Specifically, this parameter reflects the degree to which the response content covers the user-specified keywords; the higher the value, the closer the response is to the topic of interest to the user. In practice, keywords are usually extracted by the user intent parsing module and input into the scoring system as evaluation criteria.

[0055] Specifically, intent matching degree A five-point scale scoring mechanism is used, with a value range of [value missing]. This score measures how semantically the response content matches the user's intent. It is evaluated by human annotation or a semantic similarity model (such as BERTScore) to ensure accurate intent understanding. Contextual coherence. The calculation formula is: .

[0056] Specifically, this parameter is used to evaluate the logical coherence of the response content in terms of paragraph structure. The number of deviation points is usually determined by the number of semantic breaks or logical jumps identified by the context analysis module. The total number of paragraphs in the response is the paragraph segmentation result of the model output content, which is usually based on paragraph segmentation algorithms in natural language processing (such as rule segmentation based on periods and line breaks).

[0057] Specifically, the scoring mechanism is embedded in the closed-loop feedback system of the prompt word optimizer to determine whether the current prompt word needs further optimization. At that time, the system will trigger an optimization process, and combine it with other evaluation indicators (such as accuracy scores). Integrity score Efficiency rating This formula enables the system to make comprehensive decisions. Through this formula, the system can achieve multi-dimensional and quantitative evaluation of the effectiveness of prompt words, thereby improving the accuracy of data acquisition and response quality, and providing a reliable data foundation for building a high-quality RAG knowledge base.

[0058] Specifically, the relevance score RS is calculated as follows: RS = 0.4 * KWM + 0.3 * ITM + 0.3 * CTX. This score measures the degree of relevance between the generated text and the user's original needs and context. Keyword matching (KWM) is defined as the ratio of the number of expected keywords appearing in the response to the total number of expected keywords, and its value directly reflects the coverage of core terms. Intent matching (ITM) is scored using a predefined five-level scale with values ​​in the discrete set {0.2, 0.4, 0.6, 0.8, 1.0}, which can be determined by manual annotation or a semantic similarity model based on deep learning. Contextual coherence (CTX) is calculated as follows: CTX = 1 - (number of contextual deviations / total number of paragraphs in the response), where the number of deviations is determined by the number of logical breaks or topic jumps identified by the natural language processing module, and is used to evaluate the logical fluency within the text. The accuracy score AS is calculated as follows: AS = 0.5 * FC + 0.3 * CC + 0.2 * LC. This scoring system aims to objectively assess the objectivity, correctness, and internal consistency of the generated content. The Fact Accuracy (FC) score is calculated as (Total Number of Statements - Number of Incorrect Statements) / Total Number of Statements, where the number of statements refers to verifiable objective assertions in the text. The Computational Accuracy (CC) score is defined as the ratio of correctly calculated steps to the total number of calculated steps, applicable to texts containing numerical reasoning or formula derivations. The Logical Consistency (LC) score is calculated as LC = 1 - (Number of Logical Contradictions / Total Number of Arguments), quantifying the text's internal logical rigor by detecting conflicts between arguments. The Completeness score (CS) is calculated as: CS = 0.4 * RR + 0.4 * CR + 0.2 * DR. This score measures the extent to which the response content satisfies the breadth and depth of user needs. The Response Rate (RR) score is calculated as the ratio of responded needs to the total number of needs. Coverage (CR) is calculated by comparing the breadth of topics covered in the response with the expected breadth of topics, and is quantified using a five-point scale {0.2, 0.4, 0.6, 0.8, 1.0}. Depth Ratio (DR) is defined as the ratio of the actual depth of the explanation to the expected depth of the explanation; the levels are pre-defined based on problem complexity and domain knowledge hierarchy. Efficiency score (ES) is calculated as: ES = 0.5 * TE + 0.5 * LE. This score evaluates the efficiency of the generation process from the perspective of resource consumption. Time efficiency (TE) is calculated as the ratio of the target response time to the actual response time. Length efficiency (LE) is calculated as the ratio of the target number of tokens to the actual number of tokens generated; both factors constrain the system's performance in terms of timeliness and content conciseness.

[0059] Furthermore, this evaluation system is implemented in an algorithmic and programmable manner. The system automatically extracts keywords, segments paragraphs, and identifies statements and arguments using a Natural Language Processing (NLP) module, and combines this with a rule engine or pre-trained model to determine complex semantics such as intent matching degree (ITM) and logical contradictions. The raw data for each sub-indicator are normalized and calculated according to a preset weighting formula, ultimately generating independent scores for four dimensions and an optional comprehensive score. The entire process can be automated and closed-loop, integrated into a model training pipeline or online service feedback loop.

[0060] Specifically, this system is primarily deployed in the quality monitoring and optimization phase of the AIGC platform. For example, in the intelligent customer service system, relevance scores (RS) and completeness scores (CS) are used to filter out irrelevant or incomplete responses in real time; in the automatic report generation scenario, accuracy scores (AS) are used to verify facts and calculation errors. The evaluation results can directly drive prompt word engineering optimization, model fine-tuning, data filtering, or trigger manual review, forming an iterative cycle of "generation-evaluation-optimization," significantly reducing the cost of manual quality inspection.

[0061] Specifically, metrics such as Keyword Matching (KWM) and Fact Accuracy (FC) are based on precise statistics and are highly objective; Intent Matching (ITM) and Coverage (CR) introduce discrete scales or relative ratios to accommodate necessary semantic and breadth judgments. Weighting (e.g., FC accounts for 0.5 in AS) reflects the emphasis on core quality dimensions (such as fact accuracy). All parameters are standardized to ensure that metrics of different dimensions can be weighted and aggregated, and thresholds (e.g., RS < 0.8 triggers optimization) can be calibrated according to specific scenarios.

[0062] Specifically, this evaluation system also has broad applicability, mainly including: intelligent question-answering and dialogue systems: evaluating the relevance, accuracy, and completeness of answers to improve user experience; automatic content generation and review: used for quality control and grading of press releases, marketing copy, and academic abstracts; educational evaluation and learning support: automatically evaluating the accuracy, logical completeness, and depth of argumentation of students' answers; professional field report analysis: in fields such as finance, healthcare, and law, verifying the factual accuracy, data calculation correctness, and logical consistency of generated reports; model development and benchmark testing: serving as a quantitative evaluation benchmark for comparing the effects of different models or different prompts.

[0063] S22, the adaptive strategy selection algorithm calculates the scores of four optimization strategies, including instruction clarity strategy score IC_SCORE, context supplementation strategy score CS_SCORE, constraint reinforcement strategy score CE_SCORE, and structure reorganization strategy score SR_SCORE, and selects the corresponding optimization strategy to optimize the prompt words based on the score of each strategy and the preset conditions.

[0064] Specifically, the core objective of the adaptive strategy selection algorithm is to automatically and accurately select the most suitable optimization strategy based on the quantitative diagnosis of the current prompt word output effect, thereby achieving targeted and efficient improvement of prompt word quality and forming an intelligent closed loop of evaluation-decision-optimization.

[0065] Furthermore, in terms of technical implementation, the algorithm constructs a decision-making model based on multi-index fusion. This model takes the sub-indicators (such as ITM, FC, CTX, RR, etc.) generated by the aforementioned multi-dimensional evaluation system as input, and solves in parallel the instruction clarity strategy score (IC_SCORE), context supplementation strategy score (CS_SCORE), constraint reinforcement strategy score (CE_SCORE), and structural reorganization strategy score (SR_SCORE) using a predefined weighted calculation formula. Each strategy score is designed as a composite signal reflecting a specific type of optimization requirement.

[0066] Furthermore, in terms of decision-making logic, the algorithm compares the calculated scores of the four strategies with a preset activation threshold (e.g., >0.2). Then, applying the rule of "highest score exceeding the threshold," it determines the single optimization strategy to be executed in this round from the candidate strategy set. For example, if the CS_SCORE value is the highest and exceeds the threshold, the system will automatically invoke the context supplementation strategy to add missing background information or examples to the prompt word.

[0067] Specifically, this algorithm is applicable to scenarios where large language models fail to meet standards in relevance, accuracy, or completeness when responding to complex, open-domain tasks due to shortcomings in the prompts' clarity, information sufficiency, constraint strictness, or logical structure. It enables the system to replace human experience and automatically complete "problem attribution" and "strategy matching."

[0068] Specifically, this algorithm significantly improves the scientific rigor, automation, and iteration speed of the prompt word optimization process by transforming the selection process from being driven by subjective experience to being driven by objective data. It ensures the targeted and effective nature of optimization interventions and strengthens the system's adaptability and overall performance stability when facing diverse tasks.

[0069] S3. The dynamic URL list corresponding to the optimized prompt words is concatenated with the static URL list pre-stored in the network space to generate a set of URLs to be processed.

[0070] Specifically, in step S3, the system concatenates the dynamic URL list corresponding to the optimized prompt words with the pre-stored static URL list in the network space to generate a set of URLs to be processed. This is a crucial step in achieving dual-channel data acquisition. The technical implementation of this step is based on the integration and unified management of data sources, aiming to improve the comprehensiveness and systematic nature of data collection.

[0071] Furthermore, the dynamic URL list is actively obtained by the first neural network crawler agent, which embeds a prompt word optimizer, based on optimized prompt words. Its content typically consists of real-time updated webpage links, such as news, policy announcements, and research reports. The static URL list, on the other hand, is a pre-configured fixed data source, such as official websites of authoritative institutions or database interfaces, and its content has high credibility and stability. In the implementation process, the system first reads the dynamic URL list file `crawler_urls_dynamic.json` and extracts its URL fields; then it reads the static URL list file `crawler_urls_static.json` and merges it with the dynamic list. In addition, the system also supports dynamic replacement or supplementation of the self-revising URL list `crawler_urls_self_revising.json`, which has an update cycle, to adapt to the periodic changes in the data source. The concatenated URL set is uniformly stored as `crawler_urls.json`, which is a JSON object containing an array field named `urls` to hold all the URLs to be processed.

[0072] Furthermore, the concatenation operation must adhere to certain data cleaning and deduplication mechanisms. For example, the system can set a URL length threshold (e.g., no more than 2048 characters) to ensure the legitimacy of links; simultaneously, a hash algorithm (e.g., SHA-256) can be used to verify the uniqueness of URLs, avoiding duplicate data collection. In addition, the system can set URL protocol types (e.g., HTTP / HTTPS) and domain whitelists to filter invalid or potentially risky links. The concatenated URL set must meet integrity requirements, meaning the coverage of dynamic and static data sources should reach a preset domain information coverage rate (e.g., CR ≥ 0.8) to ensure data diversity and representativeness.

[0073] Specifically, this step is widely applied in scenarios requiring multi-source data fusion, such as building RAG knowledge bases, conducting industry trend analysis, and supporting large language model training. In the field of energy transition and climate change, the system achieves synchronous collection of the latest global dynamics and historical data by splicing dynamically acquired policy update links with static authoritative data sources (such as the UN Framework Convention on Climate Change website and the International Energy Agency website), providing high-quality input for subsequent data processing and analysis.

[0074] Specifically, this step significantly improves the breadth and depth of data collection by integrating dynamic and static data sources, enhancing the system's adaptability and robustness. Simultaneously, the unified URL set format provides a standardized input interface for the second neural network crawler agent (which does not embed prompt word optimizers), ensuring the efficient execution of subsequent data acquisition processes.

[0075] S4 uses a large language model without an embedded prompt word optimizer to traverse the set of URLs to be processed, obtain the target web page content and store it in a standardized format, and update the rule base with the strategy selection results during the optimization process.

[0076] Specifically, the "Second Neural Network Crawler Agent with Non-Embedded Cue Optimizer" is a neural network crawler agent specializing in web page content extraction and storage. Its core mission is to transform the list of URLs to be processed into standardized knowledge corpora suitable for training and inference of large models.

[0077] Specifically, the technical essence and operating mechanism of this agent are as follows: A parsing and understanding engine based on a large language model: This agent is not a traditional rule-based or simple statistical crawler. It integrates or calls a specially tuned large language model, possessing powerful capabilities for understanding webpage structure and extracting semantic content. Faced with diverse HTML source code, it can identify and extract core content blocks (such as body text, title, author, and publication time) like a human, while filtering out noisy information such as navigation bars, advertisements, and scripts. Its core mechanism is built on deep learning, replacing traditional keyword matching with semantic understanding capabilities. It utilizes pre-trained language models (such as BERT and GPT) to understand query intent and the deep semantics of webpage content, achieving more accurate relevance judgment. Simultaneously, it employs vectorization technology to transform webpage content into points in a high-dimensional vector space, evaluating content similarity by calculating vector distance, thereby constructing a semantic association network to guide the crawling path. It also undertakes the responsibility of standardized and semantic storage: its output requires a standardized Markdown format, which aligns with the preference of large language models for processing structured text. The agent needs to reorganize unstructured webpage content into well-structured documents containing clear titles, body paragraphs, and metadata fields; the generation of unique identifiers also reflects semantic thinking. By parsing keywords in the URL path (such as renewable-energy-2025), it doesn't simply perform string splitting. Instead, it leverages the lightweight semantic understanding capabilities of large models to extract the most representative and discriminative keyword combinations, laying the foundation for subsequent vectorized retrieval and knowledge management. As the terminal of the "data pipeline" in the large model's knowledge base, this agent is the "executive arm" of the entire system, and its work quality directly determines the quality of data input to downstream large models (such as the RAG system). It ensures that the acquired content is clean, relevant, and structured, greatly reducing the burden on subsequent large models in information digestion and knowledge fusion. The Markdown files it stores are themselves high-quality corpora tailored for large language models, facilitating direct use for fine-tuning training, contextual learning, or few-shot prompts. It complements the "Agent with embedded prompt word optimizers," together forming a dual "perception-execution" channel for large models. The former (embedded agent) is the exploration and optimization channel, utilizing the reasoning capabilities of the large model to optimize "search instructions" (prompt words); the latter (non-embedded agent) is the efficient execution and standardization channel, utilizing the parsing capabilities of the large model to perform "data harvesting". This division of labor reflects the specialized allocation of the capabilities of the large model in complex tasks.

[0078] Specifically, the core of this step lies in achieving efficient extraction and structured management of multi-source heterogeneous data in cyberspace through an automated data acquisition mechanism combined with standardized storage strategies. This agent is based on a predefined API interface (such as https: / / r.jina.ai / ).<url_link> The system accesses URLs one by one, retrieves the content of the target webpage through HTTP requests, and stores the response results in Markdown format, thereby providing a high-quality, structured text data source for the subsequent construction of the knowledge base of the large language model.

[0079] Furthermore, the Agent first reads the crawler_urls.json file generated in step S3 and extracts the list of URLs. Then, it performs asynchronous request processing for each URL, ensuring good concurrency performance and response efficiency in large-scale data acquisition scenarios. During the request process, the Agent carries authentication information (such as Authorization: Bearer) by setting request headers. <token>This ensures legitimate access to the target resource. After parsing the response content, key metadata (such as title, description, date, etc.) is extracted and combined with the URL parsing results to generate a unique identifier used for file naming and storage path management. This identifier is typically composed of keywords from the URL, such as... <keyword1> _ <keyword2> _ <timestamp>This is to ensure the uniqueness and searchability of file names.

[0080] Specifically, the storage path must follow a unified directory structure specification, such as .. / crawler / output / <file_name> The Agent needs to generate a .md file to support subsequent data retrieval and management. In addition, the Agent also needs to record metadata about the processing results, including lists of successful and failed URLs, stored in crawler_results.json respectively, to facilitate system exception handling and log analysis.

[0081] Specifically, this step is widely applicable to data collection tasks in vertical sectors such as energy transition and climate change. Through an agent that uses a non-embedded prompt optimizer, the system can extract structured text content such as real-time policy documents, research reports, and news announcements from dynamically generated URL lists, providing continuously updated data support for building the RAG knowledge base. Especially when dealing with multilingual, multi-format, and multi-source web data, this agent effectively improves data compatibility and usability through standardized processing procedures.

[0082] Specifically, this step automates and intelligently acquires and stores network data, significantly improving the efficiency and consistency of data processing. By concatenating the optimized prompt-driven URL list with static target URLs, the system can cover a wider range of data sources, ensuring the comprehensiveness and timeliness of the knowledge base content. Simultaneously, the standardized storage format (Markdown) facilitates subsequent model processing, providing high-quality input data for the training and inference of large language models.

[0083] Furthermore, S4 includes: S41, the standardized format is markdown; the markdown format includes heading, body and metadata fields.

[0084] Specifically, the technical implementation of this step is based on the integration and structuring of data sources, aiming to improve the comprehensiveness and accuracy of subsequent data acquisition.

[0085] Specifically, the system first reads a dynamic URL list generated by the embedded prompt word optimizer, formatted as `crawler_urls_dynamic.json`, which includes a "urls" field to store link information actively retrieved from cyberspace. Simultaneously, the system reads a static URL list, `crawler_urls_static.json`, which is typically maintained manually or formed from historical data accumulation and contains fixed target links. Additionally, the system can read `crawler_urls_self_revising.json`, a file used to store URLs with regular update patterns, such as periodically published policy documents or industry reports. Through JSON file reading and array concatenation operations, the system merges these three types of URLs into a unified `crawler_urls.json` file.

[0086] Furthermore, the concatenation operation must ensure the uniqueness and validity of the URLs. The system uses a hash verification mechanism (such as SHA-256) to deduplicate URLs, avoiding duplicate crawling. Simultaneously, the system sets a maximum concatenation length limit, for example, the urls array can contain a maximum of 1000 links, to control the data processing scale and improve system stability. In addition, the system supports a dynamic expansion mechanism; when the number of concatenated URLs is insufficient, steps S1 and S2 can be repeatedly executed to supplement the data source.

[0087] Specifically, this step is widely used in building RAG-based knowledge base systems, particularly in fields requiring real-time data updates, such as energy transition and climate change. By integrating dynamic and static data sources, the system can cover a wider range of information sources, improving the timeliness and authority of the data. For example, in tasks such as policy analysis and market trend forecasting, the concatenated URL list can be used as input for a non-embedded crawler agent to obtain structured data and generate Markdown-formatted knowledge documents.

[0088] Specifically, this step enables efficient integration and unified management of data sources, providing high-quality input for subsequent non-embedded crawler agents. Through standardized JSON format and deduplication mechanisms, the system ensures data consistency and reliability, thereby improving the overall efficiency of data acquisition and processing. Furthermore, this step lays the foundation for system scalability, supporting multi-channel and multi-source data collection strategies, enhancing the system's adaptability and intelligence.

[0089] S42, the unique identifier is generated by parsing keywords in the URL and combining them with underscores.

[0090] Specifically, in the neural network crawler agent processing without embedded prompt word optimizers, the system parses and processes the concatenated URL list one by one. One of the key steps is generating a unique identifier for each acquired webpage content. This identifier is generated by parsing keywords in the URL and combining them with underscores "_". For example, https: / / www.example.com / renewable-energy / 2025 is parsed as renewable_energy_2025. Technically, this method is based on Natural Language Processing (NLP) and string manipulation techniques. First, the path portion of the URL is segmented to extract semantically meaningful keywords, which are then concatenated in sequence with underscores to form a structured, highly readable, and unique identifier.

[0091] Furthermore, keyword extraction typically employs semantic analysis methods based on models such as TF-IDF or BERT to ensure that the extracted keywords accurately reflect the theme of the webpage content. In addition, the path hierarchy in the URL is also taken into consideration; for example, year information in the path (such as 2025) is retained as part of the identifier to enhance time-based traceability. In practical applications, this identifier will be used in file naming, data indexing, and subsequent Search Augmentation Generation (RAG) processes to ensure data uniqueness and manageability.

[0092] Specifically, this step is widely used in large-scale web data collection and knowledge base construction. In vertical fields such as energy transition and climate change, the system needs to process a large amount of dynamically updated web page content. By generating unique identifiers, duplicate crawling can be effectively avoided, data organization efficiency can be improved, and structured support can be provided for subsequent semantic retrieval and content analysis. This method is particularly suitable for integrating multi-source heterogeneous data, such as web page content from different sources such as government websites, international organizations, and academic institutions.

[0093] Specifically, this step significantly improves the efficiency and accuracy of data management through a structured naming mechanism. The unique identifier not only ensures the identifiability of each file but also provides fundamental support for subsequent metadata management, content classification, and semantic retrieval. This is combined with the system's multi-dimensional evaluation mechanism for prompt word optimizers (such as relevance scoring). The generation method of this identifier further enhances the semantic correlation between data and user intent, thereby improving the intelligence level and response efficiency of the overall data perception system.

[0094] S5 performs time-series monitoring of specific target links in the static URL list, dynamically replaces the keyword part in the URL according to the preset update rules, and generates an updated static URL list.

[0095] Specifically, this step is a crucial link in the system's continuous data acquisition and updating, and its technical implementation is based on a rule-driven URL reconstruction mechanism and a time-series control strategy. The system first extracts target links with regular update characteristics from a locally stored list of static URLs (such as crawler_urls_static.json). These links typically contain variable keyword parts, such as year, quarter, month, or version number for a specific topic. The system triggers the URL reconstruction process through a preset update cycle (such as monthly updates), using timestamps or calendar cycle information to replace the keyword parts in the URL with values ​​corresponding to the current time. For example, if the original URL is https: / / example.com / reports / 2024 / annual_report.pdf, then when updated in 2025, the system will replace 2024 with 2025, generating a new https: / / example.com / reports / 2025 / annual_report.pdf.

[0096] Furthermore, the system employs a regular expression matching mechanism to identify replaceable fields in the URL; for example, (d{4}) is used to match the year field. The replacement operation is based on a preset template rule base, stored in the system configuration, which includes parameters such as field matching patterns, replacement logic, and update cycles. For instance, the rule base can define that the `{year}` field should be replaced according to the current year, and the update cycle should be once a month.

[0097] Specifically, the system controls the update behavior through the following parameters: 1) update cycle (e.g., monthly, quarterly); 2) keyword field matching pattern (e.g., (d{4})); 3) current value of the replacement field (e.g., datetime.now().year); 4) URL reconstruction success rate (e.g., success_rate = success_count / total_urls), which is used to evaluate the effectiveness of the update strategy.

[0098] Specifically, this step is widely applicable to static data sources that require regular updates, such as government annual reports, industry white papers, and policy documents. By automatically reconstructing URLs, the system ensures that the crawler agent always accesses the latest data pages, thereby improving the timeliness and accuracy of the data.

[0099] S6 concatenates the updated static URL list with the dynamic URL list to form a mixed URL set containing historical and latest data.

[0100] Specifically, this step is technically implemented based on a JSON-formatted data structure, which programmatically merges lists of URLs from different sources to ensure data integrity and timeliness.

[0101] Specifically, the static URL list is typically stored in `crawler_urls_static.json`, containing predefined, long-term valid target links, such as policy documents and industry white papers. The dynamic URL list, on the other hand, is actively retrieved by the first neural network crawler agent, which embeds a prompt word optimizer, based on optimized prompt words. It is stored in `crawler_urls_dynamic.json`, containing real-time updated webpage links, such as news reports and research reports. Furthermore, the system supports a self-updating mechanism, using time-series monitoring and keyword replacement strategies in `crawler_urls_self_revising.json` to dynamically update some static links.

[0102] Furthermore, the concatenation operation reads the "urls" field from the three JSON files mentioned above, merges all URLs into a unified list, and removes duplicates to avoid repeated crawling. The concatenated set of mixed URLs is finally stored as crawler_urls.json.

[0103] Optionally, timestamps or source identifiers can be introduced during the splicing process to distinguish URLs from different channels, facilitating subsequent data tracing and priority sorting. For example, dynamic URLs can be appended with the field "source": "dynamic", static URLs with "source": "static", and self-revising URLs with "source": "self-revising", thereby achieving structured management of multi-source data.

[0104] Specifically, by integrating static and dynamic data sources, the system can simultaneously acquire long-term, stable authoritative information and real-time, cutting-edge content, providing comprehensive data input for the subsequent second neural network crawler agent that uses non-embedded prompt word optimizers. The construction of a hybrid URL set ensures data diversity and coverage, improves the efficiency and quality of knowledge base construction, and provides a richer and more accurate data foundation for training and inference of large language models.

[0105] This invention discloses a data awareness method for embedding a large language model prompt word optimizer. By introducing an intelligent optimization mechanism involving multi-dimensional evaluation and adaptive strategy iteration, it effectively addresses the core shortcomings of existing technologies, such as reliance on human experience in prompt word construction and difficulty in controlling data acquisition quality and efficiency. It achieves a closed-loop automation of the entire process, from intent parsing, automatic evaluation, and strategy optimization to multi-source data collection and standardized storage, significantly improving the relevance, accuracy, and completeness of the acquired data. Through the dual-channel fusion of dynamic and static data sources and continuous self-updating of the rule base, it enhances the system's adaptability and long-term maintenance efficiency in dealing with diverse network data, providing a high-quality and timely data foundation for the construction of a large language model knowledge base.

[0106] Example 2 Establish a data-aware system that embeds a large language model prompt word optimizer, such as Figure 2 As shown. It not only constructs a network-oriented architecture (such as...) Figure 3 The dual-channel prompt word neural network crawler agents (shown) can interact with the external network space separately, perceive data information, and, under the action of the prompt word optimizer, enhance their data perception capabilities and output high-quality content.

[0107] In one embodiment of the present invention, the working principle of the dual-channel prompt word neural network crawler agent includes: Based on the above explanation, Channel 1 continuously optimizes the cue word information in the crawler agent through a cue word optimizer, thereby actively acquiring dynamic network information links in the network space that match the target. This link information is then input into the neural network crawler agent without embedded cue word information for data acquisition and output. Channel 2, based on static network link targets, directly collects, stores, and outputs data information from the network space through the neural network crawler agent. These target links come from the output of the neural network crawler agent with embedded cue words and a known set of target links. Next, Figure 4 The working principles of the first neural network crawler agent with embedded prompts and the second neural network crawler agent without embedded prompts are described in detail.

[0108] Furthermore, the mechanism behind the API call code snippet that actively retrieves the URL is that the system proactively obtains network data through an application programming interface (API) call. Specifically, the system constructs an HTTP request pointing to a specific server endpoint (URL), which is constructed by concatenating the base URL with optimized prompt words parameters. When initiating the GET request, authentication and response control fields are set in the request header; the authentication field uses a holder token mechanism to authorize access, while the response control field instructs the server to return only metadata and not the actual content body. This call process aims to efficiently and securely retrieve dynamic network link information associated with the optimized prompt words.

[0109] Furthermore, based on the optimized prompts, the system retrieves associated network link information via an application programming interface (API) and stores these links in a structured data format. Specifically, the system generates a standard JSON file named crawler_urls_dynamic.json. This file contains an array field with the key "urls," which sequentially holds all dynamically retrieved target link addresses. This standardized file provides a unified and structured input source for subsequent data processing.

[0110] Furthermore, the system maintains a predefined set of static target links as a fixed source of data acquisition. This set is stored in a structured data format, specifically as a data object containing specific fields. Within this object, there is a list field for centrally storing link addresses, which sequentially carries all the predefined static target links. This standardized storage format provides a stable and structured input source for the system's subsequent data acquisition process.

[0111] Furthermore, the system also maintains a type of target link with regular update characteristics. For these links, the system identifies their update cycle through a time-series monitoring mechanism and dynamically replaces variable parts (such as time, version number, and other keywords) in the link based on preset rules, thereby achieving automatic maintenance and updating of the links. The updated links are stored in a structured data format, forming a self-correcting link set, providing a continuous, effective, and timely input source for data collection.

[0112] Furthermore, the system integrates link sets from multiple sources, including dynamically acquired, statically pre-stored, and self-correcting updates, to generate a unified and standardized set of target links to be processed. This set exists as a structured data object, containing a specific array field that sequentially carries the addresses of all links to be processed. This integrated and standardized data set provides a complete and unified input source for subsequent data acquisition and processing.

[0113] Furthermore, the core function of the non-embedded prompt word neural network crawler agent lies in traversing and processing the unified list of URLs to be processed generated in the aforementioned steps. Specifically, the agent accesses each target link address in the list by constructing specific HTTP requests. During the request process, access authorization is obtained by setting an authentication field containing a valid holder token in the request header; after obtaining the content of the target webpage, the agent performs standardized storage operations. During storage, the system generates a unique identifier for each webpage content as the file name. The rule for generating this identifier is: parsing the target link address, extracting the key semantic components, and combining these components by connecting them with underscores; through the above mechanism, the system realizes the automated collection and structured storage of multi-source, heterogeneous network data, providing a standardized data foundation for subsequent knowledge base construction and information processing.

[0114] Furthermore, the system stores the acquired webpage content in markup language format, which facilitates subsequent processing and analysis by a large language model. Simultaneously, the system generates and maintains specific metadata files to support in-depth data processing and management. Specifically, the system generates a result record file, which is a structured data object containing sets of successfully processed and unsuccessfully processed links. For each successfully processed link, the file stores its original web address, the corresponding webpage title, and the storage path of the generated markup language file. In addition, the system generates a path index file, using keywords as index keys, with corresponding values ​​being arrays that associate and store the paths of all markup language files related to a specific topic. The system achieves standardized storage and structured metadata management of the collected data, ensuring the traceability of the data processing process and the efficiency of subsequent knowledge processing stages.

[0115] In one embodiment of the present invention, the mechanism of the prompt word optimizer includes: Figure 4 This paper describes the computational method of the prompt word optimizer. Its working mechanism focuses on demonstrating the judgment criteria and optimization strategies for prompt word optimization, presenting the decision-making methods for when and how to optimize, forming a complete closed-loop system. Its core components include: 1) Prompt word input: receiving initial or optimized prompt words; 2) Large language model: processing prompt words and generating responses; 3) Performance evaluation: analyzing response quality and performance indicators; 4) Optimization decision: determining whether and how to optimize based on the evaluation results; 5) Prompt word reconstruction: modifying prompt words by applying the selected optimization strategy; 6) Output prompt words: outputting the optimized prompt words. The following sections introduce the large model prompt word optimizer – evaluation criteria and optimization strategy calculation formulas: Specifically, the Relevance Score is calculated as follows: RS = 0.4 * KWM + 0.3 * ITM + 0.3 * CTX; where KWM (keyword match) = number of keyword occurrences in the response / total expected keywords; ITM (intent match) = {0.2, 0.4, 0.6, 0.8, 1.0} (five-point scale score); CTX (contextual coherence) = 1 - (number of contextual deviations / total number of paragraphs in the response); optimization trigger condition: RS < 0.8; the Accuracy Score is calculated as follows: AS = 0.5 * FC + 0.3 * CC + 0.2 * LC; where FC (factual accuracy) = ... (Total number of statements - Number of incorrect statements) / Total number of statements; CC (Computational Accuracy) = (Number of correctly calculated steps) / Total number of calculated steps (1.0 if no calculations are performed); LC (Logical Consistency) = 1 - (Number of logical contradictions / Total number of arguments); Optimization trigger condition: AS < 0.7; Completeness Score: CS = 0.4 * RR + 0.4 * CR + 0.2 * DR; where, RR (Requirement Response Rate) = Number of responded requirements / Total number of requirements; CR (Coverage) = Breadth of topics covered by the response / Expected breadth of topics {0.2, 0.4, 0.6, 0.8, 1.0}; DR (Depth Ratio) = Actual depth level / Expected depth level (maximum 1.0); Optimization trigger condition: CS < 0.75; Efficiency Score: ES = 0.5 * TE + 0.5 *LE; where TE (Time Efficiency) = Target Response Time / Actual Response Time (maximum 1.0); LE (Length Efficiency) = Target Token Count / Actual Token Count (1.0 if actual < target); Optimization trigger condition: ES < target threshold (usually 0.85).

[0116] Specifically, the multi-dimensional evaluation system is designed specifically for the characteristics of content generated by large language models. For example, the evaluation of "intent matching (ITM)" usually requires the use of another "referee," the LLM, to calculate the deep semantic similarity between the generated result and the original intent; "contextual coherence (CTX)" aims to evaluate the logical disjointness that may occur in the generation of long texts by large models. This system constitutes a quantitative feedback loop for the quality of large model generation.

[0117] Specifically, the optimization strategy selection algorithms include: Instruction Clarification: IC_SCORE = 0.6 * (1 - ITM) + 0.4 * (1 - FC); application condition: IC_SCORE > 0.2 and is the highest-scoring strategy; Context Supplementation: CS_SCORE = 0.5 * (1 - CTX) + 0.3 * (1 - FC) + 0.2 * (1 - RR); application condition: CS_SCORE > 0.2 and is the highest-scoring strategy; Constraint Enhancement: CE_SCORE = 0.4 * (1 - LC) + 0.3 * (1 - CR) + 0.3 * (1 - LE); application condition: CE_SCORE > 0.2 and is the highest-scoring strategy; Structure Reorganization: SR_SCORE = 0.4 * (1 - DR) + 0.3 * (1 - RR) + 0.3 * (1 - TE); Application conditions: SR_SCORE > 0.2 and is the highest score strategy.

[0118] Specifically, the indicator network structure diagram can be deduced from the calculation formula of the cue word optimizer, such as... Figure 5 As shown, it describes the correlation between different indicator parameters: In the prompt word optimizer, the optimization judgment criterion is to trigger the optimization process when the response meets any of the following conditions: 1) Relevance is less than 0.8: The response content does not match the user's intent well; 2) Accuracy is less than 0.7: The response contains incorrect information or misleading content; 3) Completeness is less than 0.75: The response does not fully cover the key points of the user's needs; 4) Efficiency is less than the target threshold: The response generation time or token usage exceeds the expected order.

[0119] At the same time, appropriate optimization strategies are selected based on the judgment results, namely: instruction clarification: clarifying instruction language and eliminating ambiguity; context supplementation: adding necessary background information to improve the accuracy of understanding; constraint reinforcement: adding explicit constraints on output format, scope or style; and structural reorganization: adjusting the structure of prompt words to optimize information presentation.

[0120] In addition, the optimization rule base is a centrally maintained system that stores historical optimization records and their effects; preset optimization templates; optimal prompt word structures for specific analysis scenarios; and dynamically updated optimization strategy weights. Through read / write operations, the system interacts with the rule base to achieve knowledge accumulation and experience transfer.

[0121] In summary, the above formulas and calculation methods provide a data-driven framework for objectively evaluating the effectiveness of prompts and selecting the most suitable optimization strategy. The system ensures that optimization decisions are based on clear standards, rather than subjective judgments, by quantifying various indicators. The entire closed-loop system is characterized by continuous optimization, knowledge accumulation, and data-driven approaches. Specifically, the evaluation results directly drive the next round of optimization, and optimization experience is continuously accumulated into the knowledge base, ensuring that optimization decisions are made based on clear quantitative indicators. The specific optimization strategies are shown in Table 1. Table 1 Reference for Optimization Strategy Implementation Methods

[0122] Specifically, the optimization strategy library of this invention essentially encapsulates the most effective large-scale model prompting engineering paradigms currently available. For example, the "instruction clarification strategy" corresponds to the core concept of "Chain of Thought," guiding model logic through step decomposition; the "context supplementation strategy" corresponds to "Few-shot Learning," reducing model ambiguity by adding examples; and the "structural reorganization strategy" may introduce a "Role-Playing" setting, allowing the model to organize information as an expert. The system, driven by data, automatically selects the most suitable combination of prompting engineering techniques for specific tasks.

[0123] This invention discloses a data-aware system that embeds a large language model prompt word optimizer. By constructing a prompt word optimizer that embeds multi-dimensional quantitative evaluation and adaptive strategy selection algorithms, it effectively solves the core problems of existing technologies, such as prompt word construction relying on human experience, subjective optimization processes, and a lack of closed-loop feedback. This method achieves closed-loop optimization throughout the entire process, from intent input, effect evaluation, and strategy selection to dynamic prompt word reconstruction and automated multi-source data collection. The system significantly improves the relevance, accuracy, and completeness of network data acquisition, enhances the adaptive capability and decision-making intelligence of the data-aware system, and provides a reliable data foundation for building a high-quality knowledge base.

[0124] Example 3 To achieve the above invention, such as Figure 6 As shown, this embodiment also provides another data-aware method for embedding a large language model prompt word optimizer, including: S101: Based on the data, the initial prompt word is obtained as the intention input. After passing through a first neural network crawler Agent based on a large language model and embedding a prompt word optimizer, the initial result is returned and sent to the prompt word optimizer.

[0125] S102 evaluates and makes comprehensive decisions based on the calculation method of the prompt word optimizer, iterates and optimizes until it meets the standard, then exits and outputs the optimized prompt word return result.

[0126] S103, concatenate the URL list in the returned result with the target URL list obtained directly from the network space, and input it into the second neural network crawler Agent that is not embedded in the prompt word optimizer.

[0127] S104, a second neural network crawler agent without an embedded prompt word optimizer, processes the concatenated URL list to achieve dual-channel network data acquisition, output, and storage.

[0128] Specifically, in step S101, the initial prompt words are input into a first neural network crawler agent that embeds the prompt words. This agent integrates a web search plugin, which can convert the parsed intent into an actual search query. In the evaluation phase of step S102, the "fact accuracy rate" in the accuracy score is achieved by comparing the key statements in the model output with a pre-set high-precision fact database, which itself also relies on the information extraction and comparison capabilities of another large model.

[0129] Specifically, in step S101, the intention for data acquisition is first determined. In this embodiment, the focus is on information related to energy transition and climate change. Therefore, an initial prompt word P is proposed: {Query the latest information on energy transition and climate change}. Then, it is input into the first neural network crawler Agent embedded with the prompt word optimizer, and the initial result is returned as follows:

[0130] In step S102, calculations are performed based on the returned results and the prompt word optimizer mechanism. First, the basic index parameters of the returned results are evaluated, with specific values ​​shown in Table 2. Then, optimization judgment calculations are performed based on the basic index parameter values, yielding the values ​​of RS, AS, CS, ES (as shown in Table 3), and IC_SCORE, CS_SCORE, CE_SCORE, SR_SCORE (as shown in Table 4). The calculation method refers to the evaluation criteria and optimization strategy calculation formulas in the large model prompt word optimizer.

[0131] Table 2 Evaluation of Basic Indicator Parameters

[0132] Table 3 Optimization Judgment Calculation

[0133] Table 4 Calculation of Optimization Strategy Score

[0134] The optimization direction for CE_SCORE is derived from the above calculation data. Therefore, according to the optimization strategy implementation method reference in Table 1, the initial prompt word P is optimized to obtain the optimized prompt word P': {Please provide the latest information on energy transition and climate change globally by 2025. The content should include technological innovations, policy changes, and economic impacts of renewable energy, carbon neutrality, and green technologies, with particular attention to the latest developments in Europe and Asia by 2025.} This is then input into the first neural network crawler Agent embedded in the prompt word optimizer, and the optimization results are as follows:

[0135] Then, calculations are performed based on the returned results and the prompt word optimizer mechanism. The basic index parameters for evaluating the returned results are shown in Table 5. Then, optimization judgment calculations are performed on the data of the basic index parameter values, and the values ​​of RS, AS, CS, ES (as shown in Table 6), and IC_SCORE, CS_SCORE, CE_SCORE, SR_SCORE (as shown in Table 7) are obtained again.

[0136] Table 5 Re-evaluation of Basic Indicator Parameters

[0137] Table 6 Optimization Judgment and Recalculation

[0138] Table 7 Recalculation of Optimization Strategy Score

[0139] As can be seen, all of the above values ​​are less than 0.2 and meet the evaluation criteria. Therefore, the output is determined by optimizing the prompt words. The results of the optimized prompt words are returned, stored, and the URL list is extracted.

[0140] In step S103, the extracted URL list is concatenated with the target URL list directly obtained from the cyberspace to form the URL information source for this energy transition and climate change information acquisition, which is stored as crawler_urls.json. If the data volume is insufficient, steps S1 and S2 can be repeated multiple times to increase the number of URL information sources.

[0141] In step S104, a large language model agent without embedded prompt word optimizer (a second neural network crawler agent without embedded optimizer) reads crawler_urls.json to process the data sources obtained from the dual channels in the network space one by one and stores them as independent markdown files, providing data sources for the large model knowledge base.

[0142] In summary, this invention is not only a data acquisition method, but also a data supply layer optimization framework for large-scale model applications. It transforms prompts from static "incantations" into dynamic, evaluable, and optimizable "control commands," enabling the autonomous and evolutionary interaction of large models with open network environments, and providing a reliable infrastructure for building high-quality, high-efficiency RAG systems.

[0143] This invention discloses a data-aware method for embedding a large language model prompt word optimizer. Through specific application scenarios, its dynamic optimization mechanism based on quantitative evaluation and adaptive strategies is verified, effectively addressing the core shortcomings of existing technologies where prompt word construction relies on human experience and data acquisition quality is difficult to control. Taking the energy transition and climate change field as an example, it fully demonstrates the entire closed-loop process from intent input, multi-dimensional evaluation, strategy iterative optimization to high-quality data automated acquisition. It significantly improves the accuracy, completeness, and timeliness of data acquisition, and enhances the system's adaptability and long-term maintenance efficiency in vertical domains through self-updating of the rule base, providing reliable data support for building a high-quality domain knowledge base.

[0144] Example 4 To achieve the above invention, such as Figure 7 As shown, this embodiment also provides a data-aware device 10 that embeds a large language model prompt word optimizer. The device 10 includes: The intent parsing module 100 is used to obtain the initial prompt words for intent input based on data, obtain the initial results through a first neural network crawler Agent based on a large language model and embedded with a prompt word optimizer, and input the initial results into the prompt word optimizer; The evaluation calculation module 200 is used to calculate the relevance score, accuracy score, completeness score and efficiency score based on the multi-dimensional evaluation system of the prompt word optimizer. It determines whether optimization is needed based on the preset threshold. If so, it uses an adaptive strategy to select an algorithm to iteratively optimize the prompt words until the evaluation result meets the preset standard. The data source aggregation module 300 is used to concatenate the dynamic URL list corresponding to the optimized prompt words with the static URL list pre-stored in the network space to generate a set of URLs to be processed. The large language model acquisition and storage module 400 is used to traverse the set of URLs to be processed through a second neural network crawler Agent without embedded prompt word optimizer, obtain the content of the target web page and store it in a standardized format, and update the rule base with the strategy selection results during the optimization process.

[0145] In one embodiment of the present invention, the system further includes: a time-series monitoring module, used to perform time-series monitoring on specific target links in the static URL list, dynamically replace the keyword part in the URL according to a preset update rule, and generate an updated static URL list; and a secondary concatenation module, used to concatenate the updated static URL list with the dynamic URL list to form a mixed URL set containing historical data and the latest data.

[0146] This invention discloses a data perception device that embeds a large language model prompt word optimizer. Through integrated modules, it achieves quantitative evaluation and dynamic optimization of prompt words, effectively addressing the core shortcomings of existing technologies that rely on manual experience in data acquisition processes and lack adaptive optimization capabilities. The device realizes a closed-loop operation throughout the entire process, from intent parsing, multi-dimensional evaluation, and strategy iteration to automated collection and structured storage of multi-source data, significantly improving the accuracy and completeness of data acquisition and the system's adaptability. Through the collaborative work of various modules and continuous updates to the rule base, the device enhances its robustness and decision-making intelligence in complex network data environments, providing efficient automated tools to support the construction of high-quality knowledge bases.

[0147] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 8 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads the executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the data perception method of embedding a large language model prompt word optimizer described above.

[0148] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a data-aware method for embedding a large language model prompt word optimizer as described in the foregoing embodiments.

[0149] In the description of this specification, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0150] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.< / timestamp> < / keyword2> < / keyword1> < / token> < / token> < / token> < / token>

Claims

1. A data-aware method for embedding a large language model prompt word optimizer, characterized in that, include: S1, based on the data, the initial prompt word is input to obtain the intent. The initial result is obtained through a first neural network crawler Agent based on a large language model and embedded with a prompt word optimizer, and the initial result is input into the prompt word optimizer. S2, based on the multi-dimensional evaluation system of the prompt word optimizer, calculates the relevance score, accuracy score, completeness score and efficiency score, and determines whether optimization is needed according to the preset threshold. If so, the prompt words are iteratively optimized by an adaptive strategy selection algorithm until the evaluation result meets the preset standard. S3, concatenate the dynamic URL list corresponding to the optimized prompt words with the static URL list pre-stored in the network space to generate a set of URLs to be processed; S4 uses a second neural network crawler agent without an embedded prompt word optimizer to traverse the set of URLs to be processed, obtain the content of the target web pages and store it in a standardized format, and update the rule base with the strategy selection results during the optimization process.

2. The method as described in claim 1, characterized in that, The process of obtaining the initial prompt word based on the data-driven intent input, acquiring the initial result through a first neural network crawler agent based on a large language model and embedded with a prompt word optimizer, and inputting the initial result into the prompt word optimizer includes: S11, the initial prompt words include specific domain-specific words and time-range-specific words; S12, when making an API call, set "Authorization" in the request header: "Bearer <token> The "X-Respond-With": "no-content" parameter controls the response format.< / token> 3. The method as described in claim 1, characterized in that, The multi-dimensional evaluation system based on the prompt word optimizer calculates relevance score, accuracy score, completeness score, and efficiency score. It determines whether optimization is needed based on preset thresholds. If so, it iteratively optimizes the prompt words using an adaptive strategy selection algorithm until the evaluation result meets preset standards, including: S21, the formula for calculating the relevance score RS is: RS = 0.4*KWM + 0.3*ITM + 0.3*CTX; KWM is the ratio of the number of keyword occurrences in the response to the total number of expected keywords, ITM is the intent matching score based on a five-level scale, and CTX is the ratio of 1 minus the number of context deviations to the total number of paragraphs in the response; the formula for calculating the accuracy score AS is: AS = 0.5 * FC + 0.3 * CC + 0.2 * LC; FC is the factual accuracy rate, CC is the calculation accuracy rate, and LC is the logical consistency rate; the formula for calculating the completeness score CS is: CS = 0.4 * RR + 0.4 * CR + 0.2 * DR; RR is the demand response rate, CR is the coverage rate, and DR is the depth ratio; the formula for calculating the efficiency score ES is: ES = 0.5 * TE + 0.5 * LE; TE is the time efficiency, and LE is the length efficiency. S22, the adaptive strategy selection algorithm calculates the scores of four optimization strategies, including instruction clarity strategy score IC_SCORE, context supplementation strategy score CS_SCORE, constraint reinforcement strategy score CE_SCORE, and structure reorganization strategy score SR_SCORE, and selects the corresponding optimization strategy to optimize the prompt words based on the score of each strategy and the preset conditions.

4. The method as described in claim 1, characterized in that, The process involves a second neural network crawler agent, without an embedded prompt word optimizer, traversing the set of URLs to be processed, obtaining the target webpage content and storing it in a standardized format, and simultaneously updating the rule base with the strategy selection results from the optimization process. This includes: S41, the standardized format is Markdown; the Markdown format includes heading, body, and metadata fields; S42, the unique identifier is generated by parsing keywords in the URL and combining them with underscores.

5. The method as described in claim 1, characterized in that, Also includes: S5 performs time-series monitoring of specific target links in the static URL list, dynamically replaces the keyword part in the URL according to the preset update rules, and generates an updated static URL list. S6 concatenates the updated static URL list with the dynamic URL list to form a mixed URL set containing historical and latest data.

6. A data-aware device for embedding a large language model prompt word optimizer, characterized in that, include: The intent parsing module is used to obtain the initial prompt words for the intent input based on the data. It obtains the initial results through a first neural network crawler Agent based on a large language model and embedded with a prompt word optimizer, and inputs the initial results into the prompt word optimizer. The evaluation calculation module is used to calculate the relevance score, accuracy score, completeness score and efficiency score based on the multi-dimensional evaluation system of the prompt word optimizer. It determines whether optimization is needed based on the preset threshold. If so, it uses an adaptive strategy to select an algorithm to iteratively optimize the prompt words until the evaluation result meets the preset standard. The data source aggregation module concatenates the dynamic URL list corresponding to the optimized prompt words with the static URL list pre-stored in the network space to generate a set of URLs to be processed. The large language model acquisition and storage module is used to traverse the set of URLs to be processed through a second neural network crawler agent without embedded prompt word optimizer, obtain the content of the target web pages and store it in a standardized format, and update the rule base with the strategy selection results during the optimization process.

7. The apparatus as claimed in claim 6, characterized in that, Also includes: The time-series monitoring module is used to monitor specific target links in the static URL list in a time-series manner, dynamically replace the keyword part in the URL according to the preset update rules, and generate an updated static URL list. The secondary concatenation module is used to concatenate the updated static URL list with the dynamic URL list to form a mixed URL set containing historical and latest data.

8. An electronic device, comprising: processor; The memory stores executable instructions; when the processor executes the instructions, it implements the data-aware method for embedding a large language model prompt word optimizer as described in any one of claims 1-5.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, implements a data-aware method for embedding a large language model prompt word optimizer as described in any one of claims 1-5.

Images