Large model generation code verification method and device, equipment and storage medium

Abstract

The invention discloses a large model generation code verification method and device, equipment and a storage medium. The method comprises the steps that a target code generated by a large model based on natural language input and a first knowledge abstract is obtained, the first knowledge abstract comprises multiple knowledge dimensions corresponding to first reference knowledge, and the first reference knowledge is knowledge meeting a first similar condition with the natural language input in a knowledge base; generating a code abstract corresponding to the target code; based on the importance degrees of the multiple knowledge dimensions in the code abstract, second reference knowledge meeting a second similar condition with the code abstract is retrieved in the knowledge base; the first knowledge abstract and the second reference knowledge corresponding to each key knowledge dimension are compared, difference knowledge corresponding to each key knowledge dimension is determined, and the weight of the key knowledge dimension is larger than a preset threshold value; and based on the matching relationship between the difference knowledge corresponding to the plurality of key knowledge dimensions and the code abstract, and the user intention, verifying the target code to obtain a verification result.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a method, apparatus, device and storage medium for verifying code generated from a large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, it is now possible to automatically generate executable code based on users' natural language descriptions using large models, significantly improving development efficiency. However, when generating code, large models may generate seemingly reasonable but technically flawed code due to the "illusion" problem, or deviate from user intent due to a lack of real, executable factual basis, resulting in unreliable generated code. Furthermore, the black-box nature of large models and their complex structure with trillions of parameters make their reasoning process difficult to interpret, and even if unreliable code is detected, tracing the root cause of the problem is challenging.

[0003] Therefore, there is an urgent need for a verification method for code generated by large models to determine whether the code is generated based on facts and whether the generated code conforms to the user's intent. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for verifying large model generation code, which can achieve dual verification of the reliability of large model generation code from factual basis to user intent.

[0005] In a first aspect, embodiments of this application provide a method for verifying large model generation code, the method comprising: Obtain the target code generated by the large model based on natural language input and a first knowledge summary. The first knowledge summary includes multiple knowledge dimensions obtained by analyzing and summarizing the first reference knowledge. The first reference knowledge is knowledge in the knowledge base that satisfies the first similarity condition with the natural language input. The target code is parsed and logically analyzed to generate a code summary corresponding to the target code. Based on the importance of the multiple knowledge dimensions in the code summary, the weights corresponding to the multiple knowledge dimensions are determined respectively; Based on the weights corresponding to the multiple knowledge dimensions, second reference knowledge that satisfies the second similarity condition with the code summary is retrieved from the knowledge base. By comparing the first knowledge summary and the second reference knowledge corresponding to each key knowledge dimension, the differential knowledge corresponding to each key knowledge dimension is determined. The key knowledge dimension is the knowledge dimension whose weight is greater than a preset threshold among the multiple knowledge dimensions. Determine the matching relationship between the differential knowledge corresponding to each of the multiple key knowledge dimensions and the code summary; Based on the matching relationship and the user intent corresponding to the natural language input, the target code is verified to obtain the verification result.

[0006] In one possible implementation, the step of performing syntax parsing and logical analysis on the target code to generate a code digest corresponding to the target code includes: The target code is parsed and its logic is analyzed to extract the first key information of the target code; Based on the first key information, the code digest is generated.

[0007] In one possible implementation, retrieving second reference knowledge from the knowledge base that satisfies the second similarity condition to the code digest, based on the weights corresponding to the multiple knowledge dimensions, includes: In the code summary, determine the summary content corresponding to each of the multiple knowledge dimensions; Based on the weights corresponding to the multiple knowledge dimensions, the number of search results corresponding to each of the multiple knowledge dimensions is determined. Based on the number of search results corresponding to the multiple knowledge dimensions, multiple knowledge fragments that satisfy the second similarity condition with the summary content corresponding to the multiple knowledge dimensions are retrieved in the knowledge base. Based on the aforementioned multiple knowledge fragments, the second reference knowledge is determined.

[0008] In one possible implementation, the step of verifying the target code based on the matching relationship and the user intent corresponding to the natural language input to obtain a verification result includes: If the discrepancies between the knowledge corresponding to the multiple knowledge dimensions and the code summary do not match, the verification result is determined to be a verification failure, and the reason for the failure is the illusion of a large model. When the differential knowledge corresponding to the multiple knowledge dimensions matches the code summary, the inclusion relationship between the second reference knowledge and the knowledge required by the user intent is determined. If the second reference knowledge does not fully contain the knowledge required for the user's intent, the verification result is determined to be a verification failure, and the reason for the failure is a compromise of intent due to insufficient knowledge coverage.

[0009] In one possible implementation, after obtaining the verification result, the method further includes: If the verification result is a verification failure and the reason for the failure is a large model illusion, an optimization suggestion is added to the verification result. The optimization suggestion is used to prompt the supplementation of missing knowledge in the knowledge base. If the verification result is a verification failure, and the reason for the failure is a compromise of intent due to insufficient knowledge coverage, add a result description and the optimization suggestion to the verification result. The result description is used to indicate the difference between the target code and the user intent.

[0010] In one possible implementation, obtaining the target code of the large model based on natural language input and the first knowledge summary includes: The natural language input is subjected to key information extraction to obtain the second key information; Retrieve the first reference knowledge from the knowledge base that satisfies the first similarity condition to the second key information; The first reference knowledge is analyzed and summarized to determine the multiple knowledge dimensions and their corresponding knowledge, thereby obtaining the first knowledge summary. The target code is generated based on the natural language input, the first knowledge summary, and the code generation specification.

[0011] In one possible implementation, the step of analyzing and summarizing the first reference knowledge to determine the multiple knowledge dimensions and their respective corresponding reference knowledge, and obtaining a first knowledge summary, includes: Based on the natural language input, the first reference knowledge is analyzed and summarized to determine the multiple knowledge dimensions and their respective corresponding reference knowledge, as well as the missing knowledge of the first reference knowledge relative to the natural language input, thereby obtaining the first knowledge summary.

[0012] In one possible implementation, the step of verifying the target code based on the matching relationship and the user intent corresponding to the natural language input to obtain a verification result includes: The user intent is determined based on the natural language input, the second key information, and the first knowledge summary; Based on the matching relationship and the user intent, the target code is verified to obtain the verification result.

[0013] Secondly, embodiments of this application provide a verification device for large model generation code, the device comprising: The acquisition module is used to acquire the target code generated by the large model based on natural language input and a first knowledge summary. The first knowledge summary includes multiple knowledge dimensions obtained by analyzing and summarizing the first reference knowledge. The first reference knowledge is knowledge in the knowledge base that satisfies the first similarity condition with the natural language input. The generation module is used to perform syntax parsing and logical analysis on the target code and generate a code summary corresponding to the target code. The determination module is used to determine the weights corresponding to the multiple knowledge dimensions respectively based on their importance in the code digest; The retrieval module is used to retrieve second reference knowledge that satisfies the second similarity condition with the code summary in the knowledge base based on the weights corresponding to the multiple knowledge dimensions respectively. The determining module is further configured to compare the first knowledge summary and the second reference knowledge corresponding to each key knowledge dimension to determine the difference knowledge corresponding to each key knowledge dimension, wherein the key knowledge dimension is the knowledge dimension with a weight greater than a preset threshold among the multiple knowledge dimensions; The determining module is further configured to determine the matching relationship between the differential knowledge corresponding to the multiple key knowledge dimensions and the code summary; The verification module is used to verify the target code based on the matching relationship and the user intent corresponding to the natural language input, and obtain the verification result.

[0014] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements any of the possible implementations of the first aspect described above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the method in any of the possible implementations of the first aspect described above.

[0016] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a method as described in any of the possible implementations of the first aspect above.

[0017] This application embodiment parses and logically organizes the target code generated by a large model to generate a code summary corresponding to the target code. Based on the importance of multiple knowledge dimensions in the code summary, it determines the weights corresponding to each knowledge dimension, quantifying the key knowledge dimensions relied upon in generating the target code. By retrieving second reference knowledge from the knowledge base that satisfies the second similarity condition to the target code based on the weights corresponding to each knowledge dimension, it can supplement and improve the knowledge corresponding to the key knowledge dimensions. Since both the first and second reference knowledge are objective knowledge retrieved from the knowledge base, by comparing the first knowledge summary and the second reference knowledge corresponding to each key knowledge dimension, it determines the difference knowledge corresponding to each key knowledge dimension. Based on the matching relationship between the difference knowledge corresponding to each key knowledge dimension and the code summary, and the user intent corresponding to the natural language input, it verifies the target code. Based on the supplemented and improved knowledge, it can objectively determine from the root whether the target code is generated based on facts and whether it conforms to user intent, thereby achieving a dual verification of the reliability of the code generated by the large model from both factual basis and user intent. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a method for verifying large model generation code provided in one embodiment of this application; Figure 2 This is a schematic diagram of the structure of a verification device for large model generation code provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0023] Furthermore, the acquisition, storage, use, and processing of data in this application's technical solution all comply with relevant national laws and regulations.

[0024] With the rapid development of artificial intelligence technology, it is now possible to automatically generate executable code based on users' natural language descriptions using large models, significantly improving development efficiency. However, when generating code, large models may generate seemingly reasonable but technically flawed code due to the "illusion" problem, or deviate from user intent due to a lack of real, executable factual basis, resulting in unreliable generated code. Furthermore, the black-box nature of large models and their complex structure with trillions of parameters make their reasoning process difficult to interpret, and even if unreliable code is detected, tracing the root cause of the problem is challenging.

[0025] Therefore, there is an urgent need for a verification method for code generated by large models to determine whether the code is generated based on facts and whether the generated code conforms to the user's intent.

[0026] To address the related technical issues, embodiments of this application provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for verifying large model generation code.

[0027] The following describes the verification method for the large model generation code provided in the embodiments of this application.

[0028] Figure 1 This diagram illustrates a flowchart of a method for verifying large model generated code according to an embodiment of this application. This method for verifying large model generated code can be executed by a code generation and verification system. Figure 1 As shown, the verification method for large model generation code provided in this application includes the following steps: S110. Obtain the target code generated by the large model based on natural language input and the first knowledge summary. The first knowledge summary includes multiple knowledge dimensions obtained by analyzing and summarizing the first reference knowledge. The first reference knowledge is the knowledge in the knowledge base that satisfies the first similarity condition with the natural language input. S120. Perform syntax parsing and logical analysis on the target code to generate a code summary corresponding to the target code; S130. Based on the importance of multiple knowledge dimensions in the code summary, determine the weights corresponding to each of the multiple knowledge dimensions; S140. Based on the weights corresponding to multiple knowledge dimensions, retrieve second reference knowledge in the knowledge base that satisfies the second similarity condition with the code summary; S150. Compare the first knowledge summary and second reference knowledge corresponding to each key knowledge dimension to determine the differential knowledge corresponding to each key knowledge dimension. The key knowledge dimension is the knowledge dimension with a weight greater than a preset threshold among multiple knowledge dimensions. S160. Determine the matching relationship between the differential knowledge and code summaries corresponding to multiple key knowledge dimensions; S170. Based on the matching relationship and the user intent corresponding to the natural language input, the target code is verified to obtain the verification result.

[0029] This application embodiment parses and logically organizes the target code generated by a large model to generate a code summary corresponding to the target code. Based on the importance of multiple knowledge dimensions in the code summary, it determines the weights corresponding to each knowledge dimension, quantifying the key knowledge dimensions relied upon in generating the target code. By retrieving second reference knowledge from the knowledge base that satisfies the second similarity condition to the target code based on the weights corresponding to each knowledge dimension, it can supplement and improve the knowledge corresponding to the key knowledge dimensions. Since both the first and second reference knowledge are objective knowledge retrieved from the knowledge base, by comparing the first knowledge summary and the second reference knowledge corresponding to each key knowledge dimension, it determines the difference knowledge corresponding to each key knowledge dimension. Based on the matching relationship between the difference knowledge corresponding to each key knowledge dimension and the code summary, and the user intent corresponding to the natural language input, it verifies the target code. Based on the supplemented and improved knowledge, it can objectively determine from the root whether the target code is generated based on facts and whether it conforms to user intent, thereby achieving a dual verification of the reliability of the code generated by the large model from both factual basis and user intent.

[0030] The specific implementation methods for each of the above steps are described below.

[0031] In some embodiments, in S110, the large model can be short for Large Language Model (LLM). The large model can be used to generate target code corresponding to the natural language input. The natural language input can be a user query for data, such as "Query the total number of user transactions this month". Correspondingly, the target code can be Structured Query Language (SQL) code.

[0032] To improve the accuracy of the target code, before generating the target code, a first reference knowledge that meets the first similarity condition with the natural language input can be retrieved from the knowledge base. Then, the first reference knowledge is summarized and analyzed to determine multiple knowledge dimensions and their corresponding knowledge, resulting in a first knowledge summary. Finally, the target code is generated based on the first knowledge summary and the natural language input through a large model.

[0033] To further improve the accuracy of the target code, in some embodiments, the above-mentioned S110 may specifically include: Key information is extracted from the natural language input to obtain the second key information; Retrieve first reference knowledge from the knowledge base that satisfies the first similarity condition to the second key information; The first reference knowledge is analyzed and summarized to identify multiple knowledge dimensions and their corresponding knowledge, thus obtaining the first knowledge summary; The target code is generated based on natural language input, first knowledge summary, and code generation specifications.

[0034] Here, after receiving natural language input, the code generation and verification system can first use Natural Language Processing (NLP) technology to perform word segmentation, entity recognition, and initial intent screening on the natural language input, extracting key information such as core keywords, entities, and business requests to obtain the second key information. If the natural language input is "query the total number of transactions for users this month", then the second key information can include "this month", "users", and "total number of transactions".

[0035] In addition, the knowledge base can include various types of knowledge such as business rules, data dictionaries, historical cases, and syntax rules. The storage formats of this knowledge include, but are not limited to, text and vector formats. If the knowledge base stores these various types of knowledge in text format, it can include multiple knowledge texts. After obtaining the second key information, multiple knowledge texts can be filtered based on preset matching rules (such as keyword matching and regular expression matching) to obtain multiple first knowledge texts that meet the matching conditions of the second key information. These multiple first knowledge texts can form a structured Retrieval-Augmented Generation (RAG) recall reference information set to obtain first reference knowledge that meets the first similarity condition with the second key information. The first reference knowledge can include business rules, data dictionaries, historical cases, and syntax rules.

[0036] If the knowledge base stores the aforementioned multiple types of knowledge in vector form, it can include multiple knowledge vectors. After obtaining the second key information, it can be converted into vector form to obtain the first vector. Then, based on vector similarity algorithms (such as cosine similarity or Euclidean distance), the similarity between the first vector and multiple knowledge vectors is calculated to obtain multiple first similarities. The knowledge vector corresponding to the first target similarity value among the multiple first similarities that is greater than a preset similarity value can be considered the first knowledge vector. The reference knowledge corresponding to the multiple first knowledge vectors can form a structured RAG recall reference information set, obtaining first reference knowledge that satisfies the first similarity condition with the second key information.

[0037] Of course, if multiple pieces of knowledge in the knowledge base are stored in both vector and text formats, the vector matching results and text matching results can be combined to jointly determine the first reference knowledge that satisfies the first similarity condition with the second key information. Specifically, multiple candidate knowledge vectors can be obtained first through vector matching, and multiple candidate knowledge texts that satisfy the matching condition with the second key information can be obtained through text matching. Then, the reference knowledge corresponding to the candidate knowledge vectors and the reference knowledge corresponding to the candidate knowledge texts are fused to obtain a fused reference knowledge set. Finally, a structured RAG recall reference information set is formed based on the fused reference knowledge set to obtain the first reference knowledge that satisfies the first similarity condition with the second key information. The fusion process can be implemented in any of the following ways: directly merging the reference knowledge corresponding to the candidate knowledge vectors and the reference knowledge corresponding to the candidate knowledge texts into a fused reference knowledge set; using a weighted fusion algorithm to deduplicate and sort the reference knowledge corresponding to the candidate knowledge vectors and the candidate knowledge texts to obtain a fused reference knowledge set; firstly, multiple candidate knowledge texts are initially screened through text matching, and then vector matching is performed on the knowledge vectors corresponding to the screened candidate knowledge texts to determine the fused reference knowledge set based on the vector matching results.

[0038] As an example, if the knowledge base includes data related to indicator statistics tables, and these tables only record online transaction counts without data on offline transaction counts or total transaction counts, then with the natural language input "Query the total number of transactions for users this month," and the second key information including "this month," "users," and "total number of transactions," vector retrieval can retrieve the metadata of the indicator statistics tables (containing only descriptions of online transaction counts, table structure, data storage location, etc.) from the knowledge base, but will not retrieve the relevant knowledge on the total number of transactions and offline transaction counts.

[0039] Based on this, the first reference knowledge can be analyzed and summarized using a large model to determine multiple knowledge dimensions and their corresponding knowledge, thus obtaining a first knowledge summary. Specifically, the RAG recall reference information set (i.e., the first reference knowledge) is used as input, and prompts containing instructions such as "classify knowledge by knowledge dimension," "extract core information," "remove redundant content," and "organize business logic" are sent to the large model. This drives the large model to analyze and summarize fragmented recall information, generating a concise and structured RAG summary text, thus obtaining the first knowledge summary. This first knowledge summary retains the core business rules, key data constraints, and grammatical requirements of the original first reference knowledge, serving as the core knowledge basis for subsequent code generation and verification.

[0040] Based on this, in order to improve the accuracy of code generation and verification, in some embodiments, the first reference knowledge is analyzed and summarized to determine multiple knowledge dimensions and their corresponding reference knowledge, thereby obtaining a first knowledge summary, which may specifically include: Based on natural language input, the first reference knowledge is analyzed and summarized to determine multiple knowledge dimensions and their corresponding reference knowledge, as well as the missing knowledge of the first reference knowledge relative to the natural language input, thus obtaining the first knowledge summary.

[0041] Here, when analyzing and summarizing the first reference knowledge through a large model, natural language input can also be provided to the large model. Thus, the first knowledge summary not only includes multiple knowledge dimensions and their corresponding reference knowledge, but also a summary description describing the "missing knowledge of the first reference knowledge relative to the natural language input." For example, in the natural language input "Query the total number of transactions for users this month," the first knowledge summary could be something like: "Field existence: The indicator statistics table only contains the online_trade_num (number of online transactions) field, with no fields or data related to total transactions or offline transactions; Transaction count range: The table stores the details of each user's online transaction count for each month." Here, "no fields or data related to transaction count or offline transactions" can represent the missing knowledge of the first reference knowledge relative to the natural language input. "Field existence" and "Transaction count range" can be knowledge dimensions.

[0042] This application embodiment analyzes and summarizes first reference knowledge based on natural language input to obtain a first knowledge summary. This first knowledge summary not only clarifies the knowledge dimensions of the reference knowledge but also identifies any missing knowledge relative to the natural language input. Therefore, by generating and verifying code based on this first knowledge summary, the accuracy of code generation and verification can be improved.

[0043] After obtaining the first knowledge summary, the target code can be generated by the large model based on the natural language input, the first knowledge summary, and the code generation specifications. Specifically, before this, the code generation and verification model can pre-build a prompt word template to instruct the large model to generate the target code. The prompt word template can include the filling positions for the natural language input, the first knowledge summary, and the code generation specifications. If the code is SQL code, the prompt word template can also include the filling positions for table names and fields. The code generation specifications can include syntax standards, output formats, and business constraints. For example, the prompt word template could be: "Based on the first knowledge summary A, generate the target code corresponding to the natural language input B, requiring the syntax to conform to code generation specifications C, and the table name to use D, with the time field as E." Based on this, after generating the first knowledge summary, the code generation and verification model can determine the required table names and field names for the prompt word template from the first knowledge summary, and fill in the corresponding positions with the first knowledge summary A, natural language input B, code generation specifications C, the table name using D, and the time field as E, obtaining the prompt words used to generate the target code, and then inputting these prompt words into the large model. Once the large model receives the prompt word, it can leverage its powerful natural language understanding capabilities and its existing knowledge base to generate the target code based on that prompt word.

[0044] This application embodiment analyzes and summarizes the first reference knowledge to determine multiple knowledge dimensions and their corresponding knowledge, obtains a first knowledge summary, and generates target code based on the first knowledge summary, which can further improve the accuracy of the generated code.

[0045] It should be noted that this step involves a high degree of freedom in generating large models and is affected by the matching degree of RAG recall information (i.e., first reference knowledge), which carries a certain risk of generating illusions, such as generating non-existent syntax or logic that violates business rules.

[0046] Following the example above, if the natural language input is "Query the total number of transactions for users this month", and the first knowledge summary is "Field existence: The indicator statistics table only contains the online_trade_num (number of online transactions) field, and has no related fields or data for total transaction count or offline transaction count; Transaction count statistics range: The table stores the details of the online transaction count for each user each month", then because the knowledge in the first knowledge summary does not cover all user intent, the large model may generate two types of target code. First, by fabricating the total_trade_num (total number of transactions) or offline_trade_num (number of offline transactions) fields, a phantom result is generated. The target code could be, for example: SELECT SUM(total_trade_num) ASmonth_total_trade_num FROM indicator_statistics WHERE trade_month = DATE_FORMAT(NOW(), '%Y-%m'). Secondly, only the number of online transactions is counted, which means that the number of online transactions is equated with the total number of transactions by default, thus generating a compromise result. The target code could be: SELECT SUM(online_trade_num) AS month_total_trade_num FROM indicator_statistics WHERE trade_month = DATE_FORMAT(NOW(), '%Y-%m').

[0047] In addition, after obtaining the aforementioned second key information, while analyzing and summarizing the first reference knowledge to obtain the first knowledge summary, the following process can be executed in parallel: A pre-trained text vector model is used to convert the second key information into a high-dimensional dense vector. Simultaneously, the vector is normalized to eliminate dimensional differences, resulting in a second vector. Subsequently, the generated second vector is associated with metadata such as the unique session identifier corresponding to the natural language input and the natural language input timestamp, and stored in a vector database. This provides a data foundation for subsequent dialogue memory construction and reserves an interface for knowledge association retrieval in cross-turn dialogues. The text vector model can be, for example, Bidirectional Encoder Representations from Transformers (BERT) or Sentence-level BERT.

[0048] Thus, by executing the generation processes of the first knowledge summary and the second vector in parallel, the preprocessing time can be compressed to the maximum extent, providing high-quality knowledge input and data support for code generation and verification.

[0049] In addition, after generating the second vector, during the process of generating the target code based on natural language input, the first knowledge summary, and code generation specifications, the following process can be executed in parallel: According to a preset protocol format, the second vector, the first knowledge summary, the session identifier, the timestamp, and the natural language input are aggregated and encapsulated to obtain code verification reference information, which is then written to the current dialogue's dialogue memory module (such as a local cache, Redis cache, or distributed database). The preset protocol format is a predefined protocol format used to specify the storage fields of the dialogue memory. This protocol format can be, for example, JavaScript Object Notation (JSON) or Protocol Buffers (Protobuf). Furthermore, this dialogue memory module, as the core context carrier for subsequent verification steps, must ensure the real-time nature and traceability of the data, providing a complete reference for code verification.

[0050] Following the example above, taking JSON as an example, the code verification reference information can be as follows: { "session_id": "session_20251120_001", "query_key_emb": [0.12, 0.35, ..., 0.21], / / Second vector "rag_summary": "The indicator statistics table only contains the online_trade_num (number of online transactions) field; it lacks fields and data related to total transactions and offline transactions...". "query_text": "Query the total number of user transactions this month", "create_time": "2025-11-20 10:00:00"} Storing the structured data into the dialogue memory module provides a complete contextual information carrier for subsequent verification steps, ensuring that the code verification reference information can be read in real time during the code verification phase.

[0051] Furthermore, by protocol-aggregating the second vector and the first knowledge summary and writing them into a dedicated dialogue memory module, the traditional temporary storage mode of RAG knowledge, which is used and discarded immediately, is broken through. By defining storage fields through a preset protocol format, the second vector, the first knowledge summary, and the question metadata (session identifier, timestamp) are associated and stored, realizing the traceability and cross-stage reuse of knowledge. The dialogue memory module, as the data hub for the three stages of preprocessing, generation, and verification, provides a unified data interface for subsequent retrieval of second reference knowledge and the introduction of context in the verification stage. This allows the scattered RAG knowledge and question information to form a structured whole, improving the modularity and scalability of the architecture.

[0052] Furthermore, by executing target code generation and dialogue memory construction in parallel during the code generation phase, and by executing the generation of the first knowledge summary and the second vector in parallel during the aforementioned data preprocessing phase, a multi-stage parallel task scheduling pipeline architecture design was implemented. This, in turn, maximizes the efficiency of code generation and verification through two-stage parallel scheduling. This design decouples and synchronously executes non-dependent tasks, avoiding process delays caused by single-task blocking and ensuring real-time data exchange between stages. It provides architectural support for the efficient deployment of large-model code generation, and is particularly suitable for high-concurrency conversational code generation scenarios.

[0053] In some embodiments, in S120, the code digest can be obtained by extracting the core logic and intent of the target code. The code digest can serve as the core basis for secondary RAG retrieval. Secondary RAG retrieval is the process of retrieving second reference knowledge.

[0054] Following the example above, if the target code is generated based on the illusion of a large model, the code summary could be: "SQL code calculates the total number of transactions this month using SUM(total_trade_num), using the total_trade_num field which does not exist in the indicator statistics table." If the target code is generated through compromise, the code summary could be: "SQL code calculates the total number of transactions this month using SUM(online_trade_num), only counting online transactions and excluding offline transaction data."

[0055] Therefore, in order to improve the accuracy of code digest, in some embodiments, the above-mentioned S120 may specifically include: Perform syntax analysis and logical analysis on the target code to extract the first key information from the target code; Based on the first key information, a code summary is generated.

[0056] Here, the first key information may include core business logic, the object of operation, data conditions, and other key information.

[0057] By first parsing and logically organizing the generated code, extracting the core business logic, operation objects, data conditions, and other key information, and then generating a code summary based on this key information, the accuracy of the code summary can be improved.

[0058] In some embodiments, during S130, the content in the code digest may involve one or more of the aforementioned multiple knowledge dimensions. Based on the importance of the summary content corresponding to each of the multiple knowledge dimensions in the code digest, the overall importance of the multiple knowledge dimensions in the code digest can be determined.

[0059] As an example, after determining the code summary, both the code summary and the first knowledge summary can be input into the large model. The large model can be provided with prompts to determine the importance of multiple knowledge dimensions within the code summary, guiding it to perform semantic association analysis between the code summary and the first knowledge summary. The large model can first determine the summary content corresponding to each knowledge dimension in the code summary, then determine the importance of each knowledge dimension in the code summary based on the semantic importance of the summary content, and finally output the quantified weights corresponding to each knowledge dimension. This achieves dynamic weight allocation based on an attention mechanism, thereby enabling the subsequent secondary RAG retrieval to focus on specific knowledge dimensions. Furthermore, correspondingly, during the initial RAG retrieval (i.e., retrieving the first reference knowledge), the weights corresponding to each knowledge dimension are equal.

[0060] Following the example above, if the code summary is "SQL code calculates the total number of transactions this month using SUM (total_trade_num), but uses the total_trade_num field which does not exist in the indicator statistics table" or "SQL code calculates the total number of transactions this month using SUM (online_trade_num), but only counts online transactions and does not include offline transaction data," then the weight of the two dimensions "field existence" and "transaction count range" can be increased (e.g., weighted at 80%), while the weight of the two dimensions "time filtering logic" and "table name matching" can be decreased (e.g., weighted at 20%), thereby enabling subsequent secondary RAG retrieval to focus on the knowledge dimensions.

[0061] In some embodiments, in S140, based on the adjusted knowledge dimension weights, the code summary is converted into a vector and then RAG retrieval is performed again. This can recall knowledge base information that is highly related to the core information of the target code, forming verification-type RAG recall information (i.e., second reference knowledge).

[0062] This application breaks away from the traditional static model of RAG (Resource Acquisition Group) where a single search is reused throughout the entire process. The initial RAG uses natural language input as its core to achieve comprehensive knowledge retrieval, providing a foundation for code generation. The secondary RAG, on the other hand, uses the code-generated summary as the retrieval basis to trace back to the knowledge base information. That is, compared to the initial RAG search, the search terms in this step change from secondary key information to code summaries, and the search target changes from supporting code generation to supporting code verification. This dynamic weight adjustment makes the retrieval results of the secondary RAG more closely aligned with the actual logic of code generation, rather than simply matching the user's natural language input, providing a precise comparison benchmark for subsequent code verification.

[0063] Therefore, in order to improve the accuracy of code verification, in some embodiments, the above-mentioned S140 may specifically include: In the code summary, identify the summary content corresponding to each of the multiple knowledge dimensions; Based on the weights corresponding to multiple knowledge dimensions, the number of search results corresponding to each of the multiple knowledge dimensions is determined. Based on the number of search results corresponding to each of the multiple knowledge dimensions, multiple knowledge fragments that satisfy the second similarity condition with the summary content corresponding to each of the multiple knowledge dimensions are retrieved from the knowledge base. Based on multiple knowledge fragments, a second reference knowledge is determined.

[0064] Here, if the weight of a knowledge dimension is high, the number of search results for that knowledge dimension will be large; if the weight of a knowledge dimension is low, the number of search results for that knowledge dimension will be small. For example, if the weight accounts for 80%, the number of searches for that knowledge dimension can be 80; if the weight accounts for 20%, the number of searches for that knowledge dimension can be 20.

[0065] As an example, in the description above, when the large model outputs the quantized weights corresponding to multiple knowledge dimensions, it can also simultaneously output the summary content corresponding to each of the multiple knowledge dimensions in the code summary. Thus, after determining the number of search results corresponding to each of the multiple knowledge dimensions, multiple knowledge fragments that satisfy the second similarity condition with the summary content corresponding to each of the multiple knowledge dimensions can be retrieved based on the storage format of the knowledge in the knowledge base and the number of search results corresponding to each of the multiple knowledge dimensions.

[0066] As a more concrete example, if the knowledge base is stored in text format, then for each summary, it can be matched and filtered against multiple knowledge texts in the knowledge base based on preset matching rules (such as keyword matching and regular expression matching). If the number of search results is denoted as N, then the N knowledge texts with higher matching degrees can correspond to N second knowledge texts, and the reference knowledge corresponding to these N second knowledge texts can be N knowledge fragments. The knowledge fragments corresponding to multiple summary contents can be determined as the second reference knowledge corresponding to the code summary.

[0067] As a more concrete example, if the knowledge base is stored in vector form, then for each summary content, the summary content can first be converted into a third vector, and then the second similarity between this third vector and the knowledge vector can be calculated. If the number of search results is denoted as N, then the knowledge vectors corresponding to the N largest second similarities can be considered as second knowledge vectors, and the reference knowledge corresponding to the N second knowledge vectors can be considered as N knowledge fragments. The knowledge fragments corresponding to multiple summary contents can be determined as the second reference knowledge corresponding to the code summary.

[0068] Of course, if multiple knowledge items in the knowledge base are stored in both vector and text formats, the vector matching results and text matching results can be combined to retrieve multiple knowledge fragments in the knowledge base that satisfy the second similarity condition for the summary content corresponding to each of the multiple knowledge dimensions. Specifically, for the summary content corresponding to each knowledge dimension, assuming that the number of corresponding search results is N, multiple candidate knowledge vectors can be obtained first through vector matching, and multiple candidate knowledge texts that satisfy the matching condition for the summary content can be obtained through text matching. Then, the reference knowledge corresponding to the candidate knowledge vectors and the reference knowledge corresponding to the candidate knowledge texts are fused to obtain a fused candidate reference knowledge set. Finally, the candidate reference knowledge set is sorted according to a preset fusion sorting rule, and the top N reference knowledge items are selected as the N knowledge fragments that satisfy the second similarity condition for the summary content.

[0069] The candidate reference knowledge set is sorted according to a preset fusion and sorting rule, and the top N reference knowledge is selected. This can be achieved in any of the following ways: 1) Merge the reference knowledge corresponding to the candidate knowledge vector and the reference knowledge corresponding to the candidate knowledge text, and then uniformly sort all reference knowledge based on matching score and similarity score, selecting the top N as knowledge fragments; 2) Based on a weighted fusion algorithm, comprehensively score, deduplicate, and sort the candidate knowledge vector and candidate knowledge text, selecting the top N reference knowledge with the highest comprehensive score as knowledge fragments; 3) First, perform preliminary screening of candidate knowledge text through text matching, then perform vector matching and sorting on the knowledge vectors corresponding to the screened candidate knowledge text, and select the top N corresponding reference knowledge as knowledge fragments based on vector similarity sorting.

[0070] In addition, in the embodiments of this application, when retrieving the knowledge fragments corresponding to each summary content, the retrieval is not based solely on the isolated summary content, but rather takes into account the role and importance of the summary content in the entire code summary.

[0071] This application embodiment decouples code summaries by knowledge dimensions and dynamically allocates retrieval resources based on the weight of each dimension, realizing the transformation from global retrieval to dimension-based precise recall. This ensures that the secondary retrieval results closely revolve around the core knowledge dimensions and potential deviations in code generation, providing highly focused and targeted knowledge basis for subsequent code verification, thereby improving the accuracy of code verification.

[0072] In some embodiments, in S150, for key knowledge dimensions with higher weights, since these key knowledge dimensions are prioritized in the secondary RAG, the second reference knowledge is richer than the first knowledge summary for these key knowledge dimensions. Therefore, for these key knowledge dimensions, the differential knowledge corresponding to these key knowledge dimensions can be the additional reference knowledge provided by the second reference knowledge compared to the first knowledge summary. Thus, for each key knowledge dimension, the differential knowledge corresponding to each key knowledge dimension can be obtained by comparing the differences between the first knowledge summary and the second reference knowledge corresponding to that key knowledge dimension.

[0073] In some embodiments, in S160, for the differential knowledge corresponding to each key knowledge dimension, if the differential knowledge does not include the knowledge used in the code summary, it can be determined that the differential knowledge does not match the code summary; if the differential knowledge includes the knowledge used in the code summary, it can be determined that the differential knowledge matches the code summary. For example, if the total_trade_num field does not exist in the differential knowledge representation indicator statistics table, but the code summary explicitly states that the target code uses SUM(total_trade_num) to calculate the total number of transactions this month, then the two do not match. Furthermore, if there is a discrepancy between the differential knowledge corresponding to a certain key knowledge dimension and the code summary, the final matching relationship can be determined to be a mismatch.

[0074] In some embodiments, in S170, if the discrepancies between the knowledge and code summaries corresponding to multiple knowledge dimensions do not match, the verification result can be determined as verification failure, and the reason for failure is the illusion of a large model. If the discrepancies between the knowledge and code summaries corresponding to multiple knowledge dimensions match, the inclusion relationship between the second reference knowledge and the knowledge required by the user intent can be further determined. If the second reference knowledge does not fully include the knowledge required by the user intent, the verification result can be determined as verification failure, and the reason for failure is the compromise of intent due to insufficient knowledge coverage. If the second reference knowledge fully includes the knowledge required by the user intent, the verification result can be determined as verification success.

[0075] For example, if the code summary is "SQL code calculates the total number of transactions this month using SUM(total_trade_num), but uses a total_trade_num field that does not exist in the indicator statistics table," then the verification result is confirmed as a failure. The reason for the failure is that the generated SQL code used a total_trade_num field that does not exist in the indicator statistics table (insufficient data coverage, no total transaction number field), which is a large model generation illusion. The code cannot be executed and does not meet the core intent of querying the total number of transactions. If the code summary is "SQL code calculates the total number of transactions this month using SUM(online_trade_num), but only counts online transactions and does not include offline transaction data," then the verification result is confirmed as a failure. The reason for the failure is that the code only counts online transactions and does not include offline transactions (insufficient data coverage, no offline transaction data), and cannot fully realize the intent of querying the total number of transactions. In addition, verification conclusions can be attached showing that the logic of time filtering, table name matching, etc. in the code conforms to the specifications and that the statistical logic of online transaction count is correct.

[0076] Therefore, in order to improve the user experience and the accuracy of the generated code, in some embodiments, after obtaining the verification result, the method may further include: If the verification result is a verification failure and the reason for the failure is large model illusion, add optimization suggestions to the verification result. The optimization suggestions are used to prompt the supplementation of missing knowledge in the knowledge base. If the verification result is a failure, and the reason for the failure is a compromise of intent due to insufficient knowledge coverage, add a result description and optimization suggestions to the verification result. The result description is used to indicate the difference between the target code and the user's intent.

[0077] Here, based on the verification results, further targeted optimization suggestions can be provided. In the examples above, whether it's the large model generating illusions or the compromised generated code, the root cause lies in the contradiction between the query requirement for "total number of transactions" and the data that "the indicator statistics table only stores the number of online transactions." Therefore, optimization suggestions could include, for example, supplementing the data source for offline transaction counts and adding an `offline_trade_num` field if a query for the total number of transactions is required. Furthermore, for the compromised generated code mentioned above, the verification results can be annotated with "Only online transaction counts are counted" to highlight the discrepancy between the target code and the user's intent.

[0078] This application embodiment improves user experience and the accuracy of subsequent code generation by providing targeted feedback and suggestions for different reasons for verification failure, as well as providing users with result explanations and improvement paths, thus achieving a closed-loop upgrade from simple error reporting to problem diagnosis and knowledge iteration guidance.

[0079] Furthermore, to further improve the accuracy of the verification results, in some embodiments, the above-mentioned S170 may specifically include: Based on natural language input, second key information, and first knowledge summary, determine user intent; Based on the matching relationship and user intent, the target code is verified to obtain the verification result.

[0080] Here, before matching the matching relationship with the user intent, the code generation and verification module can first read the natural language input, second key information, and first knowledge summary of the current session from the session memory module. Then, based on the natural language input, second key information, and first knowledge summary, the user intent is determined, which improves the accuracy of the user intent. Thus, by verifying the target code based on the matching relationship and the user intent, the accuracy of the verification result can be further improved.

[0081] Based on the above embodiments, a complete code verification example is provided.

[0082] As an example, after generating the target code, the code verification phase can begin. Code verification can be performed by a large model. Specifically, the code generation and verification system can send clear verification task prompts to the large model, clearly defining the verification objectives (such as determining whether the target code matches the user's intent), verification dimensions (such as business logic consistency, syntax compliance, and knowledge reference matching), and output requirements (such as conclusions on whether the output matches and analysis of the reasons for deviations), allowing the large model to clearly understand its verification role and task boundaries.

[0083] Following the example above, the verification task prompt could be something like: Verify whether the generated SQL code matches the user intent of "querying the total number of user transactions this month," and determine whether the code is generated based on the existing knowledge base's indicator statistics table structure. Verification dimensions: 1. Consistency between code logic and the question intent; 2. Whether the fields used in the code exist in the indicator statistics table; 3. Whether data coverage meets the query requirements. Output requirements: Clearly state the conclusion of "match / not match / partial match." If there is a discrepancy, analyze the specific reasons (e.g., fictitious fields, missing data, logical errors).

[0084] In addition, the code generation and verification system can also read the natural language input, second key information and first knowledge summary of the current session from the session memory module, and introduce them as supplementary knowledge for verification task prompts. At the same time, it can perform structured parsing on these data to extract key content such as core business requirements and knowledge constraints, providing complete contextual support for the verification process of large models and avoiding verification bias caused by missing information.

[0085] Ultimately, by verifying task prompts, first knowledge summary, second reference knowledge, code summary, and user intent... Figure 1 The data is then sent to the large model, which can compare the consistency of the knowledge dimensions of the first knowledge summary and the second reference knowledge, and determine whether the target code deviates from the user's intent. If a deviation exists, the large model can further analyze the reasons for the deviation (such as low matching degree of RAG recall information, or illusion generated by the large model), and output targeted verification conclusions and optimization suggestions.

[0086] The embodiments of this application can ultimately return the target code to the user, as well as the verification results including the reasons for failure, the explanation of the results, and optimization suggestions. It not only presents the generated results based on existing data, but also clearly points out the problem of insufficient data coverage, and provides feasible optimization directions, thus solving the core needs of code generation and intent matching verification in scenarios with missing data.

[0087] To better describe the entire solution, a complete process of this application is introduced based on the above embodiments.

[0088] This application employs a pipelined architecture encompassing parallel preprocessing, parallel generation and storage, and serial deep verification. It achieves a closed-loop process from knowledge retrieval and code generation to intent matching verification, centered around the user's natural language input (such as user questions). By combining Retrievable Vector Retrieval (RAG), large model capabilities, and attention mechanisms, it improves processing efficiency while mitigating the risk of result bias caused by the illusion of large model generation. Notably, the two RAG retrievals during code generation and verification stages have significantly different core logics and application objectives, making them crucial design elements for ensuring verification effectiveness.

[0089] Specifically, in the data preprocessing stage (multi-task parallel execution), the core objective is to complete the three-stage preprocessing of user queries: knowledge retrieval, information vectorization, and content summarization. By executing in parallel, the preprocessing time is minimized, providing high-quality knowledge input and data support for subsequent generation and verification.

[0090] In the code generation and memory storage phase (dual tasks executed in parallel), based on the RAG summary text (i.e., the first knowledge summary) and the question key information vector (i.e., the vector corresponding to the second key information) from the preprocessing phase, two major tasks—code generation and dialogue memory construction—were completed in parallel. While generating the target code, a complete contextual information carrier was built for the subsequent verification stage. This phase, relying on the initial RAG retrieval results from the preprocessing phase to drive code generation, is the core component of the first RAG application.

[0091] The code deep verification phase (serialized progressive execution) is the core risk control step of the solution. It completes deep verification of task declaration, context introduction, and attention mechanisms through a sequential, progressive approach. Based on a secondary customized RAG retrieval, it compares the initial RAG results with the secondary RAG results to effectively identify deviations between the generated code and the intended query. This secondary RAG supplements and verifies the initial RAG, and exhibits multiple core differences from it.

[0092] By constructing a serial verification system with dual RAG knowledge dimension comparison as its core, unlike traditional single syntax verification or rule verification, the verification logic is no longer limited to whether the code is correct, but delves into whether the code matches the user's intent and whether the code is generated based on facts. It objectively realizes the dual verification of the reliability of the code generated by the large model from the factual basis to the user's intent from the root, forming a quality closed loop for the generation of large model code.

[0093] In addition, this application constructs an end-to-end intent matching closed loop from user question input to code verification output. Each step in the process revolves around user intent. The RAG retrieval in the preprocessing stage, the prompt word construction in the generation stage, and the double RAG comparison in the verification stage all take matching the user's core needs as the core objective, forming a full-link control from intent extraction to intent support to intent verification.

[0094] In this way, by deeply integrating the generation capabilities of large models with the retrieval capabilities of RAG and the focusing capabilities of attention mechanisms, a two-way interaction is achieved where generation depends on knowledge and verification feeds back into generation. This effectively reduces the problem of unreliable generated code caused by knowledge gaps or illusions in large models, and improves the reliability and accuracy of code generation results.

[0095] Based on the verification method for large model generation code provided in the above embodiments, this application also provides a specific implementation of a verification device for large model generation code. Please refer to the following embodiments.

[0096] like Figure 2 As shown, the verification device 200 for large model generation code provided in one embodiment of this application includes the following modules: The acquisition module 210 is used to acquire the target code generated by the large model based on natural language input and the first knowledge summary. The first knowledge summary includes multiple knowledge dimensions obtained by analyzing and summarizing the first reference knowledge. The first reference knowledge is the knowledge in the knowledge base that satisfies the first similarity condition with the natural language input. The generation module 220 is used to perform syntax parsing and logical analysis on the target code and generate a code summary corresponding to the target code. Module 230 is used to determine the weights of multiple knowledge dimensions based on their importance in the code summary. The retrieval module 240 is used to retrieve second reference knowledge in the knowledge base that satisfies the second similarity condition with the code summary based on the weights corresponding to multiple knowledge dimensions. The determination module 230 is also used to compare the first knowledge summary and the second reference knowledge corresponding to each key knowledge dimension to determine the difference knowledge corresponding to each key knowledge dimension. The key knowledge dimension is the knowledge dimension with a weight greater than a preset threshold among multiple knowledge dimensions. Module 230 is also used to determine the matching relationship between the differential knowledge and code summary corresponding to multiple key knowledge dimensions; The verification module 250 is used to verify the target code based on the matching relationship and the user intent corresponding to the natural language input, and obtain the verification result.

[0097] The verification device 200 for the aforementioned large model generation code is described in detail below: In some embodiments, the generation module 220 may specifically include: Extraction submodules are used to perform syntax parsing and logical analysis of the target code, extracting the first key information from the target code; The generation submodule is used to generate a code summary based on the first key information.

[0098] In some embodiments, the retrieval module 240 may specifically include: The determination submodule is used to determine the summary content corresponding to multiple knowledge dimensions in the code summary; The submodule is also used to determine the number of search results corresponding to each of the multiple knowledge dimensions based on the weights corresponding to each of the multiple knowledge dimensions. The retrieval submodule is used to retrieve multiple knowledge fragments in the knowledge base that satisfy the second similarity condition with the summary content corresponding to the multiple knowledge dimensions, based on the number of retrieval results corresponding to the multiple knowledge dimensions respectively. The determination submodule is also used to determine the second reference knowledge based on multiple knowledge fragments.

[0099] In some embodiments, the verification module 250 may specifically include: The submodule is also used to determine the verification result as verification failure when the discrepancies between the knowledge and code summary corresponding to multiple knowledge dimensions do not match, and the reason for failure is large model illusion; The submodule is also used to determine the inclusion relationship between the second reference knowledge and the knowledge required by the user intent when the difference knowledge and code summary corresponding to multiple knowledge dimensions are matched respectively. The determination submodule is also used to determine the verification result as verification failure when the second reference knowledge does not fully contain the knowledge required by the user's intent, and the reason for the failure is the compromise of intent due to insufficient knowledge coverage.

[0100] In some embodiments, the verification device 200 for large model generated code may further include: Add a module to add optimization suggestions to the verification result when the verification result is a verification failure and the reason for failure is large model illusion. The optimization suggestions are used to prompt the missing knowledge to be added to the knowledge base. The module also adds a result description and optimization suggestions to the verification result when the verification result is a failure and the reason for the failure is a compromise of intent due to insufficient knowledge coverage. The result description is used to indicate the difference between the target code and the user intent.

[0101] In some embodiments, the acquisition module 210 may specifically include: The extraction submodule is also used to extract key information from natural language input to obtain second key information; The retrieval submodule is also used to retrieve first reference knowledge from the knowledge base that satisfies the first similarity condition to the second key information; The sub-module is also used to analyze and summarize the first reference knowledge, identify multiple knowledge dimensions and their corresponding knowledge, and obtain the first knowledge summary. The generation submodule is also used to generate target code based on natural language input, first knowledge summary, and code generation specifications.

[0102] In some embodiments, the determining module 230 may specifically include: The determination submodule is also used to analyze and summarize the first reference knowledge based on natural language input, determine multiple knowledge dimensions and their corresponding reference knowledge, as well as the missing knowledge of the first reference knowledge relative to the natural language input, and obtain the first knowledge summary.

[0103] In some embodiments, the verification module 250 may specifically include: The determination submodule is also used to determine user intent based on natural language input, second key information, and first knowledge summary; The verification submodule is used to verify the target code based on the matching relationship and user intent, and obtain the verification result.

[0104] This application embodiment parses and logically organizes the target code generated by a large model to generate a code summary corresponding to the target code. Based on the importance of multiple knowledge dimensions in the code summary, it determines the weights corresponding to each knowledge dimension, quantifying the key knowledge dimensions relied upon in generating the target code. By retrieving second reference knowledge from the knowledge base that satisfies the second similarity condition to the target code based on the weights corresponding to each knowledge dimension, it can supplement and improve the knowledge corresponding to the key knowledge dimensions. Since both the first and second reference knowledge are objective knowledge retrieved from the knowledge base, by comparing the first knowledge summary and the second reference knowledge corresponding to each key knowledge dimension, it determines the difference knowledge corresponding to each key knowledge dimension. Based on the matching relationship between the difference knowledge corresponding to each key knowledge dimension and the code summary, and the user intent corresponding to the natural language input, it verifies the target code. Based on the supplemented and improved knowledge, it can objectively determine from the root whether the target code is generated based on facts and whether it conforms to user intent, thereby achieving a dual verification of the reliability of the code generated by the large model from both factual basis and user intent.

[0105] Based on the verification method for large model generation code provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Figure 3 A schematic diagram of the structure of an electronic device provided in one embodiment of this application is shown.

[0106] like Figure 3 As shown, the electronic device 300 may include a processor 310 and a memory 320 storing computer program instructions.

[0107] Specifically, the processor 310 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0108] Memory 320 may include mass storage for data or instructions. For example, and not limitingly, memory 320 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where suitable, memory 320 may include removable or non-removable (or fixed) media. Where suitable, memory 320 may be internal or external to electronic device 300. In a particular embodiment, memory 320 is a non-volatile solid-state memory.

[0109] In specific embodiments, the memory 320 may be implemented as a read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 320 may store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 320 and executed by the processor 310. The processor 310 reads and executes the computer program instructions stored in the memory 320 to implement any of the large model generation code verification methods in the above embodiments.

[0110] The processor 310 reads and executes computer program instructions stored in the memory 320 to implement any of the large model generation code verification methods in the above embodiments.

[0111] In one example, electronic device 300 may further include communication interface 330 and bus 340. Wherein, for example... Figure 3 As shown, the processor 310, memory 320, and communication interface 330 are connected via bus 340 and communicate with each other.

[0112] The communication interface 330 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0113] Bus 340 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 340 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0114] For example, the electronic device 300 can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0115] The electronic device can execute the verification method of the large model generation code in the embodiments of this application, thereby achieving... Figure 1 The verification method for generating large model code is described, and the beneficial effects of the corresponding method implementation are not elaborated here.

[0116] Furthermore, in conjunction with the verification method for large model generation code in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the large model generation code verification methods in the above embodiments. Examples of such computer-readable storage media include non-transitory computer-readable storage media, such as read-only memory (ROM).

[0117] The computer program instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the verification method of the large model generation code as shown in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0118] Based on the verification methods for large model generation code in the above embodiments, this application provides a computer program product for implementation. When the instructions in this computer program product are executed by the processor of an electronic device, they implement any of the large model generation code verification methods in the above embodiments.

[0119] The computer program products of the above embodiments are used to implement the verification method of large model generation code as shown in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0120] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0121] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0122] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0123] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0124] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for verifying code generated from a large model, characterized in that, include: Obtain the target code generated by the large model based on natural language input and a first knowledge summary. The first knowledge summary includes multiple knowledge dimensions obtained by analyzing and summarizing the first reference knowledge. The first reference knowledge is knowledge in the knowledge base that satisfies the first similarity condition with the natural language input. The target code is parsed and logically analyzed to generate a code summary corresponding to the target code. Based on the importance of the multiple knowledge dimensions in the code summary, the weights corresponding to the multiple knowledge dimensions are determined respectively; Based on the weights corresponding to the multiple knowledge dimensions, second reference knowledge that satisfies the second similarity condition with the code summary is retrieved from the knowledge base. By comparing the first knowledge summary and the second reference knowledge corresponding to each key knowledge dimension, the differential knowledge corresponding to each key knowledge dimension is determined. The key knowledge dimension is the knowledge dimension whose weight is greater than a preset threshold among the multiple knowledge dimensions. Determine the matching relationship between the differential knowledge corresponding to each of the multiple key knowledge dimensions and the code summary; Based on the matching relationship and the user intent corresponding to the natural language input, the target code is verified to obtain the verification result.

2. The method according to claim 1, characterized in that, The step of performing syntax parsing and logical analysis on the target code to generate a code summary corresponding to the target code includes: The target code is parsed and its logic is analyzed to extract the first key information of the target code; Based on the first key information, the code digest is generated.

3. The method according to claim 1, characterized in that, The step of retrieving second reference knowledge from the knowledge base that satisfies the second similarity condition to the code digest, based on the weights corresponding to the multiple knowledge dimensions, includes: In the code summary, determine the summary content corresponding to each of the multiple knowledge dimensions; Based on the weights corresponding to the multiple knowledge dimensions, the number of search results corresponding to each of the multiple knowledge dimensions is determined. Based on the number of search results corresponding to the multiple knowledge dimensions, multiple knowledge fragments that satisfy the second similarity condition with the summary content corresponding to the multiple knowledge dimensions are retrieved in the knowledge base. Based on the aforementioned multiple knowledge fragments, the second reference knowledge is determined.

4. The method according to claim 1, characterized in that, The step of verifying the target code based on the matching relationship and the user intent corresponding to the natural language input, and obtaining the verification result, includes: If the discrepancies between the knowledge corresponding to the multiple knowledge dimensions and the code summary do not match, the verification result is determined to be a verification failure, and the reason for the failure is the illusion of a large model. When the differential knowledge corresponding to the multiple knowledge dimensions matches the code summary, the inclusion relationship between the second reference knowledge and the knowledge required by the user intent is determined. If the second reference knowledge does not fully contain the knowledge required for the user's intent, the verification result is determined to be a verification failure, and the reason for the failure is a compromise of intent due to insufficient knowledge coverage.

5. The method according to claim 4, characterized in that, After obtaining the verification result, the method further includes: If the verification result is a verification failure and the reason for the failure is a large model illusion, an optimization suggestion is added to the verification result. The optimization suggestion is used to prompt the supplementation of missing knowledge in the knowledge base. If the verification result is a verification failure, and the reason for the failure is a compromise of intent due to insufficient knowledge coverage, add a result description and the optimization suggestion to the verification result. The result description is used to indicate the difference between the target code and the user intent.

6. The method according to claim 1, characterized in that, The acquisition of the target code of the large model, generated based on natural language input and the first knowledge summary, includes: The natural language input is subjected to key information extraction to obtain the second key information; Retrieve the first reference knowledge from the knowledge base that satisfies the first similarity condition to the second key information; The first reference knowledge is analyzed and summarized to determine the multiple knowledge dimensions and their corresponding knowledge, thereby obtaining the first knowledge summary. The target code is generated based on the natural language input, the first knowledge summary, and the code generation specification.

7. The method according to claim 6, characterized in that, The step of analyzing and summarizing the first reference knowledge to determine the multiple knowledge dimensions and their corresponding reference knowledge, and obtaining a first knowledge summary, includes: Based on the natural language input, the first reference knowledge is analyzed and summarized to determine the multiple knowledge dimensions and their respective corresponding reference knowledge, as well as the missing knowledge of the first reference knowledge relative to the natural language input, thereby obtaining the first knowledge summary.

8. The method according to claim 6, characterized in that, The step of verifying the target code based on the matching relationship and the user intent corresponding to the natural language input, and obtaining the verification result, includes: The user intent is determined based on the natural language input, the second key information, and the first knowledge summary; Based on the matching relationship and the user intent, the target code is verified to obtain the verification result.

9. A verification device for large model generated code, characterized in that, The device includes: The acquisition module is used to acquire the target code generated by the large model based on natural language input and a first knowledge summary. The first knowledge summary includes multiple knowledge dimensions obtained by analyzing and summarizing the first reference knowledge. The first reference knowledge is knowledge in the knowledge base that satisfies the first similarity condition with the natural language input. The generation module is used to perform syntax parsing and logical analysis on the target code and generate a code summary corresponding to the target code. The determination module is used to determine the weights corresponding to the multiple knowledge dimensions respectively based on their importance in the code digest; The retrieval module is used to retrieve second reference knowledge that satisfies the second similarity condition with the code summary in the knowledge base based on the weights corresponding to the multiple knowledge dimensions respectively. The determining module is further configured to compare the first knowledge summary and the second reference knowledge corresponding to each key knowledge dimension to determine the difference knowledge corresponding to each key knowledge dimension, wherein the key knowledge dimension is the knowledge dimension with a weight greater than a preset threshold among the multiple knowledge dimensions; The determining module is further configured to determine the matching relationship between the differential knowledge corresponding to the multiple key knowledge dimensions and the code summary; The verification module is used to verify the target code based on the matching relationship and the user intent corresponding to the natural language input, and obtain the verification result.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the verification method for the large model generation code as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the verification method for the large model generation code as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the verification method for the large model generation code as described in any one of claims 1-8.

Images