Evaluation index generation method oriented to scene self-adaption
By performing semantic analysis and code generation on the text describing the requirements for evaluation indicators, cross-scenario compatible evaluation indicators are generated, solving the problem of low efficiency in evaluation indicator generation and achieving efficient evaluation indicator code generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-04-10
AI Technical Summary
In existing technologies, the evaluation indicators are difficult to use directly across different scenarios, have low generation efficiency, and cannot meet the complex and ever-changing application needs.
By performing semantic analysis on the text describing the evaluation criteria requirements of the target scenario, a global feature vector is generated, the target formula template and atomic indicator set are predicted, the formula is completed and converted into executable code, and the code is verified to generate evaluation criteria.
It achieves cross-scenario compatibility of evaluation indicator generation code, reduces manual intervention, and improves generation efficiency.
Smart Images
Figure CN121832956A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and in particular to a scene-adaptive evaluation index generation method. BACKGROUND
[0002] Evaluation indexes are mainly used for quantitative evaluation of system performance or behavior effect, and are widely used in various application scenarios such as user interest recommendation, enterprise employee management, teaching quality evaluation, medical effect evaluation, and financial risk early warning. By designing and developing evaluation indexes of different categories and levels, an automatic evaluation system can estimate the simulation effect, link or detail that interests the user.
[0003] Currently, evaluation indexes are generally constructed based on the construction method of atomic indexes. Developers need to select relevant atomic indexes from an atomic index library based on understanding of user evaluation requirements, design reasonable calculation formulas, and manually develop calculation codes for the atomic indexes. However, as application scenarios become increasingly complex and diversified, users have higher requirements for evaluation indexes, which need to accurately and comprehensively cover all aspects of evaluation requirements. However, different application scenarios and different users often have great differences in content and definition of the required evaluation index system, making it difficult for existing evaluation index results to be directly used across scenarios, and the efficiency of manually constructing atomic indexes and developing codes is low, resulting in low efficiency of evaluation index generation. Therefore, how to provide an evaluation index generation method compatible with different scenarios and efficient is a technical problem to be solved at present. SUMMARY
[0004] The present application provides a scene-adaptive evaluation index generation method to solve the defects of existing evaluation index results that are difficult to be directly used across scenarios and low efficiency of evaluation index generation.
[0005] The present application provides a scene-adaptive evaluation index generation method, comprising the following steps.
[0006] Performing semantic analysis on the evaluation index requirement description text corresponding to the target scene to obtain a global feature vector corresponding to the evaluation index requirement description text; Based on the global feature vector, predicting a target formula template and a target atomic index set corresponding to the evaluation index requirement description text; Completing the target formula template based on the target atomic index set to obtain a completed formula; Determining executable code corresponding to the completed formula; Based on the verification result of the executable code, determining the evaluation index generation code corresponding to the evaluation index requirement description text.
[0007] The scene-oriented adaptive evaluation index generation method provided by the application carries out semantic analysis on the evaluation index demand description text corresponding to the target scene, obtains a global feature vector corresponding to the evaluation index demand description text, and comprises the following steps: A plurality of keywords in the evaluation index demand description text are determined. Each keyword is vector encoded to obtain a word vector corresponding to each keyword. Based on all word vectors and the semantic dependency relationship between the word vectors, a graph structure corresponding to the evaluation index demand description text is determined. The nodes in the graph structure represent word vectors, and the edges between the nodes represent the semantic dependency relationship between the corresponding word vectors. The word vectors corresponding to each node and neighbor node are aggregated to obtain a fusion word vector corresponding to each node. The fusion word vectors corresponding to all nodes are determined as the global feature vector corresponding to the evaluation index demand description text.
[0008] The scene-oriented adaptive evaluation index generation method provided by the application predicts a target formula template and a target atomic index set corresponding to the evaluation index demand description text based on the global feature vector, and comprises the following steps: Based on the global feature vector, a target formula template corresponding to the evaluation index demand description text is predicted from a formula template library. Based on the global feature vector and the number of index positions in the target formula template, a target atomic index set corresponding to the evaluation index demand description text is predicted from an atomic index library. The number of target atomic indexes in the target atomic index set is equal to the number of index positions in the target formula template.
[0009] The scene-oriented adaptive evaluation index generation method provided by the application completes the target formula template based on the target atomic index set to obtain a completed formula, and comprises the following steps: Based on a semantic rule, a target atomic index order corresponding to the target atomic index set is determined. The semantic rule is used to represent the semantic association relationship between the preset atomic indexes. Based on the target atomic index order, all target atomic indexes in the target atomic index set are filled into the corresponding index positions in the target formula template to obtain the completed formula.
[0010] The scene-oriented adaptive evaluation index generation method provided by the application determines an executable code corresponding to the completed formula, and comprises the following steps: From a preset code library, a target formula code corresponding to the completed formula and a target index code corresponding to all target atomic indexes in the completed formula are determined. Based on all target indicator codes and the target formula codes, the executable code corresponding to the completed formula is determined.
[0011] According to the scenario-adaptive evaluation index generation method provided by the present invention, determining multiple keywords in the evaluation index requirement description text includes: The text describing the requirements of the evaluation indicators is segmented into words to obtain multiple words corresponding to the text describing the requirements of the evaluation indicators. Part-of-speech analysis was performed on the multiple words to obtain multiple keywords in the text describing the requirements of the evaluation indicators.
[0012] According to the scenario-adaptive evaluation metric generation method provided by the present invention, the step of determining the evaluation metric generation code corresponding to the evaluation metric requirement description text based on the verification result of the executable code includes: If the verification result of the executable code is successful, the executable code is identified as the evaluation indicator generation code corresponding to the evaluation indicator requirement description text.
[0013] The present invention also provides a scene-adaptive evaluation index generation device, which includes the following modules.
[0014] The semantic analysis module is used to perform semantic analysis on the evaluation indicator requirement description text corresponding to the target scenario, and obtain the global feature vector corresponding to the evaluation indicator requirement description text. The prediction module is used to predict the target formula template and target atomic index set corresponding to the evaluation index requirement description text based on the global feature vector. The formula completion module is used to complete the target formula template based on the target atomic index set to obtain the completed formula. The code conversion module is used to determine the executable code corresponding to the completed formula; The verification module is used to determine the evaluation indicator generation code corresponding to the evaluation indicator requirement description text based on the verification result of the executable code.
[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the scene-adaptive evaluation index generation method as described above.
[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scene-adaptive evaluation index generation method as described above.
[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the scene-adaptive evaluation index generation method as described above.
[0018] This invention provides a scenario-adaptive evaluation index generation method. It performs semantic analysis on the evaluation index requirement description text of the target scenario to obtain a global feature vector. Based on the global feature vector, it predicts the target formula template and target atomic index set corresponding to the evaluation index requirement description text. It then completes the target formula template based on the target atomic index set to obtain a completed formula. The completed formula is converted into executable code, and the executable code is validated. Based on the validation results, the evaluation index generation code corresponding to the evaluation index requirement description text is determined. In this invention, semantic analysis intelligently deconstructs the evaluation index requirement description text, uniformly converting requirement descriptions from different scenarios into global feature vectors, eliminating differences between different scenarios. Simultaneously, it uses the global feature vectors for formula prediction, atomic index prediction, and formula completion, and automatically assembles standardized code for the completed formula, ensuring that the generated evaluation index generation code is compatible with different scenarios, reducing manual intervention in the code generation process, and improving the generation efficiency of evaluation index generation code. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating the scenario-adaptive evaluation index generation method provided in this embodiment of the invention.
[0021] Figure 2 This is a schematic diagram of the process for determining the global feature vector provided in an embodiment of the present invention.
[0022] Figure 3 This is a flowchart illustrating the prediction target formula template and target atomic index set provided in this embodiment of the invention.
[0023] Figure 4 This is a schematic diagram of the structure of the scene-adaptive evaluation index generation device provided in an embodiment of the present invention.
[0024] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0026] To address the problems of existing technologies where evaluation index results are difficult to use directly across different scenarios and where the generation efficiency of evaluation indices is low, this invention provides a scenario-adaptive evaluation index generation method. Figure 1 This is a flowchart illustrating the scenario-adaptive evaluation metric generation method provided in this embodiment of the invention, as shown below. Figure 1 As shown, the method includes steps 110 to 150.
[0027] Step 110: Perform semantic analysis on the evaluation indicator requirement description text corresponding to the target scenario to obtain the global feature vector corresponding to the evaluation indicator requirement description text.
[0028] Specifically, the evaluation indicator requirement description text is natural language text input by the user, used to describe the user's specific requirements for the evaluation system to perform quantitative evaluation. The target scenario indicates the specific field or environment in which the evaluation indicator requirement description text is applied. For example, the evaluation indicator requirement description text could be "Please calculate user A's work performance," and the corresponding target scenario could be an office automation system or a project management platform, etc.
[0029] After obtaining the text describing the evaluation indicator requirements, Natural Language Processing (NLP) technology is used to intelligently deconstruct the text, deeply understand its semantic information such as syntactic structure and contextual relationships, and encode this semantic information into a global feature vector for subsequent automatic code generation. This global feature vector is a high-dimensional vector representing the overall semantic information of the evaluation indicator requirement description text.
[0030] Step 120: Based on the global feature vector, predict the target formula template and target atomic index set corresponding to the evaluation index requirement description text.
[0031] Specifically, after determining the global feature vector corresponding to the evaluation indicator requirement description text, the target formula template that is highly likely to be contained in the evaluation indicator requirement description text, and the set of target atomic indicators that is highly likely to be contained in the target formula template, are predicted using this global feature vector. The target formula template is an abstract mathematical expression framework with specific operational relationships; the indicator positions in the target formula template are blank and their specific meanings are not defined. The set of target atomic indicators includes multiple target atomic indicators involved in the evaluation indicator requirement description text, and these multiple target atomic indicators correspond to multiple undefined indicator positions in the target formula template.
[0032] Step 130: Complete the target formula template based on the target atomic index set to obtain the completed formula.
[0033] Specifically, after determining the target atomic index set, each target atomic index in the target atomic index set is filled into the corresponding index position in the target formula template to obtain the completed formula corresponding to the target formula template. At this point, the completed formula is a complete data expression with clear calculation logic and meaning.
[0034] Step 140: Determine the executable code corresponding to the completed formula.
[0035] Specifically, after obtaining the completed formula, it is converted into executable code, which can be directly compiled by a computer to perform the calculation task corresponding to the completed formula.
[0036] Optionally, the executable code may follow the syntax of Python, Java, etc., and the embodiments of the present invention do not limit this.
[0037] Step 150: Based on the verification results of the executable code, determine the evaluation indicator generation code corresponding to the evaluation indicator requirement description text.
[0038] Specifically, after determining the executable code, the correctness of the code syntax and logic is verified through the static code analysis verification mechanism to obtain the verification result of the executable code. Based on the verification result, the evaluation indicator generation code corresponding to the evaluation indicator requirement description text is determined.
[0039] The scenario-adaptive evaluation index generation method provided in this invention performs semantic analysis on the evaluation index requirement description text of the target scenario to obtain a global feature vector. Based on the global feature vector, it predicts the target formula template and target atomic index set corresponding to the evaluation index requirement description text. It then completes the target formula template based on the target atomic index set to obtain a completed formula. The completed formula is converted into executable code, and the executable code is validated. Based on the validation results, the evaluation index generation code corresponding to the evaluation index requirement description text is determined. In this invention, semantic analysis is used to intelligently deconstruct the evaluation index requirement description text, uniformly converting the requirement descriptions of different scenarios into a global feature vector, eliminating differences between different scenarios. Simultaneously, the global feature vector is used for formula prediction, atomic index prediction, and formula completion. The completed formula is then automatically assembled into standardized code, ensuring that the generated evaluation index generation code is compatible with different scenarios, reducing manual intervention in the code generation process, and improving the generation efficiency of the evaluation index generation code.
[0040] In one embodiment, Figure 2 This is a flowchart illustrating the process of determining the global feature vector provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the semantic analysis of the evaluation indicator requirement description text corresponding to the target scenario to obtain the global feature vector corresponding to the evaluation indicator requirement description text includes: Identify multiple keywords in the text describing the requirements for the evaluation indicators; Each keyword is vector-encoded to obtain the word vector corresponding to each keyword; Based on all word vectors and the semantic dependencies between word vectors, the graph structure corresponding to the evaluation index requirement description text is determined; the nodes in the graph structure represent word vectors, and the edges between nodes represent the semantic dependencies between the corresponding word vectors. By aggregating the word vectors corresponding to each node and its neighboring nodes, a fused word vector corresponding to each node is obtained. The fusion word vectors corresponding to all nodes are determined as the global feature vectors corresponding to the evaluation indicator requirement description text.
[0041] Specifically, after obtaining the text describing the requirements for the evaluation indicators, multiple keywords with practical significance are extracted from this text. After obtaining these keywords, word vector encoding technology is used to map them into a continuous vector space, converting each keyword into a numerical word vector. This word vector is used to quantify the semantic information of the keyword; for example, the word vector corresponding to the keyword "User A" is [010001]. Word vectors with similar semantics are closer together in the vector space.
[0042] Next, dependency parsing is used to analyze the dependency relationships between multiple word vectors, thereby determining the semantic dependencies between them. These dependency relationships can include subject-verb, verb-object, attributive-head, or adverbial-head relationships, etc. The semantic dependency relationship characterizes the subordinate relationship between multiple word vectors, and the degree of dependency can be determined based on the dependency relationship between them. For example, the dependency relationship between "calculate" and "user A" is a verb-object relationship, thus determining the semantic dependency relationship between "calculate" and "user A" as: "user A" depends on "calculate." Since the weight of the verb-object relationship is 0.8, the degree of dependency between "calculate" and "user A" is determined to be 0.8. Based on the word vectors, semantic dependencies, and degree of dependency obtained from the above analysis, the graph structure corresponding to the requirement description text is evaluated. That is, word vectors are used as nodes. If there is a semantic dependency relationship between two word vectors, an edge is constructed between the nodes corresponding to these two word vectors, and the direction of the edge is determined according to the semantic dependency relationship. The degree of dependency between these two word vectors is then determined as the value of the edge. For example, the semantic dependency between "computation" and "user A" is: "user A" depends on "computation". Therefore, the direction of the edge between these two nodes is from node "computation" to node "user A", and the edge value is set to 0.8. Repeating the above steps yields a graph structure composed of all word vectors. This graph structure is a directed graph consisting of nodes and directed edges.
[0043] After constructing the graph structure, for each node in the graph, a pre-trained graph representation learning model is used to aggregate the word vectors corresponding to that node and its corresponding neighboring nodes, resulting in a fused word vector for that node. This fused word vector integrates the semantic information of the node itself and its neighboring nodes, enabling better capture of global dependencies when predicting target formula templates and target atomic indicator sets. This avoids interpreting word vectors in isolation, solves the problem of missing local semantics, and improves the accuracy of subsequent predictions. Finally, the fused word vectors corresponding to all nodes are concatenated to obtain the global feature vector.
[0044] Optionally, the graph representation learning module can be a graph neural network (GNN) or a graph convolutional network (GCN), etc., and the embodiments of the present invention do not limit this.
[0045] Optionally, the word vector encoding technology can be Word2Vec, but this embodiment of the invention does not limit it.
[0046] In one embodiment, determining multiple keywords in the evaluation indicator requirement description text includes: The text describing the requirements of the evaluation indicators is segmented into words to obtain multiple words corresponding to the text describing the requirements of the evaluation indicators. Part-of-speech analysis was performed on the multiple words to obtain multiple keywords in the text describing the requirements of the evaluation indicators.
[0047] Specifically, after obtaining the text describing the requirements of the evaluation indicators, the text is segmented into multiple independent words using a statistical word segmentation method. Then, the part of speech of each word is identified, and redundant words such as meaningless honorifics and auxiliary words are filtered out to obtain multiple keywords with practical meaning. These keywords can be adverbs, adjectives, verbs, and nouns, etc., but this embodiment of the invention does not limit them.
[0048] For example, taking the evaluation indicator requirement description text as "Please calculate the work execution efficiency of user A" as an example, the text is segmented into words, resulting in six words: "please", "calculate", "user A", "of", "work", "execution", and "efficiency". Then, the parts of speech of each word are identified: "please" is a polite word, "calculate" is a verb, "user A" is a noun, "of" is a particle, "work" is a noun, "execution" is a verb, and "efficiency" is a noun. After removing the polite word "please" and the particle "of" which have no practical meaning, "calculate", "user A", "work", "execution", and "efficiency" are all identified as keywords.
[0049] In one embodiment, Figure 3 This is a flowchart illustrating the prediction target formula template and target atomic index set provided in this embodiment of the invention, as shown below. Figure 3 As shown, the step of predicting the target formula template and target atomic index set corresponding to the evaluation index requirement description text based on the global feature vector includes: Based on the global feature vector, predict the target formula template corresponding to the evaluation index requirement description text from the formula template library; Based on the global feature vector and the number of indicator positions in the target formula template, predict the set of target atomic indicators corresponding to the evaluation indicator requirement description text from the atomic indicator library; the number of target atomic indicators in the target atomic indicator set is equal to the number of indicator positions in the target formula template.
[0050] Specifically, after extracting the global feature vector, this global feature vector is input into a pre-trained formula template prediction model and an atomic index prediction model, respectively. The formula template prediction model classifies the global feature vector and outputs a template prediction probability vector. The length of this vector is equal to the number of preset formula templates in the formula template library, and each element in the vector represents the probability of formula mapping between the global feature vector and the corresponding preset formula template. Then, all formula mapping probabilities are sorted in descending order, and the preset formula template with the highest probability is determined as the target formula template corresponding to the evaluation index requirement description text. For example, this target formula template could be f=A / (B*C), where A, B, and C represent undefined index positions in the target formula template.
[0051] Simultaneously, the global feature vector is classified using an atomic indicator prediction model, outputting an indicator prediction probability vector. The length of this probability vector is equal to the number of preset atomic indicators in the atomic indicator library, and each element represents the indicator mapping probability between the global feature vector and the corresponding preset atomic indicator. Then, all indicator mapping probabilities are sorted in descending order to obtain the indicator ranking result. After determining the number of indicator positions in the target formula template, the top 3 preset atomic indicators in the indicator ranking result are identified as target atomic indicators, thus obtaining the target atomic indicator set. For example, since the target formula template has 3 indicator positions, the top three preset atomic indicators in the indicator ranking result—"Work Completion Quantity," "Work Completion Time," and "Total Work Quantity"—are identified as the three target atomic indicators in the target atomic indicator set.
[0052] It should be noted that both the formula template prediction model and the atomic index prediction model can be constructed based on deep learning algorithms, such as convolutional neural networks, recurrent neural networks, graph neural networks, etc., and the embodiments of the present invention do not limit this.
[0053] Furthermore, the formula template prediction model can be trained using sample text of evaluation indicator requirements and real formula templates. During training, the model predicts the formula template based on the sample text and updates its parameters based on the difference between the predicted and real formula templates. Similarly, the atomic indicator prediction model can be trained using sample text of evaluation indicator requirements and real atomic indicators. During training, the model predicts the atomic indicators based on the sample text and updates its parameters based on the difference between the predicted and real atomic indicators.
[0054] In one embodiment, the step of completing the target formula template based on the target atomic index set to obtain the completed formula includes: Based on semantic rules, the order of target atomic indicators corresponding to the target atomic indicator set is determined; the semantic rules are used to characterize the semantic association between preset atomic indicators. Based on the order of the target atomic indicators, all target atomic indicators in the target atomic indicator set are filled into the corresponding indicator positions in the target formula template to obtain the completed formula.
[0055] Specifically, after determining the target atomic indicator set, since the target atomic indicator set includes multiple target atomic indicators and the target formula template includes multiple indicator positions, there will be multiple indicator arrangement orders when filling multiple target atomic indicators into the target formula template. Therefore, in this embodiment of the invention, semantic rules are pre-constructed based on preset atomic indicators. These semantic rules represent the semantic association relationships between preset atomic indicators, which may include inclusion relationships, parallel relationships, mutual exclusion relationships, overlapping relationships, and independent relationships, etc. Then, based on the semantic rules and the indicator position structure in the target formula template, the target semantic association relationship corresponding to the target atomic indicator set is determined, so as to determine at least one candidate atomic indicator order from multiple indicator arrangement orders. If only one candidate atomic indicator order is determined, then the candidate atomic indicator order is determined as the target atomic indicator order. If at least two candidate atomic indicator orders are determined, a confirmation message can be generated and displayed in a pop-up window to respond to the user's operation on the confirmation message, and the target atomic indicator order is determined from at least two candidate atomic indicator orders. The target atomic indicator order represents the filling position of each target atomic indicator in the target formula template. Then, the order of the target atomic indicators is verified using prior knowledge used to characterize the domain constraint rules. That is, it is verified whether the target scenario conforms to the logic of the domain and the requirements in the evaluation indicator requirement description text. If the verification passes, each target atomic indicator is filled into the corresponding indicator position in the target formula template to obtain the complete formula.
[0056] For example, taking three target atomic indicators as "work completed quantity," "work completion time," and "total work quantity," the overall indicator structure in the target formula template is a fractional structure. Therefore, according to semantic rules, it can be determined that the target atomic indicators in the numerator and denominator of the target formula template are semantically similar, and the target semantic relationship is an inclusion relationship, that is, the target atomic indicators in the numerator are included in the target atomic indicators in the denominator. Furthermore, since there are two target atomic indicators in the denominator, and the operation relationship between the two target atomic indicators is a multiplication operation, it can be determined according to semantic rules that the target semantic relationship between the two target atomic indicators in the denominator is a parallel relationship. Then, based on the inclusion relationship between the numerator and denominator, and the parallel relationship in the denominator, it is determined that all target atomic indicators form combinations of inclusion and parallel relationships. For example, there is an inclusion relationship between the work completed quantity and the total work quantity; the total work quantity and the work completion time are parallel relationships; and the work completed quantity and the work completion time are also parallel relationships. Since there is only one inclusion relationship, the quantity of work completed is in the numerator of the target formula template, and the total quantity of work is in the denominator. Furthermore, the work completion time and the total quantity of work are parallel components. Both of these target atomic indicators can be in the denominator of the target formula template, and their order is not restricted. After filling, the completed formula can be f = quantity of work completed / (total quantity of work × work completion time).
[0057] In one embodiment, determining the executable code corresponding to the completed formula includes: Determine the target formula code corresponding to the completed formula and the target indicator code corresponding to each of the target atomic indicators in the completed formula from the preset code library; Based on all target indicator codes and the target formula codes, the executable code corresponding to the completed formula is determined.
[0058] Specifically, the preset code library includes preset indicator codes corresponding to each preset atomic indicator and preset formula codes corresponding to each preset formula template. After determining the completion formula, the target indicator codes corresponding to each target atomic indicator in the completion formula are retrieved from the preset code library, and the target formula codes corresponding to the target formula templates of the completion formula are also retrieved. By nesting all the target indicator codes into the target formula codes, the executable code corresponding to the completion formula can be obtained.
[0059] In one embodiment, determining the evaluation indicator generation code corresponding to the evaluation indicator requirement description text based on the verification result of the executable code includes: If the verification result of the executable code is successful, the executable code is identified as the evaluation indicator generation code corresponding to the evaluation indicator requirement description text.
[0060] For example, when the executable code conforms to Python syntax rules, static code analysis can be performed using the `ast` (Abstract Syntax Tree) module in Python to check for syntax errors or logical flaws. The `ast` module parses the executable code into a tree-like representation to validate its syntax, such as expressions, statements, and function definitions. After passing the validation, the executable code can be used as the final evaluation metric.
[0061] The following describes the scene-adaptive evaluation index generation device provided by the present invention. The scene-adaptive evaluation index generation device described below and the scene-adaptive evaluation index generation method described above can be referred to in correspondence.
[0062] This invention also provides a scenario-adaptive evaluation metric generation device. Figure 4 This is a schematic diagram of the scene-adaptive evaluation index generation device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the scene-adaptive evaluation index generation device 400 includes: a semantic analysis module 410, a prediction module 420, a formula completion module 430, a code conversion module 440, and a verification module 450.
[0063] The semantic analysis module 410 is used to perform semantic analysis on the evaluation indicator requirement description text corresponding to the target scenario to obtain the global feature vector corresponding to the evaluation indicator requirement description text.
[0064] The prediction module 420 is used to predict the target formula template and target atomic index set corresponding to the evaluation index requirement description text based on the global feature vector.
[0065] Formula completion module 430 is used to complete the target formula template based on the target atomic index set to obtain the completed formula.
[0066] The code conversion module 440 is used to determine the executable code corresponding to the completed formula.
[0067] The verification module 450 is used to determine the evaluation indicator generation code corresponding to the evaluation indicator requirement description text based on the verification result of the executable code.
[0068] The scenario-adaptive evaluation index generation device provided in this invention performs semantic analysis on the evaluation index requirement description text of the target scenario to obtain a global feature vector. Based on the global feature vector, it predicts the target formula template and target atomic index set corresponding to the evaluation index requirement description text. It then completes the target formula template based on the target atomic index set to obtain a completed formula. The completed formula is converted into executable code, and the executable code is validated. Based on the validation result, the evaluation index generation code corresponding to the evaluation index requirement description text is determined. In this invention, semantic analysis is used to intelligently deconstruct the evaluation index requirement description text, uniformly converting the requirement descriptions of different scenarios into a global feature vector, eliminating differences between different scenarios. Simultaneously, formula prediction, atomic index prediction, and formula completion are performed using the global feature vector, and the completed formula is automatically assembled into standardized code. This ensures that the generated evaluation index generation code is compatible with different scenarios, reduces manual intervention in the code generation process, and improves the generation efficiency of the evaluation index generation code.
[0069] Optionally, the semantic analysis module 410 is specifically used for: Identify multiple keywords in the text describing the requirements for the evaluation indicators; Each keyword is vector-encoded to obtain the word vector corresponding to each keyword; Based on all word vectors and the semantic dependencies between word vectors, the graph structure corresponding to the evaluation index requirement description text is determined; the nodes in the graph structure represent word vectors, and the edges between nodes represent the semantic dependencies between the corresponding word vectors. By aggregating the word vectors corresponding to each node and its neighboring nodes, a fused word vector corresponding to each node is obtained. The fusion word vectors corresponding to all nodes are determined as the global feature vectors corresponding to the evaluation indicator requirement description text.
[0070] Optionally, the semantic analysis module 410 is specifically used for: The text describing the requirements of the evaluation indicators is segmented into words to obtain multiple words corresponding to the text describing the requirements of the evaluation indicators. Part-of-speech analysis was performed on the multiple words to obtain multiple keywords in the text describing the requirements of the evaluation indicators.
[0071] Optionally, the prediction module 420 is specifically used for: Based on the global feature vector, predict the target formula template corresponding to the evaluation index requirement description text from the formula template library; Based on the global feature vector and the number of indicator positions in the target formula template, predict the set of target atomic indicators corresponding to the evaluation indicator requirement description text from the atomic indicator library; the number of target atomic indicators in the target atomic indicator set is equal to the number of indicator positions in the target formula template.
[0072] Optionally, this formula completion module 430 is specifically used for: Based on semantic rules, the order of target atomic indicators corresponding to the target atomic indicator set is determined; the semantic rules are used to characterize the semantic association between preset atomic indicators. Based on the order of the target atomic indicators, all target atomic indicators in the target atomic indicator set are filled into the corresponding indicator positions in the target formula template to obtain the completed formula.
[0073] Optionally, the code conversion module 440 is specifically used for: Determine the target formula code corresponding to the completed formula and the target indicator code corresponding to each of the target atomic indicators in the completed formula from the preset code library; Based on all target indicator codes and the target formula codes, the executable code corresponding to the completed formula is determined.
[0074] Optionally, the verification module 450 is specifically used for: If the verification result of the executable code is successful, the executable code is identified as the evaluation indicator generation code corresponding to the evaluation indicator requirement description text.
[0075] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a scenario-adaptive evaluation metric generation method. This method includes: performing semantic analysis on the evaluation metric requirement description text corresponding to the target scenario to obtain a global feature vector corresponding to the evaluation metric requirement description text; predicting a target formula template and a set of target atomic metrics corresponding to the evaluation metric requirement description text based on the global feature vector; completing the target formula template based on the set of target atomic metrics to obtain a completed formula; determining the executable code corresponding to the completed formula; and determining the evaluation metric generation code corresponding to the evaluation metric requirement description text based on the verification result of the executable code.
[0076] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0077] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the scenario-adaptive evaluation index generation method provided by the above methods. The method includes: performing semantic analysis on the evaluation index requirement description text corresponding to the target scenario to obtain a global feature vector corresponding to the evaluation index requirement description text; predicting a target formula template and a set of target atomic indicators corresponding to the evaluation index requirement description text based on the global feature vector; completing the target formula template based on the set of target atomic indicators to obtain a completed formula; determining the executable code corresponding to the completed formula; and determining the evaluation index generation code corresponding to the evaluation index requirement description text based on the verification result of the executable code.
[0078] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the scenario-adaptive evaluation index generation method provided by the above methods. The method includes: performing semantic analysis on the evaluation index requirement description text corresponding to the target scenario to obtain a global feature vector corresponding to the evaluation index requirement description text; predicting a target formula template and a set of target atomic indicators corresponding to the evaluation index requirement description text based on the global feature vector; completing the target formula template based on the set of target atomic indicators to obtain a completed formula; determining the executable code corresponding to the completed formula; and determining the evaluation index generation code corresponding to the evaluation index requirement description text based on the verification result of the executable code.
[0079] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for generating evaluation metrics that is adaptive to specific scenarios, characterized in that, include: Semantic analysis is performed on the text describing the evaluation indicator requirements for the target scenario to obtain the global feature vector corresponding to the text describing the evaluation indicator requirements. Based on the global feature vector, predict the target formula template and target atomic index set corresponding to the evaluation index requirement description text; Based on the target atomic index set, the target formula template is completed to obtain the completed formula; Determine the executable code corresponding to the completed formula; Based on the verification results of the executable code, the evaluation indicator generation code corresponding to the evaluation indicator requirement description text is determined.
2. The method for generating scene-adaptive evaluation metrics according to claim 1, characterized in that, The semantic analysis of the evaluation indicator requirement description text corresponding to the target scenario is performed to obtain the global feature vector corresponding to the evaluation indicator requirement description text, including: Identify multiple keywords in the text describing the requirements for the evaluation indicators; Each keyword is vector-encoded to obtain the word vector corresponding to each keyword; Based on all word vectors and the semantic dependencies between word vectors, the graph structure corresponding to the evaluation index requirement description text is determined; the nodes in the graph structure represent word vectors, and the edges between nodes represent the semantic dependencies between the corresponding word vectors. By aggregating the word vectors corresponding to each node and its neighboring nodes, a fused word vector corresponding to each node is obtained. The fusion word vectors corresponding to all nodes are determined as the global feature vectors corresponding to the evaluation indicator requirement description text.
3. The method for generating scene-adaptive evaluation metrics according to claim 1, characterized in that, The step of predicting the target formula template and target atomic index set corresponding to the evaluation index requirement description text based on the global feature vector includes: Based on the global feature vector, predict the target formula template corresponding to the evaluation index requirement description text from the formula template library; Based on the global feature vector and the number of indicator positions in the target formula template, predict the set of target atomic indicators corresponding to the evaluation indicator requirement description text from the atomic indicator library; the number of target atomic indicators in the target atomic indicator set is equal to the number of indicator positions in the target formula template.
4. The method for generating scene-adaptive evaluation metrics according to claim 1, characterized in that, The process of completing the target formula template based on the target atomic index set to obtain the completed formula includes: Based on semantic rules, the order of target atomic indicators corresponding to the target atomic indicator set is determined; the semantic rules are used to characterize the semantic association between preset atomic indicators. Based on the order of the target atomic indicators, all target atomic indicators in the target atomic indicator set are filled into the corresponding indicator positions in the target formula template to obtain the completed formula.
5. The method for generating scene-adaptive evaluation metrics according to claim 1, characterized in that, Determining the executable code corresponding to the completed formula includes: Determine the target formula code corresponding to the completed formula and the target index code corresponding to each of the target atomic indices in the completed formula from the preset code library; Based on all target indicator codes and the target formula codes, the executable code corresponding to the completed formula is determined.
6. The method for generating scene-adaptive evaluation metrics according to claim 2, characterized in that, The determination of multiple keywords in the text describing the requirements for the evaluation indicators includes: The text describing the requirements of the evaluation indicators is segmented into words to obtain multiple words corresponding to the text describing the requirements of the evaluation indicators. Part-of-speech analysis was performed on the multiple words to obtain multiple keywords in the text describing the requirements of the evaluation indicators.
7. The method for generating scene-adaptive evaluation metrics according to any one of claims 1-6, characterized in that, The step of determining the evaluation indicator generation code corresponding to the evaluation indicator requirement description text based on the verification result of the executable code includes: If the verification result of the executable code is successful, the executable code is identified as the evaluation indicator generation code corresponding to the evaluation indicator requirement description text.
8. A scenario-adaptive evaluation index generation device, characterized in that, include: The semantic analysis module is used to perform semantic analysis on the evaluation indicator requirement description text corresponding to the target scenario, and obtain the global feature vector corresponding to the evaluation indicator requirement description text. The prediction module is used to predict the target formula template and target atomic index set corresponding to the evaluation index requirement description text based on the global feature vector. The formula completion module is used to complete the target formula template based on the target atomic index set to obtain the completed formula. The code conversion module is used to determine the executable code corresponding to the completed formula; The verification module is used to determine the evaluation indicator generation code corresponding to the evaluation indicator requirement description text based on the verification result of the executable code.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the scene-adaptive evaluation metric generation method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the scene-adaptive evaluation metric generation method as described in any one of claims 1 to 7.