Information processing method and device, equipment and storage medium
By generating an evaluation plan and negotiating between multiple evaluation models, the limitations of model evaluation methods in the prior art are solved, and a more accurate and comprehensive evaluation of the output of the code generation model is achieved.
Patent Information
- Application Number
- CN202510380407.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-06-27
AI Technical Summary
Existing model evaluation methods are limited and it is difficult to fully evaluate the performance of code generation models, especially in terms of multi-source domain knowledge and complex code understanding.
By obtaining input information and response content, an evaluation plan is generated, actions are performed in the operating environment associated with the code content, input and execution results are provided to multiple evaluation models, and evaluation content of multiple evaluation models is combined to generate evaluation results of response content.
By introducing multi-source knowledge and negotiation of multiple evaluation models, the evaluation accuracy of model output is improved, ensuring a comprehensive understanding and evaluation of complex code.
Smart Images
Figure CN120215903A_ABST
Abstract
Description
Technical Field
[0001] Example embodiments of the present disclosure generally relate to the field of computers, and particularly to methods, apparatuses, devices, and computer-readable storage media for information processing. Background Art
[0002] The evaluation of a model refers to the process of measuring the performance and evaluating the effectiveness of a model in fields such as machine learning and data analysis, in order to judge the quality, reliability, and applicability of the model, and to provide a basis for the selection, improvement, and application of the model. However, the currently applied model evaluation methods are very limited, so more comprehensive model evaluation methods are needed to improve the performance of the model. Summary of the Invention
[0003] In a first aspect of the present disclosure, a method for information processing is provided. The method includes: obtaining input information and response content generated based on the input information, where the response content includes code content; generating an evaluation plan for evaluating the response content based on the input information and the response content; executing a set of actions determined based on the evaluation plan in a runtime environment associated with the code content; providing the input information, the response content, and the execution result of the set of actions to multiple evaluation models to obtain multiple evaluation contents of the multiple evaluation models; and generating an evaluation result of the response content based on the multiple evaluation contents.
[0004] In a second aspect of the present disclosure, an apparatus for information processing is provided. The apparatus includes: an obtaining module configured to obtain input information and response content generated based on the input information, where the response content includes code content; a first generating module configured to generate an evaluation plan for evaluating the response content based on the input information and the response content; an execution module configured to execute a set of actions determined based on the evaluation plan in a runtime environment associated with the code content; an obtaining module configured to provide the input information, the response content, and the execution result of the set of actions to multiple evaluation models to obtain multiple evaluation contents of the multiple evaluation models; and a second generating module configured to generate an evaluation result of the response content based on the multiple evaluation contents.
[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, where the at least one memory is coupled to the at least one processing unit and stores actions for execution by the at least one processing unit. The actions, when executed by the at least one processing unit, cause the device to execute the method of the first aspect.
[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. A computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.
[0007] In a fifth aspect of the present disclosure, there is provided a computer program product. The computer program product includes computer-executable actions that, when executed by a processor, implement the method according to the first aspect of the present disclosure.
[0008] It should be understood that the content described in this content section is not intended to define the key features or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent. In the drawings, the same or similar reference numerals denote the same or similar elements, where:
[0010] Figure 1 A schematic diagram showing an example environment in which the embodiments of the present disclosure can be implemented;
[0011] Figure 2 A flowchart showing the process of information processing according to some embodiments of the present disclosure;
[0012] Figure 3 A schematic diagram showing input information and response content according to some embodiments of the present disclosure;
[0013] Figure 4 A flowchart showing the model evaluation according to some embodiments of the present disclosure;
[0014] Figure 5 A flowchart showing the evaluation of the application evaluation model according to some embodiments of the present disclosure;
[0015] Figure 6 A schematic diagram showing an evaluation report according to some embodiments of the present disclosure;
[0016] Figure 7 A schematic structural block diagram showing a device for information processing according to certain embodiments of the present disclosure;
[0017] Figure 8 A block diagram showing an electronic device capable of implementing multiple embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.
[0019] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this document, and any type of embodiment can be included under any section / subsection. In addition, the embodiments described in any section / subsection can be combined with any other embodiments described in the same section / subsection and / or different section / subsections in any manner.
[0020] In the description of the embodiments of the present disclosure, the term "comprising" and its like should be understood as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". There may be other explicit and implicit definitions hereinafter. The terms "first", "second", etc. may refer to different or the same objects. There may be other explicit and implicit definitions hereinafter.
[0021] The embodiments of the present disclosure may involve the user's data, data acquisition and / or use, etc. These aspects all comply with the corresponding laws, regulations and related provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware and confirms. Accordingly, when implementing the embodiments of the present disclosure, the data or the type of data, the scope of use, the usage scenario, etc. that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner according to the relevant laws and regulations. The specific informing and / or authorization methods may vary according to the actual situation and application scenarios, and the scope of the present disclosure is not limited in this regard.
[0022] In the solutions of this specification and embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the personal data subject, or being necessary for the performance of a contract, etc.), and will only be processed within the specified or agreed scope. If the user refuses to process personal data other than the necessary data required for the basic functions, it will not affect the user's use of the basic functions.
[0023] Code generation models have demonstrated powerful capabilities in code generation, which highlights the necessity for a comprehensive and rigorous evaluation of them. Existing evaluation methods are mainly divided into three categories: human-centered methods, metric-based methods, and model-based methods. Given that human-centered methods are labor-intensive, and metric-based methods overly rely on reference answers, model-based methods have received increasing attention due to their stronger context understanding ability and higher efficiency. However, model-based methods still have problems such as a lack of multi-source domain knowledge and insufficient understanding of complex code.
[0024] Embodiments of the present disclosure propose a solution for information processing. According to this solution, input information and response content generated based on the input information are obtained, and the response content includes code content; an evaluation plan for evaluating the response content is generated based on the input information and the response content; a set of actions determined based on the evaluation plan is executed in a running environment associated with the code content; the input information, the response content, and the execution results of the set of actions are provided to multiple evaluation models to obtain multiple evaluation contents of the multiple evaluation models; and an evaluation result of the response content is generated based on the multiple evaluation contents.
[0025] Based on such a manner, embodiments of the present disclosure can introduce multi-source knowledge by executing a set of actions determined by the evaluation plan, and comprehensively evaluate the response content using the multi-source knowledge to obtain the execution results of the set of actions. Then, multiple evaluation models are applied to evaluate the response content to generate an evaluation result of the response content, thereby improving the accuracy of content evaluation.
[0026] Example environment
[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. As Figure 1 shown, the example environment 100 may include an electronic device 110.
[0028] In some embodiments, the electronic device 110 can obtain input information and response content generated based on the input information, where the response content includes code content; generate an evaluation plan for evaluating the response content based on the input information and the response content; execute a set of actions determined based on the evaluation plan in a running environment associated with the code content; provide the input information, the response content, and the execution results of the set of actions to multiple evaluation models to obtain multiple evaluation contents of the multiple evaluation models; and generate an evaluation result of the response content based on the multiple evaluation contents.
[0029] In some embodiments, the response content may be generated by the generative model 120. The generative model 120 may be deployed, for example, on the electronic device 110, and may also be deployed on other devices, which will not be elaborated here. As an example, the generative model 120 may include a code generation model implemented based on a language model.
[0030] In some embodiments, the electronic device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, Personal Communication System (PCS) devices, personal navigation devices, Personal Digital Assistant (PDA), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the target user (such as a "wearable" circuit, etc.).
[0031] The electronic device 110 may also be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. The electronic device 110 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and so on.
[0032] It should be understood that the structures and functions of the various elements in the environment 100 are described only for exemplary purposes, without implying any limitation on the scope of the present disclosure.
[0033] Some exemplary embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.
[0034] Example process
[0035] Figure 2 A flowchart of a process 200 for information processing according to some embodiments of the present disclosure is shown. The process 200 may be implemented at the electronic device 110. The following will refer to Figure 1 Describe the process 200.
[0036] At block 210, the electronic device 110 may obtain input information and response content generated based on the input information.
[0037] In some embodiments, the response content is generated by a code generation model according to the input information, and the corresponding content includes code content. A code generation model is a model that uses machine learning techniques to convert natural language descriptions, code snippets, or other input data into executable code. The code generation model can automatically generate corresponding code segments according to given prompts or specifications, helping developers improve programming efficiency and quality.
[0038] In some embodiments, referring to Figure 3 , the electronic device 110 may obtain the input information 310 "Hi, how can I use Pandas in Python to read a CSV file, print its content, handle exceptions, count the occurrences of each word, and save the results to a new file?", then input the input information into the code generation model, and obtain the following response content 320:
[0039] "Okay! Here is a step-by-step guide on how to...
[0040] ```
[0041] def read_file(path):
[0042] def count_words(text):
[0043] def save_counts(counts,path):
[0044] ```
[0045] Please make sure to replace 'input.csv' and 'word_counts.csv' with the actual input file and output file paths."
[0046] In some embodiments, in order to accurately evaluate the response generated by the model, a complete and executable running environment needs to be constructed. Referring to Figure 4 , the electronic device 110 may provide the input information and the response content to the environment construction unit 410 to generate a set of instructions for constructing the running environment. Then execute the set of instructions to construct a running environment associated with the code content.
[0047] For example, the model agent may construct the required running environment in Docker according to the input information and the response content. The initial Docker container has been configured with network settings and external tools, and constructing the running environment includes setting up the programming environment and installing external dependencies. Referring to Figure 4, the operating environment may include a programming environment 411, dependencies 412, environment variables 413, and configuration files 414. For more complex code generation tasks, additional configuration files, environment variables, and other necessary components need to be set, which will not be elaborated here.
[0048] In block 220, the electronic device 110 may generate an evaluation plan for evaluating the response content based on the input information and the response content.
[0049] In some embodiments, to improve the quality of the response generated by the model, ensure that the output of the model meets the expectations, and provide a basis for further optimizing the model, the response generated by the model can be evaluated. Evaluation refers to measuring the quality, accuracy, and applicability of the response generated by the model through a series of criteria and methods.
[0050] In some embodiments, the evaluation plan includes multiple stages in a predetermined order, and each stage includes corresponding goals and operation methods, which helps to verify each requirement and reduce the reasoning burden of the model agent. For example, referring to Figure 4 , the evaluation plan may include four stages in a predetermined order. Stage one 431 includes the goal of "static code inspection and analysis" and the operation method of "using 'linter_analysis' to check..."; stage two 432 includes the goal of "dynamic execution analysis" and the operation method of "executing the code and analyzing..."; stage three 433 includes the goal of "web browsing analysis" and the operation method of "using 'web_browse' to obtain reference insights..."; stage four 434 includes the goal of "semantic analysis" and the operation method of "analyzing the function of...".
[0051] In some embodiments, to enhance the model agent's understanding of the task requirements and prepare for formulating a comprehensive evaluation plan, referring to Figure 4 , the electronic device 110 may provide the input information to the requirement analysis unit 420 to generate at least one requirement associated with the input information. By splitting the complex task into at least one smaller and more manageable requirement, it helps the model agent better understand the requirements. For example, requirement one "read and print the file content", requirement two "handle file read exceptions", and requirement three "count and save the occurrences of words". Then, provide at least one requirement, the input information, and the response content to the plan generation unit 430 to generate an evaluation plan for evaluating the response content.
[0052] In block 230, the electronic device 110 may execute a set of actions determined based on the evaluation plan in the operating environment associated with the code content.
[0053] In some embodiments, the goals and operation modes within each stage of the evaluation plan jointly define the actions that the model agent should take. For example, referring to Figure 4 , the goal of "Static code inspection and analysis" in stage 431 is to ensure that the response generated by the code generation model complies with the code syntax and style. The corresponding operation mode is "Use "linter_analysis" to check...".
[0054] In some embodiments, referring to Figure 4 , the electronic device 110 can generate at least one stage action corresponding to each stage of the evaluation plan by the execution analysis unit 440 based on the evaluation plan. Then, execute at least one stage action corresponding to each stage in the running environment, and the execution action can be executed in the constructed Docker container.
[0055] In some embodiments, the electronic device 110 can obtain the execution results of at least one stage action, and then the execution analysis unit 440 generates analysis content for the execution results. For example, referring to Figure 4 , the evaluation plan includes multiple stages. The action corresponding to stage 1 can be executed first to obtain the execution result 1 corresponding to stage 1, and then the analysis content 1 for the execution result 1 is generated. Next, the action corresponding to stage 2 is executed to obtain the execution result 2 corresponding to stage 2, and then the analysis content 2 for the execution result 2 is generated, and so on. That is, the model agent alternates between execution operations and analysis operations during the interaction process.
[0056] Another example is that the evaluation plan includes stage 431 and stage 432. Stage 431 includes the goal of "Static code inspection and analysis" and the operation mode of "Use "linter_analysis" to check...", and stage 432 includes the goal of "Dynamic execution analysis" and the operation mode of "Execute the code and analyze...". Then, referring to Figure 4 , dividing the execution of the action corresponding to stage 1 into multiple steps: the first step is to think "I will execute stage 1 of the evaluation plan", the second step is the action "linter_analyse - f count.py", the third step is to observe the result "Code inspection and analysis report", and the fourth step is the status "Execution". Analyzing the execution result 1 also includes multiple steps: the first step is to think "I will report the analysis content of the current stage in the evaluation plan", the second step is the action "analyse_current_step", the third step is the status "Analysis", and the fourth step is to report "There are some code style problems in this code, especially...".
[0057] Next, the actions corresponding to the second execution phase can be divided into multiple steps: the first step is to think "I will execute the second phase in the evaluation plan", the second step is the action "python3 count.py", the third step is to observe the result "execution information of the code", and the fourth step is the status "executing". Analyzing the execution result 2 also includes multiple steps: the first step is to think "I will report the analysis content of the current phase in the evaluation plan", the second step is the action "analyse_current_step", the third step is the status "analyzing", and the fourth step is to report "Based on the execution result, we believe that...". Then execute the third phase, the fourth phase, and so on, and finally obtain the evaluation report.
[0058] In some embodiments, a set of actions includes at least one of the following: a first action for dynamic execution analysis, a second action for static code inspection analysis, a third action for unit test analysis, a fourth action for interface content analysis, a fifth action for interaction analysis, a sixth action for web browsing analysis, a seventh action for semantic analysis, and an eighth action for command line analysis.
[0059] In some embodiments, for the first action of dynamic execution analysis, this action checks whether there are compilation errors and analyzes whether the execution result matches the expected result. The following commands can be used to execute the first action: "python test.py" for running Python scripts, and "gcc -o output test.c &&. / output" for compiling and executing C programs, etc.
[0060] For example, the input information is "Given an integer array nums, where each element appears three times, except for one element that appears only once. Find and return this element that appears only once. You must implement a solution with linear time complexity and can only use constant extra space (using the Python language)", and the code generation model generates the code accordingly. The analysis content corresponding to the dynamic execution analysis action can include:
[0061] "i. Output result:
[0062] After the code is executed, the number 3 is output.
[0063] ii. Verification of expected result:
[0064] According to the example usage nums = [2, 2, 3, 2] in the code, the expected output is indeed 3.
[0065] 3 is the only element that appears only once in the array, while 2 appears three times.
[0066] iii. Confirmation of correctness:
[0067] The output matches the expected result, indicating that the code correctly found the element that appears only once…
[0068] In some embodiments, a second action for static code inspection analysis, which checks for syntax errors, code issues, and style inconsistencies, helps identify potential problems in the response generated by the model without executing the code. The compliance of a file or directory with the syntax can be checked by the command "linter_analysis -f 'path'", which automatically selects an appropriate static analysis tool based on the given code file path and generates a structured inspection report.
[0069] For example, when the input information is "Design a red button that turns yellow when the mouse hovers over it and green when clicked", the code generation model generates the corresponding code. The analysis content corresponding to the static code inspection analysis action may include:
[0070] "Syntax check overview: Multiple tools are used for static syntax checking, including LYCHEE, DJLINT, CSPELL, and HTMLHINT. LYCHEE and CSPELL passed smoothly, while DJLINT and HTMLHINT reported some issues.
[0071] i. The HTML file is syntactically correct and has no major syntax errors.
[0072] ii. DJLINT's suggestions are related to search engine optimization (SEO), which does not affect the functionality but can improve the friendliness to search engines.
[0073] iii. The issues pointed out by HTMLHINT are related to naming conventions. It suggests changing "colorButton" to "colorbutton", which is a code style issue and does not affect the functionality.
[0074] iv. Generally speaking, these issues do not affect the basic functionality and style of the button, but following these suggestions can improve the code quality and maintainability."
[0075] In some embodiments, a third action for unit test analysis, which includes writing and executing unit tests to ensure that the responses generated by the model meet expectations under various conditions. Unit tests help analyze the functionality and reliability of the responses by providing different inputs and using assertions to verify the outputs. For example, consider a function that processes a list of integers and returns a list of squares only for even numbers. The unit test might provide the input "[1, 2, 3, 4]" and use the assertion statement "assert process_list([1, 2, 3, 4]) == [4, 16]" to verify that the output meets expectations.
[0076] For example, the input information is "Given an integer array nums where each element appears three times except for one element which appears only once. Find and return the single element that appears only once. You must implement a solution with linear time complexity and use only constant extra space (using the Python language)", and the code generation model generates the code accordingly. The analysis content corresponding to the unit test analysis action can include:
[0077] "Boundary case tests confirm the robustness of the code. The singleNumber function can correctly identify the unique element in various scenarios, including arrays of different lengths, arrays containing negative numbers, and arrays with a mixture of elements. This comprehensive testing ensures the reliability and correctness of the solution..."
[0078] In some embodiments, a fourth action for interface content analysis can capture screenshots of the front-end code for visual analysis. The command "screenshot_analsis -f 'path' -q 'query'" can be executed. By providing the code file path, this command can render it as a screenshot to extract relevant information such as visual elements and layout, etc., to query whether the screenshot meets the requirements.
[0079] For example, the input information is "Design a red button", and the code generation model generates the code accordingly. The analysis content corresponding to the interface content analysis action can include: "Screenshot analysis: The button appears with a red background, white text, and the label 'Click Me'. According to the screenshot analysis, the code correctly implements the initial state of the button."
[0080] In some embodiments, a fifth action for interaction analysis involves simulating user interactions with the front-end code to evaluate interaction elements. The "screenshot_analysis" command can be extended with the "-a 'actions'" parameter to interact with the interface before taking the screenshot. Interactions can include clicking on elements, filling input fields, hovering the mouse over elements, scrolling, etc.
[0081] For example, when the input information is "Design a red button that turns yellow when the mouse hovers over it", the code generation model generates the corresponding code. The analysis content corresponding to the interface content analysis action may include: "Screenshot analysis: When the mouse hovers over the button, it turns yellow. The code correctly implements the hover state of the button."
[0082] In some embodiments, the sixth action for web browsing analysis involves retrieving additional information from websites, including the latest programming technical knowledge and professional answers. It can be achieved through the command "web_browse-q 'query content'" to search the website according to the query content. In this way, the latest domain knowledge can be obtained, making the evaluation of the model output more accurate.
[0083] In some embodiments, the seventh action for semantic analysis directly applies the model's context understanding ability. Therefore, without calling external tools, the functionality, logic, complexity, security, and robustness of the model can be evaluated.
[0084] For example, when the input information is "Design a red button that turns yellow when the mouse hovers over it and turns green when clicked", the code generation model generates the corresponding code. The analysis content corresponding to the interface content analysis action may include: "Generally speaking, the CSS styles fully meet the user's requirements and correctly define the color changes of the button in different states. These styles are concise and effective, without unnecessary or redundant code. The only possible improvement is to consider using more flexible units (such as em or rem) instead of fixed pixel values to better adapt to different screen sizes…".
[0085] In some embodiments, the eighth action for command-line analysis, in addition to the operations specifically designed for code generation, also requires calling Bash commands during the analysis process to perform operations such as writing code, writing data, viewing files, and browsing the file system.
[0086] For example, when the input information is "Design a red button that turns yellow when the mouse hovers over it and turns green when clicked", the code generation model generates the corresponding code. The analysis content corresponding to the interface content analysis action may include: "Command analysis: I have successfully written the code into a new file."
[0087] In this way, comprehensive multi-source domain knowledge can be obtained, including the latest programming technical knowledge, visual and interaction information, runtime information, code inspection information, etc., thus avoiding inaccurate model output due to the lack of multi-source domain knowledge.
[0088] In block 240, the electronic device 110 can provide input information, response content, and execution results of a set of actions to multiple evaluation models to obtain multiple evaluation contents of the multiple evaluation models.
[0089] Since in practical applications, code generation tasks usually involve multiple task requirements, which results in the responses generated by large language models containing complex and numerous code snippets, but it is very difficult to understand and evaluate these complex codes. Therefore, multiple evaluation models can be used to evaluate the model's responses respectively, and then these evaluation models negotiate to obtain the final evaluation result of the model output. This way helps to reduce the inaccurate understanding of complex codes by the model and ensure a more comprehensive evaluation through the negotiation among multiple evaluation models.
[0090] In some embodiments, the multiple evaluation models correspond to multiple agents. An agent refers to an entity or program with certain intelligent behaviors and autonomous capabilities in a network environment, which can perceive the environment, process information, make decisions, and execute corresponding actions.
[0091] In some embodiments, the electronic device 110 can input the input information, response content, and execution results of a set of actions into multiple evaluation models. Then, based on the preset evaluation criteria corresponding to the multiple evaluation models, multiple evaluation contents of the multiple evaluation models are generated. For example, it can be through A1, A2,..., A n representing n evaluation models, and each evaluation model A i can evaluate the response content generated by the model according to the preset evaluation criteria to generate evaluation content. The evaluation content can include an evaluation score S i and the corresponding scoring reason R i , then A i =(S i , R i ). Referring to Figure 5 , evaluation model 511 generates "I give 4 points because xxx", evaluation model 512 generates "I give 3 points because xxx", and evaluation model 513 generates "I give 1 point because xxx".
[0092] In some embodiments, the preset evaluation criteria include the correctness, functionality, and clarity of the code, and can be set according to the actual situation. For example, the preset evaluation criteria can be as shown in Table 1 and Table 2:
[0093]
[0094] Table 1
[0095]
[0096]
[0097] Table II
[0098] In block 250, the electronic device 110 may generate an evaluation result of the response content based on multiple evaluation contents.
[0099] In some embodiments, the electronic device 110 may perform at least one round of negotiation process associated with multiple evaluation models, where each round of negotiation process includes sharing multiple evaluation contents among the multiple evaluation models to determine whether to update the multiple evaluation contents. For example, in each evaluation model A i has given an evaluation score and the corresponding scoring reasons After that, the evaluation content may be shared among the multiple evaluation models, that is, each evaluation model can obtain the evaluation content corresponding to other evaluation models. This process can be expressed as:
[0100]
[0101] Then, these evaluation models may perform multiple rounds of negotiation based on the shared multiple evaluation contents to determine whether to update the evaluation content.
[0102] In some embodiments, the electronic device 110 may obtain the first evaluation content generated by the first evaluation model and the second evaluation content generated by the second evaluation model. Then, the second evaluation content is provided to the first evaluation model to determine a first feedback action of the first evaluation model from multiple candidate feedback actions, where the first feedback action indicates whether to adjust the first evaluation content. For example, k may represent the number of rounds of negotiation. In each round of negotiation, each evaluation model A i may take one or more feedback actions according to the evaluation content provided by other evaluation models
[0103] In some embodiments, the multiple candidate feedback actions include: a first feedback action for maintaining the first evaluation content, a second feedback action for adjusting the first evaluation content, and a third feedback action for sending an evaluation query message to the second evaluation model. The candidate feedback actions may be represented as follows:
[0104]
[0105] Maintain represents the first feedback action, including the reason for maintaining the first evaluation content. Change represents the second feedback action, including the adjusted evaluation content and the reason for adjusting the first evaluation content. Query represents the third feedback action, including sending an evaluation query message to evaluation model A jSend an evaluation query message. That is, the evaluation model needs to provide corresponding reasons when taking different feedback actions. After k rounds of negotiation, the evaluation content given by each evaluation model and the feedback actions taken will be shared among multiple evaluation models:
[0106]
[0107] For example, referring to Figure 5 , the evaluation model 511 can output "Your score is unreasonable..." based on the score provided by the evaluation model 513. The evaluation model 512 can output "I don't agree with your evaluation..." based on the score provided by the evaluation model 513. The evaluation model 513 can output "I question your evaluation..." based on the score provided by the evaluation model 511. Further, the evaluation model 511 outputs "I think my score is reasonable.", that is, the evaluation model 511 makes the first feedback action. The evaluation model 512 outputs "I have no other suggestions.", that is, the evaluation model 512 makes the first feedback action. The evaluation model 513 outputs "Thank you for your suggestions. I change the score to 3.", that is, the evaluation model 513 makes the second feedback action. Then after negotiation, the evaluation model 511 outputs "The final score is 4.", the evaluation model 512 outputs "The final score is 3.", and the evaluation model 513 outputs "The final score is 3.".
[0108] In some embodiments, the electronic device 110 can generate an evaluation result of the response content based on multiple evaluation contents after at least one round of negotiation process in response to the number of rounds of the at least one round of negotiation process reaching a threshold. For example, when the number of rounds of negotiation reaches the threshold, an evaluation result of the response content can be generated according to multiple evaluation contents after at least one round of negotiation process, and the score corresponding to the final evaluation result is the average of the evaluation scores given by multiple evaluation models:
[0109]
[0110] The electronic device 110 can also generate an evaluation result of the response content based on multiple evaluation contents in response to the difference between multiple evaluation contents after the target negotiation process being less than a threshold. For example, when the evaluation contents with a score of 3 are made by multiple evaluation models, 3 can be used as the evaluation result of the response content.
[0111] In some embodiments, after obtaining the evaluation result of the response content, an evaluation report can also be generated, and the template of this evaluation report can be adjusted according to actual applications and is not limited here. Referring to Figure 6 , the final output can include the evaluation score and the evaluation report. For example, the template of the evaluation report can refer to Table 3:
[0112]
[0113]
[0114] Table III
[0115] Through this method, embodiments of the present disclosure can collect comprehensive multi-source domain knowledge through an evaluation plan, thereby avoiding situations such as inaccurate evaluation results caused by the lack of multi-source knowledge. In addition, embodiments of the present disclosure can also obtain the final evaluation result through negotiation using multiple evaluation models, avoiding inaccurate evaluation results caused by insufficient understanding of complex code.
[0116] Example devices and equipment
[0117] Embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 7 FIG. shows a schematic structural block diagram of an information processing device 700 according to certain embodiments of the present disclosure. The device 700 can be implemented as or included in the electronic device 110 discussed above. Each module / component in the device 700 can be implemented by hardware, software, firmware, or any combination thereof.
[0118] As Figure 7 shown, the device 700 includes an acquisition module 710 configured to acquire input information and response content generated based on the input information, where the response content includes code content; a first generation module 720 configured to generate an evaluation plan for evaluating the response content based on the input information and the response content; an execution module 730 configured to execute a set of actions determined based on the evaluation plan in a runtime environment associated with the code content; an acquisition module 740 configured to provide the input information, the response content, and the execution result of the set of actions to multiple evaluation models to obtain multiple evaluation contents of the multiple evaluation models; and a second generation module 750 configured to generate an evaluation result of the response content based on the multiple evaluation contents.
[0119] In some embodiments, the device 700 is further configured to provide the input information and the response content to an environment construction unit to generate a set of instructions for constructing a runtime environment; and execute the set of instructions to construct a runtime environment associated with the code content.
[0120] In some embodiments, the first generation module 720 is configured to provide the input information to a requirements analysis unit to generate at least one requirement associated with the input information; and provide the at least one requirement, the input information, and the response content to a plan generation unit to generate an evaluation plan for evaluating the response content.
[0121] In some embodiments, the evaluation plan includes multiple stages in a predetermined order, and each stage includes a corresponding objective and an operation mode.
[0122] In some embodiments, a set of actions includes at least one of the following: a first action for dynamically performing analysis; a second action for static code inspection analysis; a third action for unit test analysis; a fourth action for interface content analysis; a fifth action for interaction analysis; a sixth action for web browsing analysis; a seventh action for semantic analysis; an eighth action for command line analysis.
[0123] In some embodiments, the execution module 730 is configured to generate, by an execution analysis unit, at least one stage action corresponding to each stage of an evaluation plan based on the evaluation plan; and execute at least one stage action corresponding to each stage in a running environment.
[0124] In some embodiments, the execution module 730 is configured to obtain an execution result of at least one stage action; and generate analysis content for the execution result by an execution analysis unit.
[0125] In some embodiments, the device 700 is configured to execute at least one round of negotiation process associated with multiple evaluation models, where each round of negotiation process includes sharing multiple evaluation contents among the multiple evaluation models to determine whether to update the multiple evaluation contents.
[0126] In some embodiments, the second generation module 750 is configured to obtain first evaluation content generated by a first evaluation model and second evaluation content generated by a second evaluation model; provide the second evaluation content to the first evaluation model to determine a first feedback action of the first evaluation model from multiple candidate feedback actions, where the first feedback action indicates whether to adjust the first evaluation content.
[0127] In some embodiments, the multiple candidate feedback actions include: a first feedback action for maintaining the first evaluation content; a second feedback action for adjusting the first evaluation content; a third feedback action for sending an evaluation query message to the second evaluation model.
[0128] In some embodiments, the second generation module 750 is configured to generate an evaluation result of a response content based on the multiple evaluation contents after at least one round of negotiation process in response to the number of rounds of at least one round of negotiation process reaching a threshold; or generate an evaluation result of a response content based on the multiple evaluation contents in response to the difference between the multiple evaluation contents after a target negotiation process being less than a threshold.
[0129] In some embodiments, the multiple evaluation models correspond to multiple agents.
[0130] The units included in apparatus 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable actions stored on a storage medium. In addition to or as an alternative to the machine-executable actions, some or all of the units in apparatus 700 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0131] Figure 8 FIG. shows a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure can be implemented. It should be understood that Figure 8 the electronic device 800 shown is merely exemplary and should not constitute any limitation to the functions and scope of the embodiments described herein. Figure 8 The electronic device 800 shown can be used to implement Figure 1 the electronic device 110 shown.
[0132] As Figure 8 shown, the electronic device 800 is in the form of a general-purpose electronic device. The components of the electronic device 800 can include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 can be an actual or virtual processor and is capable of performing various processes according to programs stored in the memory 820. In a multi-processor system, multiple processing units execute computer-executable actions in parallel to improve the parallel processing ability of the electronic device 800.
[0133] The electronic device 800 generally includes multiple computer storage media. Such media can be any accessible media that can be obtained by the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be volatile memory (such as registers, caches, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 can be removable or non-removable media and can include machine-readable media, such as flash drives, magnetic disks, or any other media that can be used to store data and / or data (such as training data for training) and can be accessed within the electronic device 800.
[0134] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in Figure 8 , a disk drive for reading from and writing to a removable, non-volatile disk (such as a "floppy disk") and an optical disk drive for reading from and writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules configured to execute the various methods or actions of the various embodiments of the present disclosure.
[0135] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 800 may be implemented by a single computing cluster or multiple computer machines capable of communicating via a communication connection. Thus, the electronic device 800 may operate in a networked environment using a logical connection to one or more other servers, network personal computers (PCs), or another network node.
[0136] The input device 850 may be one or more input devices such as a mouse, keyboard, trackball, etc. The output device 860 may be one or more output devices such as a display, speaker, printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) as needed via the communication unit 840, such as a storage device, a display device, etc., communicate with one or more devices that enable a user to interact with the electronic device 800, or communicate with any device that enables the electronic device 800 to communicate with one or more other electronic devices (such as a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).
[0137] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable actions are stored, where the computer-executable actions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product being tangibly stored on a non-transitory computer-readable medium and including computer-executable actions, and the computer-executable actions being executed by a processor to implement the method described above.
[0138] Aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and the combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program actions.
[0139] These computer-readable program operations can be provided to the processing unit of a general-purpose computer, a special-purpose computer, or other programmable information processing device, thereby producing a machine such that when these operations are executed by the processing unit of the computer or other programmable information processing device, a device is produced that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program operations can also be stored in a computer-readable storage medium, and these operations cause the computer, programmable information processing device, and / or other devices to operate in a specific manner, so that the computer-readable medium storing the operations includes a manufactured article that includes operations implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0140] The computer-readable program operations can be loaded onto a computer, other programmable information processing device, or other device, such that a series of operational steps are executed on the computer, other programmable information processing device, or other device to produce a computer-implemented process, so that the operations executed on the computer, other programmable information processing device, or other device implement the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various implementations of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of an operation, and the module, segment of a program, or part of an operation includes one or more executable operations for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur in an order different from that noted in the drawings. For example, two consecutive blocks may in fact be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer operations.
[0142] The various implementations of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art in the field of this technology without departing from the scope and spirit of the described implementations. The choice of terms used herein is intended to best explain the principles of the implementations, the practical application, or the improvement of the technology in the market, or to enable other ordinary skilled persons in the field of this technology to understand the various implementation manners disclosed herein.
Claims
1. A method of information processing, comprising: Acquire input information and response content generated based on the input information, wherein the response content includes code content; generating an evaluation plan for evaluating the response content based on the input information and the response content; executing a set of actions determined based on the evaluation plan in an execution environment associated with the code content; Providing the input information, the response content, and the execution results of the set of actions to multiple evaluation models to obtain multiple evaluation contents of the multiple evaluation models; as well as Based on the plurality of evaluation contents, an evaluation result of the response content is generated.
2. The method according to claim 1, further comprising: Providing the input information and the response content to an environment construction unit to generate a set of instructions for constructing the operating environment; as well as The set of instructions is executed to construct the execution environment associated with the code content.
3. The method according to claim 1, wherein generating an evaluation plan for evaluating the response content based on the input information and the response content comprises: providing the input information to a requirement analysis unit to generate at least one requirement associated with the input information; as well as The at least one requirement, the input information, and the response content are provided to a plan generating unit to generate the evaluation plan for evaluating the response content. 4 . The method according to claim 1 , wherein the evaluation plan comprises a plurality of stages in a predetermined order, each stage comprising a corresponding goal and operation method.
5. The method of claim 1, wherein the set of actions comprises at least one of the following: The first action is used to dynamically perform analysis; The second action is used for static code checking and analysis; The third action is used for unit test analysis; The fourth action is used for interface content analysis; The fifth action is used for interactive analysis; The sixth action is used for web browsing analysis; The seventh action is used for semantic analysis; The eighth action is used for command line analysis.
6. The method of claim 1, wherein executing a set of actions determined based on the evaluation plan in an execution environment associated with the code content comprises: The execution analysis unit generates at least one stage action corresponding to each stage of the evaluation plan based on the evaluation plan; as well as The at least one stage action corresponding to each stage is executed in the operating environment.
7. The method according to claim 6, further comprising: Obtaining the execution result of the at least one stage action; as well as The execution analysis unit generates analysis content for the execution result.
8. The method according to claim 1, wherein before generating the evaluation result of the response content, the method further comprises: At least one round of negotiation process associated with the plurality of evaluation models is executed, wherein each round of negotiation process includes sharing the plurality of evaluation contents among the plurality of evaluation models to determine whether to update the plurality of evaluation contents.
9. The method according to claim 8, wherein the plurality of evaluation models include at least a first evaluation model and a second evaluation model, and generating the evaluation result of the response content based on the plurality of evaluation contents comprises: Acquire first evaluation content generated by the first evaluation model and second evaluation content generated by the second evaluation model; The second evaluation content is provided to the first evaluation model to determine a first feedback action of the first evaluation model from a plurality of candidate feedback actions, the first feedback action indicating whether to adjust the first evaluation content.
10. The method of claim 9, wherein the plurality of candidate feedback actions comprises: A first feedback action, used to maintain the first evaluation content; A second feedback action is used to adjust the first evaluation content; The third feedback action is used to send an evaluation query message to the second evaluation model.
11. The method according to claim 8, wherein: Generating the evaluation result of the response content based on the multiple evaluation contents includes: In response to the number of rounds of the at least one round of negotiation process reaching a threshold, generating the evaluation result of the response content based on the multiple evaluation contents after the at least one round of negotiation process; or In response to a difference between the plurality of evaluation contents after the target negotiation process being smaller than a threshold, the evaluation result of the response content is generated based on the plurality of evaluation contents.
12. The method according to claim 1, wherein the multiple evaluation models correspond to multiple agents.
13. An apparatus for information processing, comprising: An acquisition module, configured to acquire input information and response content generated based on the input information, wherein the response content includes code content; A first generating module is configured to generate an evaluation plan for evaluating the response content based on the input information and the response content; an execution module configured to execute a set of actions determined based on the evaluation plan in an execution environment associated with the code content; an acquisition module, configured to provide the input information, the response content and the execution results of the set of actions to multiple evaluation models, so as to obtain multiple evaluation contents of the multiple evaluation models; as well as The second generating module is configured to generate an evaluation result of the response content based on the multiple evaluation contents.
14. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing actions for execution by the at least one processing unit, the actions, when executed by the at least one processing unit, causing the electronic device to execute the method according to any one of claims 1 to 12.
15. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 12 when executed by a processor.
16. A computer program product comprising computer executable actions, wherein the computer executable actions, when executed by a processor, implement the method according to any one of claims 1 to 12.