Task adaptive generation method and system based on loop feedback, storage medium and electronic equipment

Abstract

The invention provides a task adaptive generation method and system based on loop feedback, a storage medium and electronic equipment, and the method comprises the steps: S1, generating an artificial intelligence job task based on a test instance list and artificial intelligence job feedback; s2, processing the artificial intelligence operation task to obtain an artificial intelligence operation result; s3, the artificial intelligence job result is used as a new artificial intelligence job to be fed back to the step S1; and S4, iteratively and circularly executing the steps S1 to S3. According to the method, automatic task generation is realized, dynamic adjustment is carried out according to an artificial intelligence operation result, mechanical listing of single-type questions is avoided, the generated task and the artificial intelligence operation model are effectively coupled, and automatic task generation and question answering loop optimization are truly realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a task adaptive generation method, system, storage medium and electronic device based on loop feedback. Background Art

[0002] Currently, benchmarking is widely used in the field of artificial intelligence (AI), such as ImageNet (a benchmark dataset for image recognition) and GLUE (a benchmark for natural language understanding). Although these tests can evaluate the performance of AI models on specific tasks, they usually focus on the evaluation of a single task and cannot fully examine the universal intelligence of the model. Especially in terms of test question generation, these existing benchmarks do not have the ability to generate intelligent tasks. The test questions are only listed mechanically, usually fixed on a single type of question, and cannot be dynamically adjusted according to the results of AI tasks.

[0003] For example, patent CN202410955119.8 proposes a task generation method and system. Although this invention is innovative in generating tasks, it fails to effectively couple the generated tasks with the artificial intelligence operation model. Specifically, it fails to combine the task solution feedback through a systematic mechanism to form an adaptive task generation and artificial intelligence operation cycle system. Therefore, although it has certain advantages in task generation, the lack of a feedback mechanism for artificial intelligence operation capabilities makes it impossible to achieve true automatic task generation and problem-solving cycle optimization.

[0004] Therefore, a solution is urgently needed. Summary of the invention

[0005] One of the purposes of the present invention is to provide a task adaptive generation method based on loop feedback, which generates artificial intelligence job tasks based on a test instance list and artificial intelligence job feedback, then processes the artificial intelligence job tasks to obtain artificial intelligence job results, and then uses the artificial intelligence job results as new artificial intelligence job feedback to regenerate artificial intelligence job tasks in this cycle, thereby realizing automatic task generation, dynamically adjusting according to the artificial intelligence job results, avoiding mechanically listing a single type of questions, and effectively coupling the generated tasks with the artificial intelligence job model to truly realize automatic task generation and problem-solving loop optimization.

[0006] The task adaptive generation method based on loop feedback provided by the embodiment of the present invention includes: Step S1: Generate an AI job task based on the test instance list and AI job feedback; Step S2: Processing the AI ​​operation task and obtaining the AI ​​operation result; Step S3: returning the AI ​​job result as new AI job feedback to step S1; Step S4: Iterate and loop through steps S1 to S3.

[0007] Optionally, the steps for obtaining the test instance list are as follows: Collect test tasks; Based on the task template definition standard, the test task is modeled to obtain the task template library; The task templates in the task template library are matched with the test environments in the test environment library to determine the test instance list.

[0008] Optionally, the collection test task includes: Crawl test tasks based on the test benchmark of the existing model; and / or, A test task that receives user input.

[0009] Optionally, the task template definition standard includes at least: nodes, edges and attributes of the task parsing graph.

[0010] Optionally, matching the task templates in the task template library with the test environments in the test environment library to determine the test instance list includes: Get the environmental information of the test environment; Construct the mapping relationship between the test environment information and the task template; Based on the mapping relationship, the environmental information of the test environment is added to the task template accordingly to obtain the test instance; Integrate test instances and obtain a list of test instances.

[0011] Optionally, the step S1: generating an artificial intelligence job task based on the test instance list and artificial intelligence job feedback includes: Extract test instances from the test instance list; Generate artificial intelligence tasks based on test instances; The step of extracting a test instance from the test instance list includes: When the AI ​​job feedback is empty, a test instance is extracted from the test instance list based on a random algorithm; When the artificial intelligence job feedback is not empty, the test instance requirement is determined based on the artificial intelligence job feedback, and the test instance is extracted from the test instance list based on the test instance requirement.

[0012] Optionally, determining the test instance requirements based on the artificial intelligence job feedback includes: Extracting the difference a2-a1 between the termination parsing graph and the initial parsing graph of the historical artificial intelligence task in the artificial intelligence task feedback and the difference b2-b1 between the termination parsing graph and the initial parsing graph of the artificial intelligence task; Calculate the task coverage between AI job tasks and historical AI job tasks; When b2-b1 is equal to a2-a1 or the task coverage is equal to the coverage threshold, the generated test instance requires that the category of the test instance is a task with similar structure; When b2-b1 belongs to a subset of a2-a1 or the task coverage is less than the coverage threshold, the generated test instance requires that the category of the test instance is a simplified task; When b2-b1 contains a2-a1 or the task coverage is greater than the coverage threshold, the generated test instance requires that the category of the test instance is a transcendental task.

[0013] An embodiment of the present invention provides a task adaptive generation system based on loop feedback, including: The artificial intelligence job task generation module is used to execute step S1: generate an artificial intelligence job task based on the test instance list and the artificial intelligence job feedback; The artificial intelligence operation task processing module is used to execute step S2: process the artificial intelligence operation task and obtain the artificial intelligence operation result; The step return module is used to execute step S3: return the artificial intelligence job result to step S1 as new artificial intelligence job feedback; The iterative loop module is used to execute step S4: iteratively loop execute steps S1 to S3.

[0014] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and a processor executes the computer program to implement any of the above methods.

[0015] An embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any one of the methods described above.

[0016] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0017] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 Schematic diagram of a task adaptive generation method based on loop feedback in an embodiment of the present invention; Figure 2 Schematic diagram of a characteristic analysis diagram in an embodiment of the present invention; Figure 3 It is a system application framework diagram in an embodiment of the present invention; Figure 4 Schematic diagram of a task adaptive generation system based on loop feedback in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0020] The embodiment of the present invention provides a task adaptive generation method based on loop feedback, such as Figure 1 As shown, including: Step S1: Generate an AI job task based on the test instance list and AI job feedback; Step S2: Processing the AI ​​operation task and obtaining the AI ​​operation result; Step S3: returning the AI ​​job result as new AI job feedback to step S1; Step S4: Iterate and loop through steps S1 to S3 to achieve a comprehensive evaluation of the general artificial intelligence model and use the test results to guide model training.

[0021] The steps for obtaining the test instance list are as follows: Collect test tasks; Based on the task template definition standard, the test task is modeled to obtain the task template library; Match the task templates in the task template library with the test environments in the test environment library to determine the test instance list; The collection and testing tasks include: Crawl test tasks based on the test benchmark of the existing model; and / or, Receive test tasks input by users; The task template definition standard at least includes: nodes, edges and attributes of the task parsing graph.

[0022] In the above technical solution, a list of test instances is first made. When making the list, test tasks are collected. There are two ways to collect test tasks. The first is to crawl based on the test benchmark of an existing model. An existing model refers to an existing artificial intelligence model, such as a visual detection model. An existing model will have a test benchmark. For example, when performing visual detection, crawling to obtain test tasks, such as a model visual detection capability test task, etc. Another example is that the existing model is a visual classification model, a depth prediction model, etc. The model will have a test benchmark, such as the classification accuracy on a specific data set when performing visual classification, and the depth prediction accuracy when performing depth prediction. The test tasks obtained by crawling are, for example: the test task of the classification model is to classify on a specific data set, etc.; the second type is provided by the user, who can be a technician, etc.; then, the collected test tasks are modeled based on the task template definition standard. The task template definition standard is based on graph theory and hypergraph theory, and defines the three basic elements of the feature parsing graph: nodes, edges, and attributes. When modeling the test task, the test task is converted into a feature parsing graph to obtain a task template. The feature parsing graph can represent various artificial intelligence tasks and form a unified expression framework for the task. Finally, all the obtained task templates are integrated to obtain a task template library; such as Figure 2 As shown in the figure, when the test task is converted into a feature parsing graph, it contains two feature parsing graphs: the initial state and the final state. For example, in the new task of packing luggage, the intelligent agent, room, and items are nodes. From the initial state to the target state, the attribute information such as the location of the items changes. The task template definition is based on graph theory and hypergraph theory, and requires nodes, edges, and attributes. Each task is represented by a graph structure. Specifically, each task has two parsing graphs, representing the initial state and the final state respectively. There are specific test environments in the test environment library. The task templates in the task template library are matched with the test environments in the test environment library to determine the actual test instances. The test instance list is completed by integrating all test instances.

[0023] Task generation is performed based on the test instance list and AI job feedback, i.e., AI job tasks are generated; then AI jobs are performed, i.e., AI job tasks are processed to obtain AI job results; the AI ​​job results are repeatedly executed as new AI job feedback to form an iterative loop.

[0024] like Figure 3As shown, when the present application is applied, a task generation model and an artificial intelligence job model are provided. In the task generation model, a test task collector is set to collect test tasks, and a task template library is constructed in combination with the task template definition standard. The contents of the task template library and the test environment library are matched to determine the test instance list; a test instance intelligent extractor is set to generate artificial intelligence job tasks based on the test instance list and artificial intelligence job feedback, and send them to the task generation model for processing. The task generation model processes the artificial intelligence job results and returns them to the task generation model, thereby forming a cycle to form a coupling between the task generation model and the artificial intelligence job model.

[0025] This application generates artificial intelligence job tasks based on the test instance list and artificial intelligence job feedback, then processes the artificial intelligence job tasks, obtains artificial intelligence job results, and then uses the artificial intelligence job results as new artificial intelligence job feedback to regenerate artificial intelligence job tasks in this cycle, thereby realizing automatic task generation, making dynamic adjustments based on the artificial intelligence job results, avoiding mechanically listing a single type of questions, and effectively coupling the task generation model with the artificial intelligence job model, truly realizing automatic task generation and problem-solving cycle optimization, so as to achieve a comprehensive evaluation of the general artificial intelligence model and use test results to guide model training.

[0026] In one embodiment, matching the task templates in the task template library with the test environments in the test environment library to determine the test instance list includes: Get the environmental information of the test environment; Construct the mapping relationship between the test environment information and the task template; Based on the mapping relationship, the environmental information of the test environment is added to the task template accordingly to obtain the test instance; Integrate test instances and obtain a list of test instances.

[0027] In the above technical scheme, the environmental information of the test environment is, for example, two-dimensional color image, location coordinates, attribution and other attribute information; a mapping relationship between the environmental information of the test environment and the task template is constructed, that is, a mapping relationship between the environmental information and the nodes, edges and attributes in the task template is constructed, that is, a mapping relationship between the environmental information and the nodes, edges and attributes in the task template is constructed, and the task template represents abstract information (such as "object", "agent"), and the mapping relationship indicates which specific content these abstract information can correspond to (such as "cup 1", "bread 3" and "object" form a mapping; "ChatGPT" and "agent" form a mapping); based on the mapping relationship, the environmental information of the test environment is added to the task template accordingly. When adding, the environmental information is mounted under the nodes and edges in the task template with which it has a mapping relationship, and finally the mounted content replaces the original nodes and edges, such as replacing the abstract node "object" with the specific node "bread 3" (if multiple contents are mounted, one is randomly selected) to obtain a test instance.

[0028] In one embodiment, the step S1: generating an artificial intelligence job task based on the test instance list and artificial intelligence job feedback includes: Extract test instances from the test instance list; Generate artificial intelligence tasks based on test instances; The step of extracting a test instance from the test instance list includes: When the AI ​​job feedback is empty, a test instance is extracted from the test instance list based on a random algorithm; When the artificial intelligence job feedback is not empty, determining the test instance requirement based on the artificial intelligence job feedback, and extracting the test instance from the test instance list based on the test instance requirement; Wherein, determining the test instance requirements based on artificial intelligence job feedback includes: Extracting the difference a2-a1 between the termination parsing graph and the initial parsing graph of the historical artificial intelligence task in the artificial intelligence task feedback and the difference b2-b1 between the termination parsing graph and the initial parsing graph of the artificial intelligence task; Calculate the task coverage between the AI ​​job task and the historical AI job task; the task coverage represents the coverage relationship between the two graphs. Here, we select the termination analysis graph of the historical task and the termination analysis graph of the AI ​​job task to calculate the task coverage of the two.

[0029] When b2-b1 is equal to a2-a1 or the task coverage is equal to the coverage threshold, it is defined as a similar structure task, and the generated test instance requires that the category of the test instance is a similar structure task; When b2-b1 belongs to a subset of a2-a1 or the task coverage is less than the coverage threshold, it is defined as a simplified task, and the generated test instance requires that the category of the test instance is a simplified task; When b2-b1 contains a2-a1 or the task coverage is greater than the coverage threshold, it is defined as a transcendental task, and the generated test instance requires that the category of the test instance is a transcendental task.

[0030] In the above technical solution, when the artificial intelligence job feedback is empty, it means that there is no artificial intelligence job feedback for the time being, that is, the artificial intelligence job task is generated for the first time, and based on a random algorithm (for example, there are 5 items, each random selection has the same probability of each item being selected), a test instance is extracted from the test instance list; the random algorithm can be an algorithm for randomly selecting options; after the test instance is extracted, it is used as an artificial intelligence job task.

[0031] When the AI ​​job feedback is not empty, in order to make the extracted test instances adapt to the AI ​​job feedback, the requirements for the test instance categories are determined based on the AI ​​job feedback (three categories in total: similar structure tasks, simplified tasks, and transcendent tasks), and test instances are extracted from the test instance list based on the test instance requirements. Similar structure tasks represent tasks similar to historical tasks. The model is tested on a large number of similar structure tasks, and the statistical results can reflect the characteristics of the model on the structural tasks; for simplified tasks, when the model cannot give correct results on existing complex tasks, try to reduce the difficulty of the questions and extract simplified tasks from tasks with simpler graph structures; for transcendent tasks, when the model easily solves existing tasks, try to increase the complexity of the questions and extract tasks from tasks with more complex graph structures.

[0032] Among them, the calculation formula for task coverage is as follows: Among them, the task ,Task They are the termination analysis diagrams of historical tasks and the termination analysis diagrams of artificial intelligence tasks. For the task To the task The task coverage of For the task To the task The edge node coverage of For the task To the task Attribute coverage of , When it is 0, it represents the task With the task The edge node structures of are independent of each other. When it is 1, it represents the task It's a task The subgraph of When it is 2, it represents the task With the task Isomorphism, 3 o'clock represents the task It's a task subgraph of ; ; , in, Represents the task It's a task When the subgraph is calculate ; Represents the task and tasks The number of identical attributes among all attributes of ; For the task The number of attributes.

[0033] ( ) in, , Represents the task Not a task The subgraph of and The edge node structures are independent of each other. yes The sub-graph of and isomorphic), when the task Not a task When the subgraph is calculate , Represents the task and tasks The number of identical attributes among all attributes of Represents the task and tasks The total number of attributes (if there are identical attributes, they are not counted repeatedly).

[0034] The AI ​​homework feedback of the AI ​​homework model includes: a list of historical questions, the difference between the termination parsing graph and the initial parsing graph corresponding to each task in the historical questions (a2-a1), and historical answering status; AI homework feedback determines the type of the next test instance, which includes: similar structure tasks (the difference b2-b1 between the termination parsing graph of the new question and the initial parsing graph is equal to a2-a1; or the task coverage index is equal to 3), simplified tasks (the difference b2-b1 between the termination parsing graph and the initial parsing graph of the question belongs to a subset of a2-a1; or the task coverage index is less than 3), and transcendent tasks (the difference b2-b1 between the termination parsing graph and the initial parsing graph of the question contains a2-a1; or the task coverage index is greater than 3; or the parsing graph has structures or attributes that do not exist in the original task). Therefore, the test instance requirements are determined based on this.

[0035] The embodiment of the present invention provides a task adaptive generation system based on loop feedback, such as Figure 4 As shown, including: The artificial intelligence job task generation module 1 is used to execute step S1: generate an artificial intelligence job task based on the test instance list and the artificial intelligence job feedback; The artificial intelligence operation task processing module 2 is used to execute step S2: process the artificial intelligence operation task and obtain the artificial intelligence operation result; The step returns to module 3, which is used to execute step S3: returning the artificial intelligence operation result as new artificial intelligence operation feedback to step S1; The iterative loop module 4 is used to execute step S4: iteratively loopingly executes steps S1 to S3.

[0036] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. A processor executes the computer program to implement any of the above methods.

[0037] An embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any one of the methods described above.

[0038] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A task adaptive generation method based on loop feedback, characterized in that: include: Step S1: Generate an AI job task based on the test instance list and AI job feedback; Step S2: Processing the AI ​​operation task and obtaining the AI ​​operation result; Step S3: returning the AI ​​job result as new AI job feedback to step S1; Step S4: Iterate and loop through steps S1 to S3.

2. The task adaptive generation method based on loop feedback according to claim 1, characterized in that: The steps for obtaining the test instance list are as follows: Collect test tasks; Based on the task template definition standard, the test task is modeled to obtain the task template library; The task templates in the task template library are matched with the test environments in the test environment library to determine the test instance list.

3. The task adaptive generation method based on loop feedback according to claim 2, characterized in that: The collection and testing tasks include: Crawl test tasks based on the test benchmark of the existing model; and / or, A test task that receives user input.

4. The task adaptive generation method based on loop feedback according to claim 2, characterized in that: The task template definition standard includes at least: nodes, edges and attributes of the task parsing graph.

5. The task adaptive generation method based on loop feedback according to claim 2, characterized in that: The step of matching the task templates in the task template library with the test environments in the test environment library to determine the test instance list includes: Get the environmental information of the test environment; Construct the mapping relationship between the test environment information and the task template; Based on the mapping relationship, the environmental information of the test environment is added to the task template accordingly to obtain the test instance; Integrate test instances and obtain a list of test instances.

6. The task adaptive generation method based on loop feedback according to claim 1, characterized in that: The step S1: generating an artificial intelligence task based on the test instance list and artificial intelligence task feedback, includes: Extract test instances from the test instance list; Generate artificial intelligence tasks based on test instances; The step of extracting a test instance from the test instance list includes: When the AI ​​job feedback is empty, a test instance is extracted from the test instance list based on a random algorithm; When the artificial intelligence job feedback is not empty, the test instance requirement is determined based on the artificial intelligence job feedback, and the test instance is extracted from the test instance list based on the test instance requirement.

7. The task adaptive generation method based on loop feedback according to claim 6, characterized in that: Determining test instance requirements based on artificial intelligence job feedback includes: Extracting the difference a2-a1 between the termination parsing graph and the initial parsing graph of the historical artificial intelligence task in the artificial intelligence task feedback and the difference b2-b1 between the termination parsing graph and the initial parsing graph of the artificial intelligence task; Calculate the task coverage between AI job tasks and historical AI job tasks; When b2-b1 is equal to a2-a1 or the task coverage is equal to the coverage threshold, the generated test instance requires that the category of the test instance is a task with similar structure; When b2-b1 belongs to a subset of a2-a1 or the task coverage is less than the coverage threshold, the generated test instance requires that the category of the test instance is a simplified task; When b2-b1 contains a2-a1 or the task coverage is greater than the coverage threshold, the generated test instance requires that the category of the test instance is a transcendental task.

8. A task adaptive generation system based on loop feedback, characterized in that: include: The artificial intelligence job task generation module is used to execute step S1: generate an artificial intelligence job task based on the test instance list and the artificial intelligence job feedback; The artificial intelligence operation task processing module is used to execute step S2: process the artificial intelligence operation task and obtain the artificial intelligence operation result; The step return module is used to execute step S3: return the artificial intelligence job result to step S1 as new artificial intelligence job feedback; The iterative loop module is used to execute step S4: iteratively loop execute steps S1 to S3.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 7.

Images