Task Adaptive Generation Method, System, Storage Medium and Electronic Device Based on Loop Feedback

Abstract

The present invention provides a task adaptive generation method, system, storage medium and electronic device based on loop feedback. The method includes: Step S1: generating an artificial intelligence operation task based on a test instance list and artificial intelligence operation feedback; Step S2: processing the artificial intelligence operation task to obtain an artificial intelligence operation result; Step S3: returning the artificial intelligence operation result as new artificial intelligence operation feedback to Step S1; Step S4: iteratively and circularly executing Step S1 to Step S3. The present invention realizes automatic task generation, dynamically adjusts according to the artificial intelligence operation result, avoids mechanically listing single-type questions, effectively couples the generated task with the artificial intelligence operation model, and truly realizes the loop optimization of automatic task generation and problem solution.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and particularly relates to a task adaptive generation method, system, storage medium and electronic device based on loop feedback. Background Art

[0002] Currently, benchmarking is widely used in the field of artificial intelligence (AI). Typical examples include ImageNet (a benchmark dataset for image recognition) and GLUE (a benchmark for natural language understanding). Although these tests can evaluate the performance of AI models on specific tasks, they usually focus on the evaluation of a single task and cannot comprehensively examine the general intelligence of the models. Especially in terms of test question generation, these existing benchmark tests do not have the ability to generate intelligent tasks. The test questions are simply listed mechanically, usually fixed on a single type of question, and cannot be dynamically adjusted according to the results of AI operations.

[0003] For example, patent CN202410955119.8 proposes a task generation method and system. Although this invention is innovative in task generation, it fails to effectively couple the generated tasks with the AI operation model. Specifically, it fails to form an adaptive task generation and AI operation loop system by combining the solution feedback of tasks through a systematic mechanism. Therefore, although it has certain advantages in task generation, the lack of a feedback mechanism for AI operation capabilities makes it unable to achieve true automatic task generation and loop optimization of problem solving.

[0004] Therefore, a solution is urgently needed. Summary of the Invention

[0005] One of the objectives of the present invention is to provide a task adaptive generation method based on loop feedback. Based on a list of test instances and AI operation feedback, AI operation tasks are generated, then the AI operation tasks are processed to obtain AI operation results. Next, the AI operation results are used as new AI operation feedback to regenerate AI operation tasks in this cycle, realizing automatic task generation, dynamically adjusting according to the AI operation results, avoiding mechanically listing single-type questions, effectively coupling the generated tasks with the AI operation model, and truly realizing automatic task generation and loop optimization of problem solving.

[0006] The task adaptive generation method based on loop feedback provided by the embodiments of the present invention includes:

[0007] Step S1: Generate AI operation tasks based on a list of test instances and AI operation feedback;

[0008] Step S2: Process the artificial intelligence job task to obtain the artificial intelligence job result;

[0009] Step S3: Use the artificial intelligence job result as a new artificial intelligence job feedback and return to Step S1;

[0010] Step S4: Iteratively execute Steps S1 to S3 in a loop.

[0011] Optionally, the steps for obtaining the test instance list are as follows:

[0012] Collect test tasks;

[0013] Based on the task template definition criteria, model the test tasks to obtain a task template library;

[0014] Match the task templates in the task template library with the test environments in the test environment library to determine the test instance list.

[0015] Optionally, the collecting of test tasks includes:

[0016] Crawl test tasks according to the test benchmarks of existing models;

[0017] And / or,

[0018] Receive test tasks input by users.

[0019] Optionally, the task template definition criteria at least include: nodes, edges, and attributes of the task parsing graph.

[0020] Optionally, the matching of the task templates in the task template library with the test environments in the test environment library to determine the test instance list includes:

[0021] Obtain the environmental information of the test environment;

[0022] Build a mapping relationship between the environmental information of the test environment and the task template;

[0023] Based on the mapping relationship, add the environmental information of the test environment into the task template accordingly to obtain test instances;

[0024] Integrate the test instances to obtain the test instance list.

[0025] Optionally, Step S1: Generating an artificial intelligence job task based on the test instance list and the artificial intelligence job feedback includes:

[0026] Extract test instances from the test instance list;

[0027] Generate an artificial intelligence job task based on the test instances;

[0028] Among them, extracting test instances from the test instance list includes:

[0029] When the artificial intelligence job feedback is empty, extract test instances from the test instance list based on a random algorithm;

[0030] When the artificial intelligence job feedback is not empty, determine the test instance requirements based on the artificial intelligence job feedback, and extract test instances from the test instance list based on the test instance requirements.

[0031] Optionally, determining the test instance requirements based on the artificial intelligence job feedback includes:

[0032] Extract the difference a2 - a1 between the termination parsing graph and the initial parsing graph of the historical artificial intelligence job task and the difference b2 - b1 between the termination parsing graph and the initial parsing graph of the artificial intelligence job task in the artificial intelligence job feedback;

[0033] Calculate the task coverage between the artificial intelligence job task and the historical artificial intelligence job task;

[0034] When b2 - b1 is equal to a2 - a1 or the task coverage is equal to the coverage threshold, generate the test instance requirement that the category of the test instance is a similar structure task;

[0035] When b2 - b1 belongs to the subset of a2 - a1 or the task coverage is less than the coverage threshold, generate the test instance requirement that the category of the test instance is a simplified task;

[0036] When b2 - b1 contains a2 - a1 or the task coverage is greater than the coverage threshold, generate the test instance requirement that the category of the test instance is a transcendent task.

[0037] A task adaptive generation system based on loop feedback provided by an embodiment of the present invention includes:

[0038] An artificial intelligence job task generation module, configured to execute step S1: generate an artificial intelligence job task based on the test instance list and the artificial intelligence job feedback;

[0039] An artificial intelligence job task processing module, configured to execute step S2: process the artificial intelligence job task to obtain an artificial intelligence job result;

[0040] A step return module, configured to execute step S3: return the artificial intelligence job result as a new artificial intelligence job feedback to step S1;

[0041] An iterative loop module, configured to execute step S4: iteratively loop to execute steps S1 to S3.

[0042] A computer-readable storage medium provided by an embodiment of the present invention has a computer program stored thereon, and a processor executes the computer program to implement the method described in any one of the above.

[0043] An electronic device provided by an embodiment of the present invention includes a memory and a processor. A computer program is stored in the memory, and the processor executes the computer program to implement the method described in any one of the above.

[0044] Other features and advantages of the present invention will be described in the following specification, and in part will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained by the structure specifically pointed out in the written specification and the drawings.

[0045] The technical solution of the present invention will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. In the drawings:

[0047] Figure 1 It is a schematic diagram of a task adaptive generation method based on loop feedback in an embodiment of the present invention;

[0048] Figure 2 It is a schematic diagram of a feature analysis diagram in an embodiment of the present invention;

[0049] Figure 3 It is a system application framework diagram in an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of a task adaptive generation system based on loop feedback in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] The following describes the preferred embodiments of the present invention with reference to the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0052] An embodiment of the present invention provides a task adaptive generation method based on loop feedback, as Figure 1 shown, including:

[0053] Step S1: Generate an artificial intelligence job task based on a test instance list and an artificial intelligence job feedback;

[0054] Step S2: Process the artificial intelligence job task to obtain an artificial intelligence job result;

[0055] Step S3: Return the artificial intelligence job result as a new artificial intelligence job feedback to Step S1;

[0056] Step S4: Iteratively loop through Steps S1 to S3; to achieve a comprehensive evaluation of the general artificial intelligence model and use the test results to guide model training.

[0057] Among them, the steps for obtaining the test instance list are as follows:

[0058] Collect test tasks;

[0059] Based on the task template definition criteria, model the test tasks to obtain a task template library;

[0060] Match the task templates in the task template library with the test environments in the test environment library to determine the test instance list;

[0061] Among them, the collection of the test tasks includes:

[0062] Crawl test tasks according to the test benchmarks of existing models;

[0063] And / or,

[0064] Receive test tasks input by users;

[0065] Among them, the task template definition criteria include at least: nodes, edges, and attributes of the task parsing graph.

[0066] In the above technical solution, first make a test instance list. When making it, collect test tasks. There are two ways to collect test tasks. The first is to crawl based on the test benchmarks of existing models. Existing models refer to existing artificial intelligence models, such as: visual detection models, etc. Existing models will have test benchmarks, such as: for visual detection, crawl to obtain test tasks, such as: model visual detection ability test tasks, etc. Another example: if the existing model is a visual classification model, depth prediction model, etc., the model will have test benchmarks, such as: the classification accuracy rate on a specific data set when performing visual classification, the depth prediction accuracy rate when performing depth prediction. Then the test tasks crawled are, for example: the test task of the classification model is to perform classification on a specific data set, etc.; the second is provided by the user independently. The user can be a technician, etc. Then, model the collected test tasks based on the task template definition criteria. The task template definition criteria are based on graph theory and hypergraph theory, and define three basic elements of the feature parsing graph: nodes, edges, attributes. When modeling the test tasks, convert the test tasks into the form of a feature parsing graph, that is, obtain the task template. The feature parsing graph can represent various artificial intelligence tasks and form a unified expression framework for tasks. Finally, integrate all the obtained task templates to obtain a task template library; asFigure 2 As shown, when a test task is transformed into a feature analysis diagram, it includes two types of feature analysis diagrams, namely the initial state and the termination state. For example, for the new task of packing luggage, the agent, the room, and the items are nodes. From the initial state to the target state, attribute information such as the location of the items changes; the task template definition is based on graph theory and hypergraph theory, and it is required to include nodes, edges, and attributes. Each task is represented by a graph structure. Specifically, each task has two analysis diagrams, respectively representing the initial state and the termination state. There are specific test environments in the test environment library. Match the task templates in the task template library with the test environments in the test environment library to determine the actual test instances; integrating all test instances means completing the production of the test instance list.

[0067] Based on the test instance list and the artificial intelligence homework feedback, generate artificial intelligence homework tasks, that is, generate artificial intelligence homework tasks; then perform artificial intelligence homework, that is, process the artificial intelligence homework tasks to obtain artificial intelligence homework results; use the artificial intelligence homework results as new artificial intelligence homework feedback and repeat the execution to form an iterative loop.

[0068] As Figure 3 As shown, when this application is applied, a task generation model and an artificial intelligence homework model are provided. In the task generation model, a test task collector is set to collect test tasks, a task template library is constructed in combination with the task template definition standard, and the contents in the task template library and the test environment library are matched with each other to determine the test instance list; a test instance intelligent extractor is set, which is used to generate artificial intelligence homework tasks based on the test instance list and the artificial intelligence homework feedback, and send them to the task generation model for processing. The task generation model processes to obtain the artificial intelligence homework results and then returns them to the task generation model. In this way, a cycle is formed to form the coupling of the task generation model and the artificial intelligence homework model.

[0069] This application generates artificial intelligence homework tasks based on the test instance list and the artificial intelligence homework feedback, then processes the artificial intelligence homework tasks to obtain artificial intelligence homework results. Then, use the artificial intelligence homework results as new artificial intelligence homework feedback to regenerate artificial intelligence homework tasks. In this way, a cycle is formed to realize automatic task generation, dynamically adjust according to the artificial intelligence homework results, avoid mechanically listing single-type questions, effectively couple the task generation model and the artificial intelligence homework model, truly realize automatic task generation and problem-solving cycle optimization, so as to comprehensively evaluate the general artificial intelligence model and use the test results to guide model training.

[0070] In one embodiment, the matching of the task templates in the task template library with the test environments in the test environment library to determine the test instance list includes:

[0071] Obtain the environmental information of the test environment;

[0072] Build the mapping relationship between the environmental information of the test environment and the task template;

[0073] Based on the mapping relationship, add the environmental information of the test environment into the task template accordingly to obtain test instances;

[0074] Integrate the test instances to obtain a list of test instances.

[0075] In the above technical solution, the environmental information of the test environment is, for example: attribute information such as two-dimensional color images, position coordinates, ownership, etc.; building the mapping relationship between the environmental information of the test environment and the task template, that is, building the mapping relationship between the environmental information and the nodes, edges, and attributes in the task template respectively, that is, building the mapping relationship between the environmental information and the nodes, edges, and attributes in the task template respectively. The task template represents abstract information (such as "object", "agent"), and the mapping relationship indicates which specific contents these abstract information can correspond to (such as "water cup 1", "bread 3" form a mapping with "object"; "ChatGPT" forms a mapping with "agent"); based on the mapping relationship, add the environmental information of the test environment into the task template accordingly. When adding, mount the environmental information under the nodes and edges in the task template with which it has a mapping relationship, and finally replace the original nodes and edges with the mounted content, such as replacing the abstract node "object" with the specific node "bread 3" (if multiple items are mounted, randomly select one) to obtain test instances.

[0076] In one embodiment, step S1: Generate an artificial intelligence operation task based on the test instance list and the artificial intelligence operation feedback, including:

[0077] Extract test instances from the test instance list;

[0078] Generate an artificial intelligence operation task based on the test instances;

[0079] Among them, the extracting test instances from the test instance list includes:

[0080] When the artificial intelligence operation feedback is empty, extract test instances from the test instance list based on a random algorithm;

[0081] When the artificial intelligence operation feedback is not empty, determine the test instance requirements based on the artificial intelligence operation feedback, and extract test instances from the test instance list based on the test instance requirements;

[0082] Among them, the determining the test instance requirements based on the artificial intelligence operation feedback includes:

[0083] Extract the difference a2 - a1 between the termination parsing graph and the initial parsing graph of the historical AI homework task and the difference b2 - b1 between the termination parsing graph and the initial parsing graph of the AI homework task;

[0084] Calculate the task coverage between the AI homework task and the historical AI homework task; the task coverage represents the coverage relationship between the two graphs. Here, select the termination parsing graph of the historical task and the termination parsing graph of the AI homework task to calculate their task coverage.

[0085] When b2 - b1 is equal to a2 - a1 or the task coverage is equal to the coverage threshold, define it as a similar structure task, and the requirement for generating a test instance is that the category of the test instance is a similar structure task;

[0086] When b2 - b1 is a subset of a2 - a1 or the task coverage is less than the coverage threshold, define it as a simplified task, and the requirement for generating a test instance is that the category of the test instance is a simplified task;

[0087] When b2 - b1 contains a2 - a1 or the task coverage is greater than the coverage threshold, define it as a transcendent task, and the requirement for generating a test instance is that the category of the test instance is a transcendent task.

[0088] In the above technical solution, when the AI homework feedback is empty, it means that there is no AI homework feedback temporarily, that is, the AI homework task is generated for the first time. Based on a random algorithm (for example, there are 5 items, randomly selected each time, and the probability of each item being selected is the same), extract a test instance from the test instance list; the random algorithm can be an algorithm for randomly selecting options; after extracting the test instance, immediately use it as the AI homework task.

[0089] When the AI homework feedback is not empty, in order to make the extracted test instance adapt to the AI homework feedback, based on the AI homework feedback, determine the requirements for the category of the test instance (a total of three categories: similar structure tasks, simplified tasks, transcendent tasks), and based on the test instance requirements, extract a test instance from the test instance list. Similar structure tasks represent tasks similar to historical tasks. The model is tested on a large number of similar structure tasks, and the statistical results can reflect the characteristics of the model on this type of structure task; for simplified tasks, when the model cannot give the correct result on existing complex tasks, try to reduce the difficulty of the question and extract simplified tasks from tasks with a simpler graph structure; for transcendent tasks, when the model easily solves existing tasks, try to increase the complexity of the question and extract tasks from tasks with a more complex graph structure.

[0090] Among them, the calculation formula for the task coverage is as follows:

[0091]

[0092] Among them, the task , and the task are respectively the termination analysis diagram of the historical task and the termination analysis diagram of the artificial intelligence operation task. is the task for the task task coverage. is the task for the task edge node coverage. is the task for the task attribute coverage rate; , when it is 0, it means that the task and the task are irrelevant to each other in terms of edge node structure. when it is 1, it means that the task is a subgraph of the task . when it is 2, it means that the task is isomorphic to the task . when it is 3, it means that the task is a subgraph of the task ; ;

[0093] ,

[0094] Among them, represents that when the task is a subgraph of the task , use to calculate ; represents the number of identical attributes among all attributes of the task and the task ; is the number of attributes of the task .

[0095]

[0096]

[0097] ( )

[0098] Among them, , represent that the task is not a subgraph of the task ( and The edge node structures are independent of each other, is a subgraph of and is isomorphic to), when the task is not a subgraph of the task use to calculate , represents the number of identical attributes among all the attributes of the task and the task , represents the total number of attributes of the task and the task (if there are identical attributes, they are not counted repeatedly).

[0099] The AI homework feedback of the AI homework model includes: a list of historical questions, the difference (a2 - a1) between the termination parsing graph and the initial parsing graph corresponding to each task in the historical questions, and the historical answering situation; the AI homework feedback determines the type of the next test instance, and the types include: similar structure tasks (the difference b2 - b1 between the termination parsing graph and the initial parsing graph of the new question is equal to a2 - a1; or the task coverage index is equal to 3), simplified tasks (the difference b2 - b1 between the termination parsing graph and the initial parsing graph of the question is a subset of a2 - a1; or the task coverage index is less than 3), and transcendent tasks (the difference b2 - b1 between the termination parsing graph and the initial parsing graph of the question contains a2 - a1; or the task coverage index is greater than 3; or the parsing graph has a structure or attribute that does not exist in the original task). Therefore, the requirements for the test instance are determined based on this.

[0100] An embodiment of the present invention provides a task adaptive generation system based on cyclic feedback, as Figure 4 shown, including:

[0101] An AI homework task generation module 1, configured to execute step S1: generate an AI homework task based on the test instance list and the AI homework feedback;

[0102] An AI homework task processing module 2, configured to execute step S2: process the AI homework task to obtain an AI homework result;

[0103] A step return module 3, configured to execute step S3: return the AI homework result as a new AI homework feedback to step S1;

[0104] An iterative loop module 4, configured to execute step S4: iteratively loop through steps S1 to S3.

[0105] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and a processor executes the computer program to implement the method described in any one of the above.

[0106] An embodiment of the present invention provides an electronic device, which includes a memory and a processor. A computer program is stored in the memory, and the processor executes the computer program to implement the method described in any one of the above.

[0107] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A task adaptive generation method based on cyclic feedback, characterized in that Including: Step S1: Generate an artificial intelligence homework task based on a test instance list and artificial intelligence homework feedback; Step S2: Process the artificial intelligence homework task to obtain an artificial intelligence homework result; Step S3: Return the artificial intelligence homework result as new artificial intelligence homework feedback to Step S1; Step S4: Iteratively execute Steps S1 to S3 in a loop; The said Step S1: Generate an artificial intelligence homework task based on a test instance list and artificial intelligence homework feedback, including: Extract test instances from the test instance list; Generate an artificial intelligence homework task based on the test instances; Among them, the said extracting test instances from the test instance list includes: When the artificial intelligence homework feedback is empty, extract test instances from the test instance list based on a random algorithm; When the artificial intelligence homework feedback is not empty, determine the test instance requirements based on the artificial intelligence homework feedback, and extract test instances from the test instance list based on the test instance requirements; The said determining the test instance requirements based on the artificial intelligence homework feedback includes: Extract the difference a2 - a1 between the termination parsing graph and the initial parsing graph of the historical artificial intelligence homework task and the difference b2 - b1 between the termination parsing graph and the initial parsing graph of the artificial intelligence homework task in the artificial intelligence homework feedback; Calculate the task coverage between the artificial intelligence homework task and the historical artificial intelligence homework task; When b2 - b1 is equal to a2 - a1 or the task coverage is equal to the coverage threshold, generate the test instance requirement that the category of the test instance is a similar structure task; When b2 - b1 belongs to the subset of a2 - a1 or the task coverage is less than the coverage threshold, generate the test instance requirement that the category of the test instance is a simplified task; When b2 - b1 contains a2 - a1 or the task coverage is greater than the coverage threshold, generate the test instance requirement that the category of the test instance is a transcendent task.

2. The task adaptive generation method based on loop feedback according to claim 1, wherein The obtaining steps of the said test instance list are as follows: Collect test tasks; Model the test tasks based on the task template definition criteria to obtain a task template library; Match the task templates in the task template library with the test environments in the test environment library to determine the test instance list.

3. The task adaptive generation method based on loop feedback according to claim 2, characterized in that, The said collecting test tasks includes: Crawl test tasks according to the test benchmarks of existing models; And / or Receive test tasks input by users.

4. The task adaptive generation method based on loop feedback according to claim 2, wherein The said task template definition criteria at least include: nodes, edges, and attributes of the task parsing graph.

5. The task adaptive generation method based on loop feedback according to claim 2, wherein The said matching the task templates in the task template library with the test environments in the test environment library to determine the test instance list includes: Obtain the environmental information of the test environment; Construct a mapping relationship between the environmental information of the test environment and the task templates; Based on the mapping relationship, add the environmental information of the test environment into the task templates accordingly to obtain test instances; Integrate the test instances to obtain the test instance list.

6. A task adaptive generation system based on loop feedback, characterized in that, Including: An artificial intelligence homework task generation module, used to execute Step S1: Generate an artificial intelligence homework task based on a test instance list and artificial intelligence homework feedback; An artificial intelligence homework task processing module, used to execute Step S2: Process the artificial intelligence homework task to obtain an artificial intelligence homework result; A step return module, configured to execute step S3: Return the artificial intelligence job result as a new artificial intelligence job feedback to step S1; An iterative loop module, configured to execute step S4: Iteratively loop to execute steps S1 to S3; The step S1: Generate an artificial intelligence job task based on the test instance list and the artificial intelligence job feedback, including: Extract test instances from the test instance list; Generate an artificial intelligence job task based on the test instances; Among them, the extracting test instances from the test instance list includes: When the artificial intelligence job feedback is empty, extract test instances from the test instance list based on a random algorithm; When the artificial intelligence job feedback is not empty, determine the test instance requirements based on the artificial intelligence job feedback, and extract test instances from the test instance list based on the test instance requirements; The determining the test instance requirements based on the artificial intelligence job feedback includes: Extract the difference a2 - a1 between the termination parsing graph and the initial parsing graph of the historical artificial intelligence job task and the difference b2 - b1 between the termination parsing graph and the initial parsing graph of the artificial intelligence job task in the artificial intelligence job feedback; Calculate the task coverage between the artificial intelligence job task and the historical artificial intelligence job task; When b2 - b1 is equal to a2 - a1 or the task coverage is equal to the coverage threshold, generate the test instance requirement that the category of the test instance is a similar structure task; When b2 - b1 belongs to the subset of a2 - a1 or the task coverage is less than the coverage threshold, generate the test instance requirement that the category of the test instance is a simplified task; When b2 - b1 contains a2 - a1 or the task coverage is greater than the coverage threshold, generate the test instance requirement that the category of the test instance is a transcendent task.

7. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and the processor executes the computer program to implement the method according to any one of claims 1 - 5.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor. A computer program is stored in the memory, and the processor executes the computer program to implement the method according to any one of claims 1 - 5.

Images