Corpus processing method and device based on normal form evolution, equipment, medium and product

Abstract

The invention discloses a corpus processing method and device based on normal form evolution, equipment, a storage medium and a program product, and relates to an artificial intelligence technology. The method comprises the steps of obtaining a knowledge graph of an instruction sample set of a target field; the knowledge graph comprises a task network of each instruction sample in the instruction sample set; extracting a task normal form network from the knowledge graph, wherein the task normal form network is a common network in task networks of the same task type; the task paradigm network is evolved to obtain a task paradigm evolution network; and generating a learning corpus based on the task normal form evolution network, wherein the learning corpus is used for learning the target domain model. Learning corpora are processed through an evolution mechanism based on a task normal form network, the fineness of the learning corpora is improved, dependence on a large amount of labeled data is reduced, the data preparation cost and difficulty of model learning are reduced, the learning corpora are used for learning of a target domain model, and the learning efficiency is improved. And the understanding capability of the model on field tasks with high specialty and focusing performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a corpus processing method, device, equipment, medium and product based on paradigm evolution. Background Art

[0002] In the development of artificial intelligence, models have become a hot topic of research due to their powerful learning capabilities and wide range of application scenarios. However, the models lack sufficient adaptability and generalization capabilities when performing domain tasks, which limits the application of models in changeable and complex industry fields.

[0003] Existing methods for improving model capabilities are mainly model-centric, such as direct fitting, guidance, and distillation. However, the lack of meticulous management of corpus often leads to insufficient understanding of professional and focused domain tasks, and requires a large amount of domain annotated data, resulting in high model training costs. Summary of the invention

[0004] The present invention provides a corpus processing method, device, equipment, medium and product based on paradigm evolution to solve the problem that the existing model capability improvement methods are mainly model-centric, the management of corpus is not meticulous, resulting in insufficient precision and depth of understanding of professional and focused field tasks, and requiring a large amount of field annotated data, resulting in high model training costs.

[0005] In a first aspect, an embodiment of the present invention provides a corpus processing method based on paradigm evolution, comprising:

[0006] Acquire a knowledge graph of an instruction sample set in a target domain; the knowledge graph includes a task network of each instruction sample in the instruction sample set;

[0007] Extracting a task paradigm network from the knowledge graph, wherein the task paradigm network is a common network among task networks of the same task type;

[0008] Evolving the task paradigm network to obtain a task paradigm evolution network;

[0009] A learning corpus is generated based on the task paradigm evolution network, and the learning corpus is used for learning a target domain model.

[0010] In a second aspect, an embodiment of the present invention provides a corpus processing method based on paradigm evolution, comprising:

[0011] A knowledge graph acquisition module, used to acquire a knowledge graph of an instruction sample set in a target field; the knowledge graph includes a task network of each instruction sample in the instruction sample set;

[0012] A network extraction module, used to extract a task paradigm network from the knowledge graph, wherein the task paradigm network is a common network among task networks of the same task type;

[0013] A network evolution module, used for evolving the task paradigm network to obtain a task paradigm evolution network;

[0014] The corpus generation module is used to generate learning corpus based on the task paradigm evolution network, and the learning corpus is used for learning the target domain model.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the corpus processing method based on paradigm evolution described in any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the corpus processing method based on paradigm evolution described in any embodiment of the present invention when executed.

[0020] In a fifth aspect, an embodiment of the present invention provides a computer program product including a computer program, which, when executed by a processor, implements the corpus processing method based on paradigm evolution described in any embodiment of the present invention.

[0021] The technical solution of the embodiment of the present invention is to obtain the knowledge graph of the instruction sample set of the target domain; the knowledge graph includes the task network of each instruction sample in the instruction sample set; extract the task paradigm network from the knowledge graph, the task paradigm network is the common network in the task network of the same task type; evolve the task paradigm network to obtain the task paradigm evolution network; generate learning corpus based on the task paradigm evolution network, and the learning corpus is used for learning the target domain model. The instruction sample is processed by the evolution mechanism based on the task paradigm network to obtain the learning corpus, which improves the precision of the learning corpus. The existing model capability improvement method is mainly model-centered, and the management of the corpus is not refined, resulting in insufficient understanding of the precision and depth of professional and focused domain tasks, and requires a large amount of domain annotation data, resulting in the problem of high model training cost. It has the effect of reducing the dependence on a large amount of annotation data, reducing the data preparation cost and difficulty of model learning, and using the learning corpus for the learning of the target domain model has the beneficial effect of improving the model's understanding ability of professional and focused domain tasks and improving the generalization ability of the model.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of a corpus processing method based on paradigm evolution provided by an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of the structure of a corpus processing device based on paradigm evolution provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of the structure of an electronic device for implementing the corpus processing method based on paradigm evolution according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Figure 1 The flowchart of a corpus processing method based on paradigm evolution provided by an embodiment of the present invention is applicable to the case of processing learning corpus for model learning based on paradigm evolution. The method can be executed by a corpus processing device based on paradigm evolution. The corpus processing device based on paradigm evolution can be implemented in the form of hardware and / or software. The corpus processing device based on paradigm evolution can be configured in an electronic device. Figure 1 As shown, the method includes:

[0030] S110, obtaining a knowledge graph of an instruction sample set in a target domain; the knowledge graph includes a task network of each instruction sample in the instruction sample set.

[0031] Among them, the target field can be one or more actual application fields, such as finance, science and technology, electricity, and life services, etc., and the embodiments of the present invention are not limited to this. The instruction sample set can be understood as a set of instruction samples, and the instruction samples can include sample questions and sample answers. In order for the large model to better understand the intention of the instruction, the instruction sample can also include a chain of thought (CoT). The chain of thought can be understood as inserting an intermediate reasoning step between the question and the answer, and improving the performance of large language models in complex reasoning tasks by simulating the step-by-step thinking process of humans when solving problems.

[0032] The knowledge graph can be understood as a relational network obtained by connecting various types of information contained in the instruction sample set. The knowledge graph includes the task network of each instruction sample in the instruction sample set. The task network can be understood as a relational network composed of task events based on the instruction sample and the relationship between each task event. The task network can be specifically represented as a task execution workflow represented by nodes and edges. Nodes can be task events, and edges can be relationships between events. Nodes and edges constitute the network structure of the task network.

[0033] Specifically, the instruction sample set in the target field is analyzed, the task execution information in each instruction sample is extracted, the task network of the instruction sample is constructed according to the task execution information, and the task network of each instruction sample is stored in the knowledge graph.

[0034] S120. Extract a task paradigm network from the knowledge graph; the task paradigm network is a common network among task networks of the same task type.

[0035] Among them, the task paradigm network can be understood as a network composed of task events that can perform specific tasks or have certain specific execution capabilities.

[0036] Specifically, the task network in the knowledge graph is classified and analyzed to obtain task networks of multiple task types; and the common network in the task networks of the same task type is determined as the task paradigm network.

[0037] Exemplarily, frequent subgraph mining or GNN and subgraph matching techniques are used to compare the nodes, edges and network structures of different task networks to determine the common network in task networks of the same task type, and the common network is determined as the task paradigm network.

[0038] It is understandable that in the process of building a knowledge graph, only one node needs to be built for the same event, and there is no need to build nodes repeatedly. This is essentially a process of compressing the model's ability to perform different tasks. In the process of determining the task paradigm network based on the task network in the knowledge graph, the model's capabilities are once again clustered and compressed.

[0039] S130, evolving the task paradigm network to obtain a task paradigm evolution network.

[0040] Among them, the task paradigm evolution network can be understood as the task paradigm network obtained through capability evolution. The task paradigm network can be considered as the core capability operator of the large model to complete different tasks, and is the "meta-capability" of the large model. The capability evolution of the task paradigm network is a method that can improve the capability of the large model from the essence, core and fundamental, and can enable the model to have better generalization ability.

[0041] Specifically, the task paradigm network can be evolved from at least one dimension of the node operation, node link and network structure of the task paradigm network, so that the model's capability range and performance accuracy can be further improved. The evolution method can include rule-based evolution or evolution using a large model. In order to ensure the correctness of the evolution result, an audit mechanism can also be introduced to verify the evolution result.

[0042] S140. Generate learning corpus based on the task paradigm evolution network, and use the learning corpus for learning the target domain model.

[0043] The learning corpus can be understood as a data set used to fine-tune the basic large model. The target domain model can be understood as a model learned by fine-tuning the learning corpus of the target domain. The target domain model can be a deep learning model or a large model.

[0044] Exemplarily, the method of generating learning corpus can be a rule-based generation method, a corpus generation template-based method, or a large model technology-based generation method. The generated learning corpus can be processed and stored in a structured corpus to facilitate subsequent learning and verification of the model, improve the generalization ability of the model, enable the target domain model to form a systematic and complete domain task processing capability, and improve the professionalism and efficiency of the model in a specific field.

[0045] The technical solution of the embodiment of the present invention is to obtain the knowledge graph of the instruction sample set of the target domain; the knowledge graph includes the task network of each instruction sample in the instruction sample set; extract the task paradigm network from the knowledge graph, the task paradigm network is a common network in the task network of the same task type; evolve the task paradigm network to obtain the task paradigm evolution network; generate learning corpus based on the task paradigm evolution network, and the learning corpus is used for learning the target domain model. The instruction samples are processed by the evolution mechanism based on the task paradigm network to obtain the learning corpus, which improves the precision of the learning corpus, reduces the dependence on a large amount of labeled data, reduces the data preparation cost and difficulty of model learning, and uses the learning corpus for the learning of the target domain model, which improves the model's ability to understand professional and focused domain tasks and improves the generalization ability of the model.

[0046] As an optional embodiment of the embodiment of the present invention, S130, evolving the task paradigm network to obtain a task paradigm evolution network, includes: evolving at least one of the node operations, node links and network structures of the task paradigm network to obtain a task paradigm evolution network.

[0047] The task paradigm network includes nodes and node links. Nodes are used to represent task events, and events can be understood as the execution steps of tasks. The attributes of nodes include the input object, output object, and operation of the event; Operation (OP for short) is used to describe the specific processing method and process of the event, which can be a text description, or an API interface or tool call. The node link (LINK) represents the relationship between the input object and the output object of the node, that is, link = {input->output}.

[0048] Specifically, the evolution of the task paradigm network may include evolving at least one of the node operation, node link, and network structure of the task paradigm network. The evolution of the node operation of the task paradigm network may be referred to as OP evolution, the evolution of the node link of the task paradigm network may be referred to as LINK evolution, and the evolution of the network structure of the task paradigm network may be referred to as structure evolution.

[0049] Both OP evolution and LINK evolution will change the network structure of the task paradigm network, which can be considered as a passive structural change of the task paradigm network. The purpose of structural evolution is to generate a new task paradigm network by actively changing the network structure to give the task paradigm network new functions. Of course, the process of structural evolution will inevitably generate corresponding changes to the task paradigm network OP and LINK.

[0050] In an optional embodiment, after evolving the task paradigm network and obtaining the task paradigm evolution network, the task paradigm evolution network can also be input into the network verification module to verify the feasibility of the task paradigm evolution network. The verification content includes but is not limited to checking the consistency, integrity and execution efficiency of the network, and outputting the evolved and verified task paradigm network. For the network that passes the feasibility verification, it is output as a valid result after evolution. For the network that fails the verification, the reason for the failure needs to be recorded, and it may be necessary to return to the corresponding evolution step for adjustment. The technical means for implementing verification in this process include but are not limited to: using automated testing tools for batch verification to improve verification efficiency and accuracy; using model verification methods to ensure the logical correctness of each node and task process; using performance evaluation tools to measure the execution efficiency of the network and identify possible performance bottlenecks.

[0051] As an optional embodiment of the present invention, the node operations in the task paradigm network are evolved, including: without changing the input object and output object of the node, performing at least one evolutionary operation of node operation splitting, node operation merging and node operation modification on the node operations in the task paradigm network.

[0052] Specifically, for the evolution of node operations in the task paradigm network, the node operation is mainly evolved without changing the node link link={input->output} relationship, that is, keeping the input object and output object of the node unchanged and not affecting the upstream and downstream nodes. OP evolution includes: node operation splitting (abbreviated as OP splitting), node operation merging (abbreviated as OP merging) and node operation modification (abbreviated as OP modification); OP modification may include OP deletion. Node operation splitting is used to split the original node operation into multiple intermediate node operations without changing the input object and output object of the node. Node operation merging is used to merge multiple original node operations into a comprehensive node operation without changing the input object and output object of the node. Node operation modification is used to modify the original node operation to the target node operation without changing the input object and output object of the node.

[0053] It should be noted that there is no requirement for consistency in the functions of the task paradigm network before and after OP evolution, that is, the task paradigm network can generate new capabilities through OP evolution.

[0054] In a specific example, for node-operation splitting, the nodes in the task paradigm network are represented as {input->OP org ->output}, without changing the input object input and the output object output of the node, the node after OP split can be expressed as {input->[…OP i …->OP org ->…OP j …]->output}. The implementation methods of OP splitting can include: rule-based method or large model generation method. In order to ensure the correctness of the result of OP splitting, an audit mechanism can also be introduced to verify the result of OP splitting. By splitting a node operation in the task paradigm network into multiple intermediate node operations, the input-output relationship of the node is ensured to remain unchanged while enhancing the details and granularity of the task workflow.

[0055] Exemplarily, the prompt information for generating OP splitting results using the large model technology may include: task description information, description information of the original node operation, description information of the OP splitting requirements, description information of the intermediate node operation, and output result format requirements, etc. The description information of the OP splitting requirements may include consistency, integrity, and invariance of input and output objects. For example, the prompt information may be expressed as: "Please play the role of a problem solution expert. Your task is to complete the node operation (OP) evolution of the task paradigm network. It is required to keep the original node input object and output object unchanged and split the OP into multiple intermediate OPs.

[0056] Original node operation (OPorg ) is: [Extract the original node operation (OP org )’s name, description, input objects, and output objects].

[0057] The OP split follows the following requirements:

[0058] (1) Define the intermediate node operation (OP i and OP j ):OP i : Description OP org Previous node operation part; OP j : Description OP org The node operation part follows.

[0059] Generates an intermediate node operation (OP) that logically fits between the original input and output i and OP j ), ensuring that the final input->output relationship remains unchanged.

[0060] (2) Ensure consistency and integrity:

[0061] Verify the intermediate node operation (OP i and OP j ) Whether the consistency and integrity of the task process is maintained;

[0062] Make sure that the new sequence {input->[...OP i ...->OP org ->...OP j ...]->output} is logical and supports the original workflow.

[0063] (3) Detailed description of node operations:

[0064] Provides intermediate node operations (OP i and OP j ), including its methods and processes. The description can be in text format, API interface, tool call, etc.

[0065] Output result format requirements: List the name, description, input object and output object of each node operation in sections.

[0066] Please perform the evolution operation of OP split according to the above rules. "

[0067] In another specific example, for node-operation merging, a node in the task paradigm network can be represented as {input1->OP org1 ->output1}, {input2->OP org2 ->output2}, {input3->OPorg3 ->output3}, ..., each original node operation OP org Interconnected, that is, there are edges between nodes, and the output object of the upstream node is the input of the connected downstream node. Under the premise of not changing the input object and output object of the node, each node operates OP org1 ,OP org2 ,OPorg3,… can be merged into a more streamlined and efficient OP i , the node after OP split can be represented as {input i ->OP i =[OP org1 ,OP org2 ,OP org3 ,…]->output i},input i ∈{input1,input2,input3,…},output i ∈{output1,output2,output3…}. The implementation of OP merging can include: rule-based method or large model generation method. In order to ensure the correctness of the result of OP merging, an audit mechanism can also be introduced to verify the result of OP merging. By merging multiple node operations in the task paradigm network into a comprehensive node operation, the input-output relationship of the node is ensured to remain unchanged while improving the task processing efficiency.

[0068] Exemplarily, the prompt information of the OP merge result generated by the large model technology may include: task description information, description information of the original node operation, description information of the OP merge requirement, description information of the integrated node operation and output result format requirements, etc. The OP merge requirement description information may include consistency, integrity and invariance of input and output objects. For example, the prompt information may be expressed as:

[0069] “Please play the role of a problem-solving expert. Your task is to complete the node operation evolution of the task paradigm network. You are required to keep the input and output objects of the nodes unchanged and transform the interconnected original node operations (OP org 1,OP org2 ,O Porg3 ,...) into a more streamlined and efficient integrated node operation (OP i ). Now we need to handle the relationship between multiple nodes. Please follow the steps below:

[0070] (1) Analyze the original node operation:

[0071] Confirm all original node operations to be merged (OP org1 ,OPorg2 ,OP org3 ,...), including their respective names, descriptions, input objects, and output objects; verify whether these node operations are indeed interconnected, that is, the output of one node operation serves as the input of the next node operation.

[0072] (2) Define the comprehensive node operation (OP i ):

[0073] Generate a new comprehensive node operation (OP i ), which is used to replace the corresponding multiple original node operations while maintaining the same input->output relationship.

[0074] Ensure OP i The design takes into account the functions of all original node operations and can implement these functions in a more efficient way.

[0075] (3) Ensure consistency:

[0076] Verify the merged OP i Whether the consistency and integrity of the task process is maintained.

[0077] Ensure that the new synthesis node operation {input->OP i =[OP org1 , OP org2 , OP org3 , …]->output} is logical and supports the original workflow.

[0078] (4) Describe the operation in detail:

[0079] Provides comprehensive node operations (OP i ), including methods and processes, which can also be in text format, API interface or tool call, etc.

[0080] The output result format requirements are as follows: list the name, description, input object and output object of each node operation in sections, as well as the name, description, input object, output object and combined node operation of the comprehensive operation node.

[0081] Please perform the evolution operations merged by OP according to the above rules."

[0082] In another specific example, for node operation modification, a node in the task paradigm network can be represented as {input->OP org ->output}, without changing the input object input and the output object output of the node, the modified node operation can be expressed as {input->OP new->output}. The implementation methods of OP modification can include: rule-based method or large model generation method. In order to ensure the correctness of the results of OP modification, an audit mechanism can also be introduced to verify the results of OP modification. By modifying multiple node operations in the task paradigm network into target node operations, the input-output relationship of the node is ensured to remain unchanged while optimizing the network structure and improving the efficiency of task execution.

[0083] Exemplarily, the prompt information of the OP modification result generated by the large model technology may include: task description information, description information of the original node operation, description information of the OP modification requirements, description information of the target node operation, and output result format requirements, etc. The description information of the OP modification requirements may include consistency, integrity, and invariance of input and output objects. For example, the prompt information may be expressed as:

[0084] “Please play the role of a problem-solving expert. Your task is to complete the operation (OP) evolution of the task paradigm network node, which requires keeping the input and output objects of the node unchanged and performing the operation (OP) on the original node. org ) to modify and form the target node operation (OP new ). Now we need to handle the specific operations in a single step. Please follow the following rules:

[0085] (1) Extract and analyze the original node operation (OP) in detail org ): [name, description, input object, and output object]. Confirm the role of this node operation in the task flow and its relationship with other node operations.

[0086] (2) Define the target node operation (OP new ):

[0087] Generate a target node operation (OP) that is logically suitable to replace the original node operation new ), ensuring that the final input->output relationship remains unchanged. OP new It should be possible to improve efficiency, reduce complexity or enhance functionality without changing the original input and output links.

[0088] (3) Ensure consistency:

[0089] Verify the target node operation (OP new ) Whether the consistency and integrity of the task process is maintained.

[0090] Make sure {input->OP new ->output} is logical and supports the original workflow.

[0091] (4) Describe the operation in detail:

[0092] Provides target node operation (OP new ), including its methods and processes. The description can be in text format, API interface, tool call, etc.

[0093] The output result format is as follows: List the name, description, input object, and output object of the original node operation and the target node operation in sections. "

[0094] As an optional embodiment of the embodiment of the present invention, evolving the node links in the task paradigm network includes: modifying the description of the input objects and output objects of the nodes in the task paradigm network, and establishing evolved node links.

[0095] Specifically, by modifying the description of the input object and output object of the node in the task paradigm network, new associations can be generated between the nodes, that is, new execution steps in the corresponding business. Based on the evolution results of the node links, type conversion nodes can be added to achieve data matching of the input object and the output object. In addition, based on the data matching relationship after the change of the input object and the output object, the evolved LINK relationship is established, and new edges are added to the graph accordingly.

[0096] Exemplarily, the prompt information for generating the node link evolution result using the large model technology may include: description information of the target node, description information of the LIN evolution requirements, output result format requirements, etc. The description information of the LIN evolution requirements may include consistency and completeness. For example, the prompt information may be expressed as:

[0097] “Please play the role of a problem-solving expert. Your task is to complete the link evolution of the task paradigm network nodes. The goal is to modify the description of the input and output of the nodes in the task paradigm network to create new associations between the nodes. Please follow the following rules:

[0098] (1) Identify the target node:

[0099] Extract the nodes in the task paradigm network that need to modify the input or output, including the node name, description, current input and output.

[0100] (2) Define the evolved link:

[0101] Generate new input and output descriptions for the target node, ensuring that these links can form new logical associations between nodes.

[0102] (3) Ensure consistency:

[0103] Verify whether the evolved links maintain the consistency and integrity of the task process; ensure that the evolved link relationships are logical and support the improvement of the original workflow.

[0104] (4) Detailed description link:

[0105] Provide a detailed description of the evolved link, including its methods and processes. The description can be in text format, API interface, tool call, etc.

[0106] The output result format requirements are as follows: list the name, description, input object and output object of the target node, as well as the input object, output object and description of the evolved link in sections.

[0107] Please perform LINK evolution according to the above rules."

[0108] As an optional embodiment of the embodiment of the present invention, evolving the network structure in the task paradigm network includes:

[0109] Performing network segmentation on the task paradigm network to obtain multiple task stages;

[0110] Under the premise of not changing the input object and the output object of the node, the network structure of the task paradigm network is optimized by using the alternative scheme of the task stage.

[0111] Specifically, following the principle of segmentation of the computational graph, the task paradigm network is segmented into multiple task stages, each of which can represent an independent link in the task process. Without changing the input and output objects of the nodes, by finding an alternative solution with a simpler structure and the same input and output for each task stage, the internal structure of the task paradigm network is optimized to improve the efficiency of task execution.

[0112] Exemplarily, the method of network segmentation and network structure optimization can be implemented based on rules or large model technology.

[0113] The prompt information for segmenting the task paradigm network using the large model technology may include: task paradigm network segmentation task description information, task paradigm network analysis method description information, segmentation point description information, task phase description information and output format requirements. The prompt information prompt may be, for example:

[0114] “Please play the role of a computational graph design expert. Your task is to complete the network segmentation of the task paradigm network and subdivide the task paradigm network into multiple task stages. Each task stage represents an independent link in the task process. Please follow the following rules:

[0115] (1) Analyze the task paradigm network:

[0116] Extract all nodes and edges in the task paradigm network, confirm the input object, output object and operation description of each node; determine the overall structure and task flow of the task paradigm network.

[0117] (2) Determine the split point:

[0118] Follow the general segmentation principles of the computational graph and identify the key segmentation points in the task flow; the segmentation points should be able to divide the task flow into multiple independent task stages.

[0119] (3) Task-splitting paradigm network:

[0120] The task paradigm network is divided according to the determined dividing points to ensure that each task stage is independent and the connection relationship between each task stage is clear.

[0121] (4) Describe the task phase after segmentation:

[0122] Provide a detailed description of each task stage, including its name, included nodes, input objects, output objects, and operation descriptions, to ensure that the description of the task stage is clear and facilitates subsequent analysis and optimization.

[0123] The output result format is required to be as follows: list the name of each task stage, the nodes included, the input objects, the output objects, and the operation description in sections. "

[0124] The prompt information for optimizing the network structure of the task paradigm network using the large model technology may include: network structure optimization task description information, task phase analysis description information, optimization solution description information, network optimization process description information, optimized task phase description information and output format requirements. The prompt information may be, for example:

[0125] “Please play the role of a task process optimization expert. Your task is to optimize the internal structure of the task paradigm network without affecting the network input and output to improve the efficiency of task execution. Please follow the following rules:

[0126] (1) Analyze each task stage:

[0127] Extract all nodes and edges in each task stage, confirm their input objects, output objects and operation descriptions; determine the overall structure and task flow of each task stage.

[0128] (2) Finding optimization solutions:

[0129] For each task stage, look for alternative solutions with simpler structures that can achieve the same input and output. The alternative solutions should be able to simplify the task process and improve execution efficiency.

[0130] (3) Optimizing the Task Paradigm Network:

[0131] According to the optimization scheme found, the internal structure of the task paradigm network is adjusted to ensure that the optimized structure improves the efficiency of task execution without affecting the input and output.

[0132] (4) Describe the optimized task phase:

[0133] Provide a detailed description of each optimized task stage, including its name, included nodes, input objects, output objects, and operation descriptions, to ensure that the description of the optimized task stage is clear and convenient for subsequent analysis and verification.

[0134] The output result format requirements are as follows: For each task stage contained in the optimized task paradigm network, list the name of the task stage, the nodes contained, the input objects, the output objects and the operation description in sections. "

[0135] As an optional embodiment of the embodiment of the present invention, evolving the network structure in the task paradigm network further includes:

[0136] Identify target task stages whose input objects are the same as the types of output objects of the task stages from the task paradigm network; and add the target task stages to the task stages of the task paradigm network one by one according to their structural complexity.

[0137] Specifically, target task phases whose input objects are the same as the output objects of the task phase are identified from the task paradigm network, and the target task phases that meet the conditions are sorted according to the structural complexity and added to the task phases of the network one by one. When executing the task, the optimal solution can be selected from the task paradigm network based on the evaluation of performance and effect, thereby realizing the functional expansion of the task paradigm network.

[0138] As an optional embodiment of the embodiment of the present invention, the S110 of acquiring the knowledge graph of the instruction sample set in the target field includes:

[0139] S111, extracting task event information from the thought chain of each instruction sample in the instruction sample set of the target domain; the task event information includes: event name, input object, output object, operation and relationship between events.

[0140] The task event information can be understood as the information about the events included in the execution steps of the task, which may include: event name, input object (input), output object (output), operation (op) and the relationship between events (link).

[0141] Specifically, for each instruction sample in the instruction sample set of the target domain, the event name, input object, output object, operation and the relationship between events of each step corresponding to the event are extracted from the thought chain CoT of the instruction sample;.

[0142] Exemplarily, the method of extracting task event information from the thought chain of each instruction sample can be to extract task event information from the thought chain of the instruction sample by using named entity recognition (NER), event extraction (EE), relation extraction (RE) and other technologies.

[0143] S112. Create a node for each event. The name of the node is the event name of the event. The attributes of the node include: the input object, output object and operation of the event. The input object and the output constitute a node link.

[0144] S113. Construct directed edges according to the relationships between the events.

[0145] Specifically, directed edges between nodes are constructed based on the relationship between events, and the directed edges represent the data flow and control flow from one event to another event to another event.

[0146] S114. Construct a task network according to each of the nodes and each of the directed edges, and store the task network in a knowledge graph.

[0147] Specifically, all nodes and directed edges are combined into a task execution flow graph, which is used as the task network of the sample instruction. The task network fully represents the workflow of the task. The task networks of all samples are stored in the knowledge graph.

[0148] As an optional embodiment of the embodiment of the present invention, before extracting task event information from the thought chain of each sample in the instruction sample set, the method further includes:

[0149] Classifying the instruction samples in the instruction sample library into instruction samples containing thought chains and instruction samples not containing thought chains;

[0150] For instruction samples that do not contain a thinking chain, sample questions and sample answers in the instruction samples that do not contain a thinking chain are input into a thinking chain generation model to obtain a thinking chain.

[0151] Specifically, according to whether the instruction sample contains a thought chain, the instruction samples in the instruction sample library are divided into two types: instruction samples containing a thought chain CoT, denoted as S wCoT ={Q, CoT, A}; instruction samples that do not include CoT are denoted as S woCoT＝ {Q,A}; where Q represents a sample question, A represents a sample answer, and CoT represents a chain of thought.woCoT The instruction sample can be converted to S by supplementing the corresponding CoT description wCoT Sample. Exemplarily, the classification method of quality samples can be based on rules, large model analysis, or deep learning-based classification models, etc., and the embodiments of the present invention do not limit or elaborate on this.

[0152] For instruction samples that do not contain CoT, a thinking chain of the instruction sample is generated based on the thinking chain generation big model. Exemplarily, the thinking chain generation big model can use an open source big model, an online big model service, or a fine-tuned domain big model. Taking the instruction sample S1={Q1,A1} as an example, the thinking chain generation prompt information is S1Prompt. S1Prompt is input into the thinking chain generation big model, and the output result is the thinking chain CoT1 corresponding to the instruction sample S1, thereby forming a new sample S1={Q1,CoT1,A1} containing CoT.

[0153] This embodiment generates a thought chain for an instruction sample that does not contain a thought chain, thereby providing a necessary basis for subsequently extracting task event information from the thought chain of the instruction sample.

[0154] As an optional embodiment of the embodiment of the present application, after inputting the thought chain generation prompt information into the thought chain model, obtaining the thought chain of the instruction sample that does not contain the thought chain, and before extracting the task event information from the thought chain of each instruction sample in the instruction sample set, it also includes:

[0155] A1. Input the thought chain and the sample questions in the instruction sample not containing the thought chain into the thought chain generation model to obtain the output answer.

[0156] Among them, the output answer is the answer output by the large model generated by the thinking chain for the sample question.

[0157] Specifically, question and answer prompt information is generated based on sample questions in instruction samples that do not contain thinking chains and the generated thinking chains, and the question and answer prompt information is input into a large model for generating thinking chains to obtain output answers corresponding to the sample questions.

[0158] A2. Verify the thought chain of the instruction sample that does not contain a thought chain according to the comparison result between the output answer and the sample answer.

[0159] Specifically, the output answer and the sample answer are compared, and the thinking chain of the instruction sample that does not contain the thinking chain is verified according to the comparison result. If the comparison result is that the output answer and the sample answer are consistent, it can be considered that the thinking chain of the instruction sample that does not contain the thinking chain has been verified; if the comparison result is that the output answer and the sample answer are inconsistent, it can be considered that the thinking chain of the instruction sample that does not contain the thinking chain has not been verified.

[0160] Exemplarily, methods for comparing the output answer and the sample answer may include: an evaluation method based on expert experience, an evaluation method based on approximate indicators, an evaluation method based on verification platforms and test cases, an evaluation method based on verifier engineering, an internal consistency and self-feedback method, a confidence estimation and hallucination detection method, or a combination of one or more thereof.

[0161] Evaluation methods based on expert experience mainly rely on the experience of domain experts to conduct comparative analysis of sample answers and output answers, evaluate whether the output answers meet the expected correctness standards, identify deviations, and provide detailed feedback and correction suggestions.

[0162] The evaluation method based on approximate indicators mainly evaluates the correctness of the generated results by calculating the similarity index between the sample answer and the output answer. Natural language processing technology can be used to compare two text sequences and calculate their overlap to obtain a quantitative similarity score.

[0163] The evaluation method based on the verification platform and test cases mainly evaluates the correctness of the output answers by running a series of predefined test cases on a special verification platform. The test cases are designed to cover various expected input and output scenarios. By comparing the sample answers with the output answers, it can automatically verify whether the output of the thinking chain generation model meets the predetermined standards.

[0164] The evaluation method based on Verifier Engineering mainly uses automated verifiers to perform verification tasks and provide feedback to the basic model. It combines automation and manual feedback, optimizes model performance through a closed-loop feedback mechanism, analyzes sample answers and output answers, identifies inconsistencies, and provides improvement suggestions.

[0165] The internal consistency and self-feedback based rating method mainly evaluates the correctness of the output answer by analyzing the internal consistency of the generation process. It checks whether the model maintains logical coherence during the decoding process and whether it can produce consistent results under different sampling strategies. Through the self-feedback mechanism, the model can self-evaluate the consistency and correctness of its output, thereby improving its reasoning ability and reducing errors.

[0166] Confidence estimation and hallucination detection are mainly achieved by letting the thought chain generate a large model to evaluate the correctness of the output answer. The model will give an output answer and be asked whether the output answer is true or false. The model's log probability P(True) is used as the confidence score. At the same time, hallucination detection techniques, such as SelfCheckGPT, are used to determine whether the output answer contains factual errors.

[0167] A3. If the verification result is that the thinking chain of the instruction sample that does not contain the thinking chain fails to pass the verification, the difference information between the output answer and the sample answer is obtained.

[0168] Specifically, if the thinking chain of the instruction sample that does not contain the thinking chain fails to pass the verification, the difference information between the output answer and the sample answer is determined, and the difference information may include: missing information, redundant information, wrong information, inconsistent expression, etc.

[0169] A4. Input the generated thinking chain, the output answer, the instruction sample not containing the thinking chain and the difference information into the thinking chain generation model to obtain the revised thinking chain of the instruction sample not containing the thinking chain.

[0170] Specifically, the output answer and thinking chain output by the thinking chain generation model, the sample questions and sample answers contained in the instruction samples, and the difference information between the output answers and the sample answers are used to generate thinking chain correction prompt information, and the thinking chain correction prompt information is input into the thinking chain generation model to obtain the corrected thinking chain of the instruction sample that does not contain the thinking chain.

[0171] Exemplarily, the chain of thought correction prompt information may include: chain of thought correction task description, output answer and chain of thought generated by the large model, sample questions and sample answers contained in the instruction sample, and difference information between the output answer and the sample answer. For example, the chain of thought correction prompt information may be:

[0172] “Please play the role of a solution expert. Your task is to understand the problem, analyze the difference between the current answer and the standard answer, analyze the reasons for the difference, and correct the specific steps to solve the problem.

[0173] Known original question (Q): [question in the sample];

[0174] Model answer (A): [answer from the sample];

[0175] Current Chain of Thought (CoT): [Currently used CoT];

[0176] Current answer (A'): [newly generated answer];

[0177] The difference between the current answer and the standard answer (ΔA): [specific description of the difference];

[0178] Based on the above information, please revise the original problem-solving steps to accurately and completely solve the problem.

[0179] The revised Chain of Thought (CoT) is: [what should be generated]."

[0180] As an optional embodiment of the embodiment of the present application, the generating of learning corpus based on the task paradigm evolution network includes:

[0181] Decomposing the task paradigm evolution network into a plurality of task units;

[0182] Each of the task units is input into the corpus to generate a large model to obtain the learning corpus.

[0183] Specifically, decomposing the task paradigm evolution network into multiple task units and inputting the corpus into each of them to generate a large model can obtain richer and more targeted learning corpus. Using the learning corpus to fine-tune the large model can significantly enhance the ability of the large model to solve complex problems.

[0184] Exemplarily, a method of decomposing a task paradigm evolution network into multiple task units may be to introduce a task extraction module to decompose a complex task paradigm evolution network into multiple more detailed task units.

[0185] Figure 2 The following is a schematic diagram of the structure of a corpus processing device based on paradigm evolution provided by an embodiment of the present invention. Figure 2 As shown, the device includes: a knowledge graph acquisition module 210, a task paradigm network extraction module 220, a network evolution module 230 and a corpus generation module 240; wherein,

[0186] A knowledge graph acquisition module 210 is used to acquire a knowledge graph of an instruction sample set in a target domain; the knowledge graph includes a task network of each instruction sample in the instruction sample set;

[0187] A network extraction module 220 is used to extract a task paradigm network from the knowledge graph, where the task paradigm network is a common network among task networks of the same task type;

[0188] A network evolution module 230, used to evolve the task paradigm network to obtain a task paradigm evolution network;

[0189] The corpus generation module 240 is used to generate learning corpus based on the task paradigm evolution network, and the learning corpus is used for learning the target domain model.

[0190] The technical solution of the embodiment of the present invention is to obtain the knowledge graph of the instruction sample set of the target domain; the knowledge graph includes the task network of each instruction sample in the instruction sample set; extract the task paradigm network from the knowledge graph, the task paradigm network is a common network in the task network of the same task type; evolve the task paradigm network to obtain the task paradigm evolution network; generate learning corpus based on the task paradigm evolution network, and the learning corpus is used for learning the target domain model. The instruction samples are processed by the evolution mechanism based on the task paradigm network to obtain the learning corpus, which improves the precision of the learning corpus, reduces the dependence on a large amount of labeled data, reduces the data preparation cost and difficulty of model learning, and uses the learning corpus for the learning of the target domain model, which improves the model's ability to understand professional and focused domain tasks and improves the generalization ability of the model.

[0191] Optionally, the network evolution module 230 is used to:

[0192] At least one of the node operations, node links and network structures of the task paradigm network is evolved to obtain a task paradigm evolution network.

[0193] Optionally, the network evolution module 230 includes:

[0194] A node operation evolution unit, configured to perform at least one evolution operation of node operation splitting, node operation merging and node operation modification on the node operation in the task paradigm network without changing the input object and the output object of the node;

[0195] Among them, the node operation splitting is used to classify the original node operation into multiple intermediate node operations; the node operation merging is used to merge multiple original node operations into a comprehensive node operation; and the node operation modification is used to modify the original node operation into a target node operation.

[0196] Optionally, the network evolution module 230 includes:

[0197] The node link evolution unit is used to modify the description of the input object and the output object of the node in the task paradigm network and establish the evolved node link.

[0198] Optionally, the network evolution module 230 includes:

[0199] The network structure evolution unit is used to segment the task paradigm network to obtain multiple task stages; without changing the input object and output object of the node, the network structure of the task paradigm network is optimized by using the alternative scheme of the task stage.

[0200] Optionally, the network structure evolution unit further includes:

[0201] Identify target task stages whose input objects are the same as the types of output objects of the task stages from the task paradigm network; and add the target task stages to the task stages of the task paradigm network one by one according to their structural complexity.

[0202] Optionally, the knowledge graph acquisition module 210 includes:

[0203] An event information extraction module is used to extract task event information from the thought chain of each instruction sample in the instruction sample set of the target domain; the task event information includes: event name, input object, output object, operation and relationship between events;

[0204] A node creation unit, used to create a node for each event, the name of the node is the event name of the event, and the attributes of the node include: the input object, output object and operation of the event; the input object and the output constitute a node link;

[0205] An edge construction unit, used for constructing directed edges according to the relationship between the events;

[0206] The task network construction unit is used to construct a task network according to each of the nodes and each of the directed edges, and store the task network in a knowledge graph.

[0207] Optionally, also include:

[0208] A sample classification module is used to classify the instruction samples in the instruction sample library into instruction samples containing thought chains and instruction samples not containing thought chains before extracting task event information from the thought chain of each sample in the instruction sample set;

[0209] The thinking chain generation module is used to input the sample questions and sample answers in the instruction samples that do not contain the thinking chain into the thinking chain generation model to obtain the thinking chain of the instruction samples that do not contain the thinking chain.

[0210] Optionally, also include:

[0211] The answer output module is used for inputting the thought chain generation prompt information into the thought chain model to obtain the thought chain of the instruction sample not containing the thought chain, and before extracting the task event information from the thought chain of each instruction sample in the instruction sample set, inputting the thought chain and the sample questions in the instruction sample not containing the thought chain into the thought chain generation model to obtain the output answer;

[0212] A thought chain verification module, used for verifying the thought chain of the instruction sample not containing the thought chain according to the comparison result between the output answer and the sample answer;

[0213] a difference information determination module, configured to obtain difference information between the output answer and the sample answer if the verification result is that the thought chain of the instruction sample not including the thought chain fails to pass the verification;

[0214] The thought chain correction module is used to input the generated thought chain, the output answer, the instruction sample not containing the thought chain and the difference information into the thought chain generation model to obtain the corrected thought chain of the instruction sample not containing the thought chain.

[0215] Optionally, the corpus generation module 240 is specifically used to:

[0216] Decomposing the task paradigm evolution network into a plurality of task units;

[0217] Each of the task units is input into the corpus to generate a large model to obtain the learning corpus.

[0218] The corpus processing device based on paradigm evolution provided in the embodiment of the present invention can execute the ... method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0219] Figure 3 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0220] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0221] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0222] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a corpus processing method based on paradigm evolution.

[0223] In some embodiments, the corpus processing method based on paradigm evolution may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the corpus processing method based on paradigm evolution described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the corpus processing method based on paradigm evolution in any other appropriate manner (e.g., by means of firmware).

[0224] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0225] In some embodiments, the corpus processing method based on paradigm evolution can be implemented as a computer program, which is invisibly included in a computer program product. When the computer program is executed by a processor, the corpus processing method based on paradigm evolution of the present invention is implemented. The computer program product can be understood as a software product that mainly implements its solution through a computer program. The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or completely on a remote machine or server.

[0226] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0227] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0228] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0229] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0230] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0231] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A corpus processing method based on paradigm evolution, characterized in that: include: Acquire a knowledge graph of an instruction sample set in a target domain; the knowledge graph includes a task network of each instruction sample in the instruction sample set; Extracting a task paradigm network from the knowledge graph, wherein the task paradigm network is a common network among task networks of the same task type; Evolving the task paradigm network to obtain a task paradigm evolution network; A learning corpus is generated based on the task paradigm evolution network, and the learning corpus is used for learning a target domain model.

2. The method according to claim 1, characterized in that The step of evolving the task paradigm network to obtain a task paradigm evolution network includes: At least one of the node operations, node links and network structures of the task paradigm network is evolved to obtain a task paradigm evolution network.

3. The method according to claim 2, characterized in that Evolving the node operations in the task paradigm network includes: Under the premise of not changing the input object and the output object of the node, performing at least one evolution operation of node operation splitting, node operation merging and node operation modification on the node operation in the task paradigm network; Among them, the node operation splitting is used to classify the original node operation into multiple intermediate node operations; the node operation merging is used to merge multiple original node operations into a comprehensive node operation; and the node operation modification is used to modify the original node operation into a target node operation.

4. The method according to claim 2, characterized in that: Evolving the node links in the task paradigm network includes: The descriptions of the input objects and output objects of the nodes in the task paradigm network are modified to establish the evolved node links.

5. The method according to claim 2, characterized in that: Evolving the network structure in the task paradigm network, including: Performing network segmentation on the task paradigm network to obtain multiple task stages; Under the premise of not changing the input object and the output object of the node, the network structure of the task paradigm network is optimized by using the alternative scheme of the task stage.

6. The method according to claim 5, characterized in that Evolving the network structure in the task paradigm network also includes: identifying, from the task paradigm network, a target task phase whose input objects are of the same type as the output objects of the task phase; The target task stages are added one by one to the task stages of the task paradigm network according to their structural complexity.

7. The method according to claim 1, characterized in that The knowledge graph for obtaining the instruction sample set of the target domain includes: Extracting task event information from the thought chain of each instruction sample in the instruction sample set of the target domain; the task event information includes: event name, input object, output object, operation and relationship between events; Create a node for each event, the name of the node is the event name of the event, and the attributes of the node include: the input object, output object and operation of the event; the input object and the output constitute a node link; Constructing directed edges according to the relationships between the events; A task network is constructed according to each of the nodes and each of the directed edges, and the task network is stored in a knowledge graph.

8. The method according to claim 7, characterized in that Before extracting task event information from the thought chain of each sample in the instruction sample set, it also includes: Classifying the instruction samples in the instruction sample library into instruction samples containing thought chains and instruction samples not containing thought chains; For instruction samples that do not contain a thinking chain, sample questions and sample answers in the instruction samples that do not contain a thinking chain are input into a thinking chain generation model to obtain a thinking chain of the instruction samples that do not contain a thinking chain.

9. The method according to claim 8, characterized in that After inputting the thought chain generation prompt information into the thought chain model to obtain the thought chain of the instruction sample not containing the thought chain, and before extracting the task event information from the thought chain of each instruction sample in the instruction sample set, the method further includes: Input the thought chain and the sample questions in the instruction sample not including the thought chain into the thought chain generation model to obtain output answers; Verifying the thought chain of the instruction sample not including the thought chain according to the comparison result between the output answer and the sample answer; If the verification result is that the thought chain of the instruction sample not including the thought chain fails to pass the verification, obtaining the difference information between the output answer and the sample answer; The generated thinking chain, the output answer, the instruction sample not containing the thinking chain and the difference information are input into the thinking chain generation model to obtain the revised thinking chain of the instruction sample not containing the thinking chain.

10. The method according to claim 1, characterized in that The generating learning corpus based on the task paradigm evolution network includes: Decomposing the task paradigm evolution network into a plurality of task units; Each of the task units is input into the corpus to generate a large model to obtain the learning corpus.

11. A corpus processing device based on paradigm evolution, comprising: A knowledge graph acquisition module, used to acquire the knowledge graph of the instruction sample set in the target field; The knowledge graph includes a task network of each instruction sample in the instruction sample set; A network extraction module, used to extract a task paradigm network from the knowledge graph, wherein the task paradigm network is a common network among task networks of the same task type; A network evolution module, used for evolving the task paradigm network to obtain a task paradigm evolution network; The corpus generation module is used to generate learning corpus based on the task paradigm evolution network, and the learning corpus is used for learning the target domain model.

12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the corpus processing method based on paradigm evolution according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the corpus processing method based on paradigm evolution according to any one of claims 1 to 10 when executed.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the corpus processing method based on paradigm evolution according to any one of claims 1 to 10.

Images