A task processing method and system for collaborative learning of multiple intelligent nodes

Abstract

The present invention provides a task processing method and system for collaborative learning of multiple intelligent nodes, relating to the field of artificial intelligence technology, comprising: using multiple intelligent nodes to construct a target distributed intelligent network with multiple node communities; obtaining a domain task, determining a target intelligent node and a target domain label, establishing a task evaluation function and an incremental residual layer, performing collaborative learning in the node community corresponding to the target domain label, and generating an incremental residual module, wherein the incremental residual module has a layer correspondence with the network layer with the target domain label in the basic artificial neural network under the target intelligent node; the target intelligent node processes the domain task based on the incremental residual module, and shares the incremental residual module with other intelligent nodes in the node community corresponding to the target domain label, thereby solving the problem that the addition of new domain skills to existing large models requires centralized processing of a large amount of data, and there is privacy leakage in the training process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a task processing method and system for collaborative learning of multiple intelligent nodes. Background Art

[0002] Existing data-driven deep neural network technologies rely on vast amounts of data. One learning approach involves centralizing all data in a single data center to train a specific deep neural network model. This poses data security issues, primarily the leakage of private or confidential data. Some collaborative learning methods attempt to retain data locally on intelligent nodes, transmitting only the gradients during model training to the collaborative learning intelligent nodes. While this approach protects local data from leakage to a certain extent, since multiple intelligent nodes are training the same model, the resulting models are homogenized and require regular synchronization of the complete model to prevent the transferred gradients from being shared between different nodes. Summary of the Invention

[0003] In order to overcome the above-mentioned technical defects, the purpose of the present invention is to provide a task processing method and system for collaborative learning of multiple intelligent nodes, so as to solve the problem that the existing large model needs to centrally process a large amount of data to add new field skills, and there is privacy leakage in the training process.

[0004] The present invention discloses a task processing method for collaborative learning of multiple intelligent nodes, comprising:

[0005] A target distributed intelligent network with multiple node communities is constructed using multiple intelligent nodes, wherein each intelligent node includes a basic artificial neural network composed of a backbone network and an incremental residual module, wherein each network layer is associated with at least one domain label, and each node community corresponds to a unique domain label, so that intelligent nodes with the same domain label constitute a node community;

[0006] Acquire a domain task, determine a target intelligent node and a target domain label, establish a task evaluation function and an incremental residual layer, perform collaborative learning within a node community corresponding to the target domain label, and generate an incremental residual module that acts on a basic artificial neural network under the target intelligent node, wherein the incremental residual module has a layer correspondence relationship with a network layer with the target domain label in the basic artificial neural network under the target intelligent node;

[0007] The target intelligent node processes the domain task based on the incremental residual module, and shares the incremental residual module with other intelligent nodes in the node community corresponding to the target domain label.

[0008] Preferably, the step of establishing a task evaluation function and an incremental residual layer, performing collaborative learning within the node community corresponding to the target domain label, and generating an incremental residual module acting on the basic artificial neural network under the target intelligent node includes:

[0009] Randomly initializing the parameters of the incremental residual layer and screening out intelligent nodes for collaborative learning within the node community;

[0010] The incremental residual layer is spliced ​​in the basic artificial neural network under the target intelligent node and each intelligent node performing collaborative learning to respectively execute local tasks corresponding to the domain tasks, output a global evaluation value based on the task evaluation function, iteratively optimize the parameters of the incremental residual layer, and generate the incremental residual module.

[0011] Preferably, the target intelligent node processes the domain task based on the incremental residual module, including:

[0012] splicing the incremental residual module with the basic artificial neural network under the target intelligent node according to the layer correspondence, wherein the i-th incremental residual layer under the incremental residual module is connected in parallel to the i-th layer with the target domain label in the basic artificial neural network under the target intelligent node;

[0013] The output of the i-1th layer of the basic artificial neural network under the target intelligent node is obtained, and is input into the i-th layer of the basic artificial neural network and the i-th incremental residual layer. After the output, it is fused and then input into the i+1th layer of the basic artificial neural network.

[0014] Preferably, when the basic artificial neural network under the target intelligent node contains multiple domain labels:

[0015] The output of the i-1th layer of the basic artificial neural network under the target intelligent node is obtained, and the network layer corresponding to other domain labels is used for processing and then fused to the output of the domain task processed by the target intelligent node based on the incremental residual module.

[0016] Preferably, any incremental residual layer in the incremental residual module includes an upper layer backbone network label, a lower layer backbone network label and a backbone network;

[0017] The i-th incremental residual layer is connected to the i+1-th layer of the basic artificial neural network under the target intelligent node through the upper backbone network label, and is connected to the i-1-th layer of the basic artificial neural network under the target intelligent node through the lower backbone network label, and the incremental residual module is spliced ​​with the basic artificial neural network under the target intelligent node.

[0018] Preferably, before establishing the incremental residual layer, the method further includes:

[0019] Determine the characteristic description of the task based on the domain task;

[0020] Broadcasting the feature description of the task within the node community corresponding to the target domain label;

[0021] When the node community includes an intelligent node response including an incremental residual block corresponding to the domain task, the incremental residual block is obtained based on the sharing within the node community, and an incremental residual module is generated instead to act on the basic artificial neural network under the target intelligent node.

[0022] Preferably, after sharing the incremental residual module to other smart nodes in the node community corresponding to the target domain label, the method further includes:

[0023] For any of the other smart nodes,

[0024] Obtaining a new residual layer code sequence of the incremental residual module to self-check whether there is a missing incremental residual layer;

[0025] If so, request other smart nodes in the node community to share the incremental residual module.

[0026] Preferably, after sharing the incremental residual module to other smart nodes in the node community corresponding to the target domain label,

[0027] Periodically checking whether each smart node in the node community has conflicting residual layers, where the conflicting residual layers are different incremental residual layers generated corresponding to the same domain task;

[0028] If so, the task evaluation function is used to determine the optimal incremental residual layer.

[0029] The present invention also provides a task processing system for collaborative learning of multiple intelligent nodes, which executes the above-mentioned task processing method, including:

[0030] Multiple smart nodes, building a target distributed intelligent network with multiple node communities,

[0031] Each smart node includes:

[0032] A basic artificial neural network, wherein each network layer in the basic artificial neural network is associated with at least one domain label, each node community corresponds to a unique domain label, so that smart nodes with the same domain label constitute a node community;

[0033] An incremental residual module is used to perform collaborative learning generation within a node community corresponding to a target domain label of the domain task based on the obtained domain task, wherein the incremental residual module has a layer correspondence relationship with a network layer with the target domain label in a basic artificial neural network under the target intelligent node;

[0034] The incremental residual module is shared with other smart nodes in the node community corresponding to the target domain label.

[0035] Preferably, the basic artificial neural network of each intelligent node adopts a Transformer structure, and the incremental residual block uses an adapter module;

[0036] Each smart node in the same node community shares the network layer structure corresponding to the domain label.

[0037] Compared with the existing technology, the above technical solution has the following beneficial effects:

[0038] The present application provides a task processing method and system for collaborative learning of multiple intelligent nodes. Intelligent nodes are divided into different node communities based on domain labels, and incremental learning is implemented using incremental residual blocks within each node community. By passing the structure and weight parameters of the newly added residual blocks to the node community collaboration, the fusion of the newly added residual blocks for the same task can be achieved, solving the problem that the addition of new domain skills to existing large models requires centralized processing of large amounts of data, and there is privacy leakage in the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of a first embodiment of a task processing method and system for collaborative learning of multiple intelligent nodes according to the present invention;

[0040] Figure 2 This is a schematic diagram showing the node community in the first and second embodiments of the task processing method and system for multi-intelligent node collaborative learning of the present invention;

[0041] Figure 3 A schematic diagram of the network structure of any intelligent node in Embodiments 1 and 2 of a task processing method and system for collaborative learning of multiple intelligent nodes according to the present invention;

[0042] Figure 4 A schematic diagram of a flowchart of an intelligent node acquiring incremental residual blocks from other intelligent nodes in a node community in embodiments 1 and 2 of a task processing method and system for multi-intelligent node collaborative learning according to the present invention;

[0043] Figure 5 This is a schematic diagram of the network structure of the basic artificial neural network and the incremental residual module connected in the first and second embodiments of the task processing method and system for multi-intelligent node collaborative learning described in the present invention. DETAILED DESCRIPTION

[0044] The advantages of the present invention are further described below with reference to the accompanying drawings and specific embodiments.

[0045] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0046] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0047] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0048] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0049] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0050] In the following description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0051] Example 1: This embodiment discloses a task processing method for collaborative learning of multiple intelligent nodes. The task includes natural language processing and can be loaded into a computing system composed of multiple interconnected intelligent nodes, each of which is composed of a neural network. The intelligent nodes can collaboratively learn without transferring learning data. The basic artificial neural network (model) in different intelligent nodes does not need to be exactly the same, and there is no need to transfer model gradients during collaborative learning. For details, see Figure 1-Figure 5 ,include:

[0052] S10: Using multiple intelligent nodes to construct a target distributed intelligent network having multiple node communities, wherein each intelligent node includes a basic artificial neural network, the basic artificial neural network consisting of a backbone network and an incremental residual module, wherein each network layer is associated with at least one domain label, and each node community corresponds to a unique domain label, such that intelligent nodes with the same domain label constitute a node community;

[0053] In this embodiment, the basic artificial neural network of each intelligent node in the target distributed intelligent network can use the same pre-trained language model as the backbone network, and can have different incremental residual modules, thereby generating different basic artificial neural network structures. All intelligent nodes in the target distributed intelligent network are divided into multiple node communities, each node community has one and only one domain label, and there can be overlapping intelligent nodes between different node communities. Collaborative learning of multiple intelligent nodes is carried out in one community. Each intelligent node has at least one domain label, which can represent the domain knowledge possessed by the (intelligent) node. A node can contain multiple domain labels; each intelligent node in a node community must contain a domain label consistent with the community; an intelligent node can be located in multiple communities in the same domain; when an intelligent node contains multiple domain labels, it can be located in multiple domain communities. As an example, refer to Figure 2 ,Domains D1, D2, and D3 form communities C1, C2, and C3 respectively, node N7 has domains D1, D2, and D3, point N8 has nodes D2 and D3, and so on.

[0054] As an illustration, domain labels / knowledge are embodied as causal relationships between concepts, and can be represented as multi-layer neural network structures and neural network weight parameters through the training learning process, where shallow neural network layers correspond to the underlying concepts in domain knowledge, and high-level neural network layers correspond to the high-level concepts in domain knowledge; for node communities with non-empty intersection nodes, the multi-layer neural networks corresponding to the different domain knowledge represented by the smart nodes within them will share the shallow neural network layers.

[0055] In the system applying the natural language task processing method of collaborative learning of multiple intelligent nodes provided in this embodiment, at the starting moment of execution, all intelligent nodes can be set to have an initialized neural network (i.e., a basic artificial neural network); and when two intelligent nodes have the same domain label, they will both contain the initialized neural network layer corresponding to the domain knowledge, which may include a backbone network and a shared incremental residual module.

[0056] S20: Obtain a domain task, such as Chinese-Japanese translation, determine a target intelligent node and a target domain label, establish a task evaluation function and an incremental residual layer, perform collaborative learning within a node community corresponding to the target domain label, and generate an incremental residual module that acts on a basic artificial neural network under the target intelligent node, wherein the incremental residual module has a layer correspondence relationship with a network layer with the target domain label in the basic artificial neural network under the target intelligent node;

[0057] This task is a task with a domain label. For new tasks within its own domain label, in this embodiment, the intelligent node learns new knowledge. The learned new knowledge is represented as an incremental residual neural network layer (i.e., an incremental residual module) parallel to the neural network layer in the basic artificial neural network.

[0058] Specifically, the task evaluation function and the incremental residual layer are established, collaborative learning is performed in the node community corresponding to the target domain label, and an incremental residual module acting on the basic artificial neural network under the target intelligent node is generated, including: randomly initializing the parameters of the incremental residual layer, screening out the intelligent nodes performing collaborative learning in the node community; splicing the incremental residual layer in the basic artificial neural network under the target intelligent node and each intelligent node performing collaborative learning to respectively execute local tasks corresponding to the domain task, outputting a global evaluation value based on the task evaluation function, iteratively optimizing the parameters of the incremental residual layer, and generating the incremental residual module.

[0059] The above-mentioned intelligent nodes that perform collaborative learning in the collaborative learning process perform local tasks corresponding to the domain tasks, which are tasks matched according to the task description features of the domain tasks. The description features of the tasks can be based on the compressed encoding output of several bottom layers of the basic artificial neural network (with all domain labels / common network structure).

[0060] Specifically, as an example, perform the following steps c1: When the node Start studying the field New task collection , formulate a new task target evaluation function Obj, generate a column of incremental residual layers corresponding to the existing neural network layer (i.e. the basic artificial neural network of this node) , The highest number of layers in the neural network structure. The incremental residual layer contains two types of parameters. One type is the parameters describing the structure of the incremental residual layer itself and the parameters of the structure connected to the original network (basic artificial neural network), which are collectively denoted as τ. The other type is the parameters of the network weights in the incremental residual layer, denoted as θ. The above two types of parameters are randomly initialized, and the weight parameters in the incremental residual layer are learned using the optimization algorithm to obtain the incremental residual layer. and assessed value and the number of new tasks , and then the incremental residual layer Send to containing node And the field label is Smart nodes in all communities .

[0061] c2: node Extracting a new task set The features of the new task target evaluation function Obj are sent to the node containing And the field label is The collection of smart nodes in all communities ;

[0062] c3: node collection Each node in Based on new task set characteristics, select the task data set that meets the permission requirements and matches the characteristics of the new task set from its own task data ;node The incremental residual layer is spliced ​​into the neural network of this node according to the layer correspondence to form a temporary neural network model; the node Using Task Data Collection Evaluate the temporary neural network model with the target evaluation function Obj to obtain the evaluation value of the temporary neural network , then and the number of new tasks for node evaluation Passed to the initiating node .

[0063] c4: node Summarized from The evaluation value and number of evaluation tasks of each node in the , plus the number of evaluation tasks of itself and assessed value , forming a set of evaluation values and the set of evaluation tasks , and calculate the global evaluation value V based on the weighted average;

[0064] Repeat the process from c1 to c4, record the number of repetitions, and use the gradient-free optimization algorithm to optimize the parameters τ and θ of the newly added residual layer until the global evaluation value V is greater than the set threshold, or the number of repetitions exceeds the set number, fix the parameters, and generate the incremental residual module.

[0065] Specifically, before establishing the incremental residual layer, it also includes: determining the characteristic description of the task based on the domain task; broadcasting the characteristic description of the task in the node community corresponding to the target domain label; when the node community includes an intelligent node response of the incremental residual block corresponding to the domain task, the incremental residual block is obtained based on the sharing within the node community, and an incremental residual module is generated instead of the basic artificial neural network acting on the target intelligent node.

[0066] As mentioned above, this feature description is also used in the step of generating the incremental residual module. The feature description of this task is the requirement of the incremental residual block corresponding to this task, which is the compressed encoding output by the bottom several layers of the neural network (basic artificial neural network) backbone. In this embodiment, if there are nodes in the node community Already contains new tasks All incremental residual blocks, and the access rights of these incremental residual blocks are in a shareable state, then the data structure objects of these incremental residual blocks, that is, layer module objects, are sent to the node That is, there is no need to use the above-mentioned optimization algorithm to generate incremental residual modules. Instead, they are directly obtained through sharing within the node community to improve data / task processing efficiency. By passing the structure and weight parameters of the newly added residual block to the community collaboration node, there is no need to send any information such as node local data, data backpropagation gradients, or data features to other nodes in the node community, which also provides higher security.

[0067] In this embodiment, when an intelligent node receives a new task and this task cannot be effectively processed by the existing neural network model, a new incremental residual block needs to be generated by the incremental residual layer. The intelligent node can share the characteristics and evaluation functions of the task with other nodes, and request them to provide feedback and evaluation values, so as to obtain the incremental residual layer with optimal parameters, that is, the determined incremental residual block.

[0068] S30: The target intelligent node processes the domain task based on the incremental residual module, and shares the incremental residual module with other intelligent nodes in the node community corresponding to the target domain label.

[0069] Specifically, the target intelligent node processes the domain task based on the incremental residual module, see Figure 3 , including: splicing the incremental residual module with the basic artificial neural network under the target intelligent node according to the layer correspondence, wherein the i-th incremental residual layer (Res-i) under the incremental residual module is connected in parallel to the i-th layer (layer-i) with the target domain label in the basic artificial neural network under the target intelligent node; obtaining the output of the i-1th layer of the basic artificial neural network under the target intelligent node, inputting it into the i-th layer of the basic artificial neural network and the i-th incremental residual layer, and then fusing the output and inputting it into the i+1th layer of the basic artificial neural network.

[0070] As a supplementary explanation, in this embodiment, the neural network (basic artificial neural network) backbone network in the intelligent node can adopt a Transformer structure, and the incremental residual block uses an Adaptor module. The structure and weight parameters of the neural network backbone and the incremental residual block are stored in a data structure. The basic unit of the data structure is the layer module, that is, the layer in the Transformer network and the layer in the Adaptor module can both be stored using a layer module data structure; in addition to the network structure parameters and network weight parameters required as the neural network Transformer Layer and Adaptor, the layer module of the Transformer network contains domain labels, and the layer module of the incremental residual block contains: the incremental residual block ID, domain labels (such as D1, D2, D3, etc. in the figure), task labels (such as T1, T2, etc. in the figure), and task feature codes (that is, feature descriptions of the above tasks).

[0071] Since the target distributed intelligent network in this embodiment is based on layers, when the incremental residual module is spliced ​​with the basic module, and the two have a layer correspondence relationship, the i-th layer of the basic artificial neural network corresponds to the i-th layer in the incremental residual module in parallel. Specifically, any incremental residual layer in the incremental residual module includes an upper-layer backbone network label, a lower-layer backbone network label, and a backbone network; the i-th incremental residual layer is connected to the i+1-th layer of the basic artificial neural network under the target intelligent node through the upper-layer backbone network label, and is connected to the i-1-th layer of the basic artificial neural network under the target intelligent node through the lower-layer backbone network label, and the incremental residual module is spliced ​​with the basic artificial neural network under the target intelligent node. That is, the above-mentioned upper-layer backbone network label and lower-layer backbone network label are used as the connection identifier of the i-th incremental residual layer, so that the generated incremental residual layer can autonomously connect to the network layer under the corresponding domain label in the basic artificial neural network.

[0072] In this embodiment, as mentioned above, a node can contain multiple domain labels, and the above-mentioned domain tasks only rely on the incremental learning under a domain label (node ​​community) corresponding to the incremental residual block. Therefore, when the basic artificial neural network under the target intelligent node contains multiple domain labels, that is, the intelligent node is located in multiple node communities, the incremental residual block only updates the network parameters corresponding to the domain label. Specifically: the output of the i-1th layer of the basic artificial neural network under the target intelligent node is obtained, and the network layer corresponding to other domain labels is used for processing and then fused to the output of the domain task processed by the target intelligent node based on the incremental residual module. That is, the network layer corresponding to other domain labels will not be connected to the incremental residual block, and will not affect the network layer corresponding to other domain labels. That is, different communities with the same domain label can realize the fusion of incremental residual blocks for the same task, and will not affect data processing in different domains.

[0073] In this embodiment, as a preferred embodiment, after sharing the incremental residual module to other intelligent nodes in the node community corresponding to the target domain label, it also includes: for any intelligent node among the other intelligent nodes, obtaining the newly added residual layer code sequence of the incremental residual module to self-check whether there is a missing incremental residual layer; if so, requesting other intelligent nodes in the node community to share the incremental residual module.

[0074] In this embodiment, freezing the structural parameters τ and weight parameters θ of the incremental residual layer can be regarded as creating a new residual layer, giving the new residual layer a code The code contains the domain id, task id and creation time information; and records the domain at this time Except for the initial layer, the code sequence of all other newly added residual layers in the previous order Get the field Nodes of new task knowledge The newly added knowledge, i.e. the parameters τ and θ of the newly added residual layer, and the code , the pre-order added residual layer code sequence , passed to a with the same fields Labeled nodes .

[0075] node Add a new residual layer code sequence based on the previous sequence , check whether it already contains all the previous additional residual layers; if there are missing previous additional residual layers, on the one hand, mark the missing previous additional residual layer codes and their positions in the neural network (incremental residual module) structure; on the other hand, you can add nodes The field label is Other nodes in the community request to obtain the missing pre-order additional residual layers. Other nodes decide whether to share them with the node based on whether they contain these pre-order additional residual layers and the permission settings for these residual layers. ;

[0076] In this embodiment, as a preferred embodiment, after sharing the incremental residual module to other intelligent nodes in the node community corresponding to the target domain label, it can also include: periodically checking whether there are conflicting residual layers in each intelligent node in the node community, wherein the conflicting residual layers are different incremental residual layers generated corresponding to the same domain task; if so, using the task evaluation function to determine the optimal incremental residual layer.

[0077] It is understandable that periodically checking whether there are new residual layers with the same task id but different codes between different nodes in the same community (which can be found from the above new residual layer codes ), which is called the conflict residual layer in this embodiment. If there is one, a task set can be constructed in the node community. The evaluation function of the task is used to evaluate and compare the neural network model containing the conflict residual layer. The newly added residual layer with a high evaluation value is selected from the conflict residual layer as the winning residual layer, and other conflict residual layers are eliminated.

[0078] In the target distributed intelligent network of the present embodiment, first, the intelligent nodes in the network are divided into different communities based on domain labels. The intelligent nodes in the same community can decide whether to participate in a multi-intelligent node collaborative learning based on their own wishes (specifically, they can adjust the screening conditions of the nodes that perform collaborative learning in the node community corresponding to the target domain label). Secondly, in the collaborative learning process, an incremental residual module is used to implement incremental learning. The structural parameters and weight parameters of the incremental residual module are optimized using a gradient-free optimization algorithm. By passing the structure and weight parameters of the newly added residual block to the community collaboration node, there is no need to send any information such as node local data, data backpropagation gradient, or data features to other nodes in the community, thereby ensuring that the data and its associated objects only remain local to the node. Finally, in the collaborative learning process, different communities with the same domain label can realize the fusion of the newly added residual blocks of the same task, realize the cognitive alignment of different node communities, and thus realize the task processing of multi-intelligent node collaborative learning.

[0079] Embodiment 2: The present invention also provides a task processing system for collaborative learning of multiple intelligent nodes, which executes the task processing (natural language processing) method for collaborative learning of multiple intelligent nodes described in the above embodiment 1, including: multiple intelligent nodes, which can be loaded on one or more computer devices to construct a target distributed intelligent network with multiple node communities, wherein each intelligent node includes: a basic artificial neural network, each network layer in the basic artificial neural network is associated with at least one domain label, and each node community corresponds to a unique domain label, so that intelligent nodes with the same domain label constitute a node community; an incremental residual module, which is used to perform collaborative learning and generation in the node community corresponding to the target domain label of the domain task based on the obtained domain task, wherein the incremental residual module has a layer correspondence relationship with the network layer with the target domain label in the basic artificial neural network under the target intelligent node; the incremental residual module is shared to other intelligent nodes in the node community corresponding to the target domain label.

[0080] Optionally, the basic artificial neural network of each intelligent node adopts the Transformer structure, and the incremental residual block uses the adapter module; each intelligent node in the same node community shares the network layer structure corresponding to the domain label.

[0081] As an example, computers are used as intelligent nodes. Each computer contains at least one domain label and a neural network (basic artificial neural network) corresponding to the domain label. The structure of the neural network adopts the Transformer structure as described above. Multiple intelligent nodes are interconnected through the Internet to form a dedicated intelligent network.

[0082] Multiple smart nodes with the same domain label logically constitute a node community. In a dedicated smart network, there are multiple domains, and a domain can have multiple overlapping node communities. Different communities in the same domain can have overlapping smart nodes. A smart node with multiple domain labels can also exist in different communities with different domain labels.

[0083] Specifically, for example, the basic (neural) network backbone in the smart node adopts the Transformer structure, and the incremental residual block of the neural network uses the Adaptor module; the neural network backbone adopts pre-training to determine the initialization parameters; different smart nodes in the same field contain the same neural network backbone corresponding to the same field; neural network backbones in different fields can share the same Transformer Layer; Figure 3 The white rectangles form the neural network backbone of the smart node. Multiple domain tasks with the same modality input share some neural network layers in the neural network backbone. When the smart node is initialized, the neural network backbone is initialized using pre-trained parameters. Subsequently, when encountering a new task, learning the new task generates an incremental residual block of the neural network.

[0084] The structure and weight parameters of the backbone and incremental residual blocks of the neural network are stored in a data structure. The basic unit of this data structure is the layer module. The layers in the Transformer and the layers in the Adaptor can both be stored in the layer module data structure. In addition to being the network structure parameters and network weight parameters required by the neural network Transformer Layer and Adaptor, the layer module also contains Figure 3 and Figure 5 As shown in the figure, there are also the following special points: the layer module of the backbone network contains the domain label; the layer module of the task incremental residual block contains: the incremental residual block ID, domain label, task label, task feature code; the incremental residual block ID of the preceding task of the task incremental residual block.

[0085] The system provided in this embodiment can perform the following:

[0086] A smart node Start learning a new task in a domain When it is in the same area, all the node communities in the same area broadcast the requirements for the incremental residual blocks corresponding to this task; the feature description of the task in the broadcast content is the compressed encoding of the output of several layers at the bottom of the neural network backbone;

[0087] If there are nodes in the node community Already contains new tasks All incremental residual blocks, and the access rights of these incremental residual blocks are in a shareable state, then the data structure objects of these incremental residual blocks, that is, layer module objects, are sent to the node ;if After receiving the layer module object, if the ID of the incremental residual block of its predecessor task is missing, the learning of the missing predecessor task is initiated as described in Implementation 1;

[0088] If there is no node in the node community that contains or can share the new task The corresponding incremental residual block uses the neuroevolution algorithm to learn a set of Adaptor blocks as a new task Corresponding incremental residual blocks; When implementing the neuroevolution algorithm, in each round of the population (task set), as an option, the top 10% of individuals (tasks) are selected and sent to several collaborative learning nodes in the community , filter collaborative learning nodes, and send to these collaborative learning nodes (1) the feature description of the task, (2) the task evaluation function; each collaborative learning node Complete the following actions: (1) Find a number of tasks that match the task description features from the local data, which does not exceed the upper limit. (2) According to the data structure of the received incremental residual block, that is, the layer module object, insert the incremental residual block on the basis of the existing local neural network (basic artificial neural network) to obtain the verified neural network. (3) Use the verified neural network to infer the selected local tasks, and its output results are processed by the evaluation function to obtain the evaluation value. (4) Then return the evaluation result to the collaborative learning initiator node ; Collaborative learning initiator The evaluation values ​​of all collaborative learning nodes and their own local task evaluation values ​​are combined as the final evaluation value of this round of the group. The subsequent processing is similar to the neuroevolution algorithm. When the neuroevolution algorithm is completed, the structure and weight parameters of the incremental residual block are determined.

[0089] As an illustration, become a collaborative learning node The following conditions need to be met: (1) The node is willing to participate in collaborative learning (precondition); (2) The node already contains all the previous incremental residual blocks for learning the task, which can be achieved by checking whether the ID of the previous incremental residual block exists in the current network; Finally, the node initiating collaborative learning The new task The corresponding layer module object of the incremental residual block is sent to the shared smart node in the community.

[0090] It should be noted that the embodiments of the present invention have better practicability and do not impose any form of limitation on the present invention. Any technician familiar with the field may use the technical content disclosed above to change or modify it into an equivalent effective embodiment. However, any modification or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A task processing method for collaborative learning of multiple intelligent nodes, characterized in that: include: A target distributed intelligent network with multiple node communities is constructed using multiple intelligent nodes, wherein each intelligent node includes a basic artificial neural network composed of a backbone network and an incremental residual module, wherein each network layer is associated with at least one domain label, and each node community corresponds to a unique domain label, so that intelligent nodes with the same domain label constitute a node community; Acquire a domain task, determine a target intelligent node and a target domain label, establish a task evaluation function and an incremental residual layer, perform collaborative learning within a node community corresponding to the target domain label, and generate an incremental residual module that acts on a basic artificial neural network under the target intelligent node, wherein the incremental residual module has a layer correspondence relationship with a network layer with the target domain label in the basic artificial neural network under the target intelligent node; The target intelligent node processes the domain task based on the incremental residual module, and shares the incremental residual module with other intelligent nodes in the node community corresponding to the target domain label; The task evaluation function and the incremental residual layer are established, collaborative learning is performed in the node community corresponding to the target domain label, and the incremental residual module acting on the basic artificial neural network under the target intelligent node is generated, including: Randomly initializing the parameters of the incremental residual layer and screening out intelligent nodes for collaborative learning within the node community; The incremental residual layer is spliced ​​in the basic artificial neural network under the target intelligent node and each intelligent node performing collaborative learning to respectively perform local tasks corresponding to the domain tasks, a global evaluation value is output based on the task evaluation function, and the parameters of the incremental residual layer are iteratively optimized to generate the incremental residual module, and Before establishing the incremental residual layer, it also includes: Determine the characteristic description of the task based on the domain task; Broadcasting the feature description of the task within the node community corresponding to the target domain label; When the node community includes an intelligent node response including an incremental residual block corresponding to the domain task, the incremental residual block is obtained based on the sharing within the node community, and an incremental residual module is generated instead to act on the basic artificial neural network under the target intelligent node.

2. The task processing method according to claim 1, characterized in that: The target intelligent node processes the domain task based on the incremental residual module, including: splicing the incremental residual module with the basic artificial neural network under the target intelligent node according to the layer correspondence, wherein the i-th incremental residual layer under the incremental residual module is connected in parallel to the i-th layer with the target domain label in the basic artificial neural network under the target intelligent node; The output of the i-1th layer of the basic artificial neural network under the target intelligent node is obtained, and is input into the i-th layer of the basic artificial neural network and the i-th incremental residual layer. After the output, it is fused and then input into the i+1th layer of the basic artificial neural network.

3. The task processing method according to claim 2, wherein: When the basic artificial neural network under the target intelligent node contains multiple domain labels; The output of the i-1th layer of the basic artificial neural network under the target intelligent node is obtained, and the network layer corresponding to other domain labels is used for processing and then fused to the output of the domain task processed by the target intelligent node based on the incremental residual module.

4. The task processing method according to claim 2, wherein: Any incremental residual layer in the incremental residual module includes an upper layer backbone network label, a lower layer backbone network label and a backbone network; The i-th incremental residual layer is connected to the i+1-th layer of the basic artificial neural network under the target intelligent node through the upper backbone network label, and is connected to the i-1-th layer of the basic artificial neural network under the target intelligent node through the lower backbone network label, and the incremental residual module is spliced ​​with the basic artificial neural network under the target intelligent node.

5. The task processing method according to claim 1, characterized in that: After sharing the incremental residual module to other smart nodes in the node community corresponding to the target domain label, the method further includes: For any of the other smart nodes, Obtaining a new residual layer code sequence of the incremental residual module to self-check whether there is a missing incremental residual layer; If so, request other smart nodes in the node community to share the incremental residual module.

6. The task processing method according to claim 1, characterized in that: After sharing the incremental residual module to other smart nodes in the node community corresponding to the target domain label, including Periodically checking whether each smart node in the node community has conflicting residual layers, where the conflicting residual layers are different incremental residual layers generated corresponding to the same domain task; If so, the task evaluation function is used to determine the optimal incremental residual layer.

7. A task processing system for collaborative learning of multiple intelligent nodes, characterized in that: The task processing method according to any one of claims 1 to 6 includes: Multiple smart nodes, building a target distributed intelligent network with multiple node communities, Each smart node includes: A basic artificial neural network, wherein each network layer in the basic artificial neural network is associated with at least one domain label, each node community corresponds to a unique domain label, so that smart nodes with the same domain label constitute a node community; An incremental residual module is used to perform collaborative learning generation within a node community corresponding to a target domain label of the domain task based on the obtained domain task, wherein the incremental residual module has a layer correspondence relationship with a network layer with the target domain label in a basic artificial neural network under the target intelligent node; The incremental residual module is shared with other smart nodes in the node community corresponding to the target domain label.

8. The task processing system according to claim 7, wherein: The basic artificial neural network of each intelligent node adopts the Transformer structure, and the incremental residual block uses the adapter module; Each smart node in the same node community shares the network layer structure corresponding to the domain label.

Images