Network structure construction method and device for multi-task scenarios

Abstract

The present application relates to artificial intelligence, and provides a network structure construction method, apparatus, device and storage medium for a multi-task scenario, including: obtaining a training set, inputting the training sub-text data corresponding to each target semantic task into a multi-task network model of a network structure to be determined step by step, obtaining the sub-prediction results corresponding to each target semantic task, and adjusting the network parameters of the multi-task network model until the current target network parameters corresponding to the current network structure are obtained; obtaining a search space corresponding to the multi-task network model, obtaining a verification set, adjusting the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space, dividing the implicit state vector of the multi-task network model into a plurality of ordered sub-implicit state vectors during the search, until the output result of the multi-task network model on the verification set meets the convergence condition, obtaining the target structural parameters, and obtaining the trained multi-task network model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and storage medium for constructing a network structure for a multi-task scenario. Background Art

[0002] Machine Learning (ML) is a branch of artificial intelligence. The purpose of machine learning is to enable machines to learn based on prior knowledge so that they have the logical ability to classify and judge. Machine learning models represented by neural networks continue to develop and are increasingly being applied to various industries.

[0003] The multi-task learning mechanism is widely used in modern artificial intelligence product applications. Multi-task means that the input needs to obtain corresponding recognition results for different tasks. The original solution is to train a model for each subtask. After deployment, each model needs to be trained once. Training is time-consuming and the prediction speed is slow. Engineers manually try different neural network architectures and then determine the target architecture based on the performance of the validation set. Due to the complexity of network architecture learning in multi-task scenarios, it is difficult to manually design a very good neural network structure. Traditional model structure automatic search methods are mainly aimed at classification problems and cannot be directly applied to the automatic search of model structures in multi-task scenarios. The model structure of multi-task scenarios is constructed by manual continuous trial and error, which is highly complex, inefficient, and has a large system resource usage rate. Summary of the invention

[0004] Based on this, it is necessary to provide a network structure construction method, device, computer equipment and storage medium for multi-task scenarios to address the above technical problems, automatically discover the network architecture that best suits the existing multi-task scenario data set, and effectively reduce resource consumption during differentiable search through partial links, so that the search converges faster and more stably, improving efficiency and reducing system resource utilization.

[0005] A method for constructing a network structure for a multi-task scenario, the method comprising:

[0006] Acquire a training set, wherein the training set includes a plurality of training subsamples corresponding to different target semantic tasks, and the training subsamples include training subtext data and training sublabel data;

[0007] Inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure step by step, obtaining the sub-prediction results corresponding to each target semantic task, and adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data, until obtaining the current target network parameters corresponding to the current network structure;

[0008] The method comprises the steps of: obtaining a search space corresponding to the multi-task network model, forming a differentiable network search space, obtaining a verification set, adjusting the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set, dividing the implicit state vector of the multi-task network model into a plurality of ordered sub-implicit state vectors during the search, obtaining the sub-implicit state vectors corresponding to the current search in a preset order, inputting the sub-implicit state vectors into the corresponding network layer for training, and obtaining an updated multi-task network model, returning to the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step, until the output result of the multi-task network model on the verification set meets the convergence condition, obtaining the target structural parameters, obtaining the network parameters matching the target structural parameters, and obtaining the trained multi-task network model according to the target structural parameters and the matching network parameters.

[0009] In one embodiment, the searching of the differentiable network search space is performed by at least one of the following sharing methods:

[0010] Matrix parameter sharing of multi-head attention in the differentiable network search space;

[0011] When searching the pooling layer of the multi-task network model, multiple operators based on the capsule network share the parameters of the mapping network;

[0012] The connection relationship between the nodes of the multi-task network model is obtained, the nodes having the same starting node are grouped into node sets, and the operators corresponding to the nodes in different node sets share parameters.

[0013] In one embodiment, the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure to obtain the sub-prediction results corresponding to each target semantic task comprises:

[0014] Segment the current training subtext data corresponding to the current target semantic task, and map each segmented word to a corresponding vector to form a vector set;

[0015] The encoder extracts semantic features from the vector set, and obtains a sub-prediction result corresponding to the current target semantic task based on the semantic features, wherein the current target semantic task is one of the target semantic tasks.

[0016] In one embodiment, the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure to obtain the sub-prediction results corresponding to each target semantic task comprises:

[0017] Calculating the similarity between the current training subtext data corresponding to the current target semantic task and the candidate texts in the database, and obtaining similar subtext data matching the current training subtext data;

[0018] Inputting the first vector set corresponding to the current training subtext data into a first encoder to extract semantic features to obtain a first semantic feature, and inputting the second vector set corresponding to the similar subtext data into a second encoder to extract semantic features to obtain a second semantic feature;

[0019] A sub-prediction result corresponding to the current target semantic task is obtained according to the first semantic feature and the second semantic feature.

[0020] In one of the embodiments, the weights of the first encoder and the second encoder are shared.

[0021] In one embodiment, adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction result and the corresponding training sub-label data includes:

[0022] Obtain the sub-prediction results and training sub-label data corresponding to each target semantic task, and obtain the sub-differences corresponding to each target semantic task;

[0023] Obtain the task weights corresponding to each target semantic task, and weight each sub-difference according to the task weights to obtain statistical sub-differences;

[0024] The network parameters of the multi-task network model are adjusted according to the statistical sub-differences.

[0025] A network structure construction device for a multi-task scenario, the device comprising:

[0026] An acquisition module is used to acquire a training set, wherein the training set includes a plurality of training subsamples corresponding to different target semantic tasks, and the training subsamples include training subtext data and training sublabel data;

[0027] A network parameter adjustment module is used to input the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure in steps, obtain the sub-prediction results corresponding to each target semantic task, and adjust the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data until the current target network parameters corresponding to the current network structure are obtained;

[0028] A network structure construction module is used to obtain a search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set. During the search, the implicit state vector of the multi-task network model is divided into a plurality of ordered sub-implicit state vectors, and the sub-implicit state vectors corresponding to the current search are obtained in a preset order. The sub-implicit state vectors are input into the corresponding network layer for training to obtain an updated multi-task network model, and the network parameter adjustment module is returned until the output result of the multi-task network model on the verification set meets the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matching network parameters.

[0029] In one embodiment, the network structure building module is further used to search the differentiable network search space in at least one of the following sharing modes:

[0030] Matrix parameter sharing of multi-head attention in the differentiable network search space;

[0031] When searching the pooling layer of the multi-task network model, multiple operators based on the capsule network share the parameters of the mapping network;

[0032] The connection relationship between the nodes of the multi-task network model is obtained, the nodes having the same starting node are grouped into node sets, and the operators corresponding to the nodes in different node sets share parameters.

[0033] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire a training set, wherein the training set includes a plurality of training subsamples corresponding to different target semantic tasks, and the training subsamples include training subtext data and training sublabel data;

[0035] Inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure step by step, obtaining the sub-prediction results corresponding to each target semantic task, and adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data, until obtaining the current target network parameters corresponding to the current network structure;

[0036] The method comprises the steps of: obtaining a search space corresponding to the multi-task network model, forming a differentiable network search space, obtaining a verification set, adjusting the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set, dividing the implicit state vector of the multi-task network model into a plurality of ordered sub-implicit state vectors during the search, obtaining the sub-implicit state vectors corresponding to the current search in a preset order, inputting the sub-implicit state vectors into the corresponding network layer for training, and obtaining an updated multi-task network model, returning to the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step, until the output result of the multi-task network model on the verification set meets the convergence condition, obtaining the target structural parameters, obtaining the network parameters matching the target structural parameters, and obtaining the trained multi-task network model according to the target structural parameters and the matching network parameters.

[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0038] Acquire a training set, wherein the training set includes a plurality of training subsamples corresponding to different target semantic tasks, and the training subsamples include training subtext data and training sublabel data;

[0039] Inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure step by step, obtaining the sub-prediction results corresponding to each target semantic task, and adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data, until obtaining the current target network parameters corresponding to the current network structure;

[0040] The method comprises the steps of: obtaining a search space corresponding to the multi-task network model, forming a differentiable network search space, obtaining a verification set, adjusting the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set, dividing the implicit state vector of the multi-task network model into a plurality of ordered sub-implicit state vectors during the search, obtaining the sub-implicit state vectors corresponding to the current search in a preset order, inputting the sub-implicit state vectors into the corresponding network layer for training, and obtaining an updated multi-task network model, returning to the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step, until the output result of the multi-task network model on the verification set meets the convergence condition, obtaining the target structural parameters, obtaining the network parameters matching the target structural parameters, and obtaining the trained multi-task network model according to the target structural parameters and the matching network parameters.

[0041] The above-mentioned network structure construction method, device, computer equipment and storage medium for multi-task scenarios have the function of automatically discovering the network architecture that best suits the existing multi-task scenario data set. The accuracy of the multi-task system can be improved without manually trying many different models. The resource consumption during differentiable search is effectively reduced through partial links, and the search convergence is made faster and more stable. While improving the system accuracy, it reduces the manpower and computing resource costs required for system development, improves efficiency and reduces system resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 An application environment diagram of a network structure construction method for a multi-task scenario in one embodiment;

[0043] Figure 2 A schematic diagram of a flow chart of a network structure construction method for a multi-task scenario in one embodiment;

[0044] Figure 3 A structural block diagram of a network structure construction device for a multi-task scenario in one embodiment;

[0045] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The network structure construction method for multi-task scenarios provided in this application can be applied to Figure 1 In the application environment shown. Figure 1 FIG. 1 is an application environment diagram for the network structure construction method for a multi-task scenario in an embodiment. Figure 1 As shown, the application environment includes a terminal 110 and a server 120. The terminal and the server communicate with each other through a network, and the communication network can be a wireless or wired communication network, such as an IP network, a cellular mobile communication network, etc., wherein the number of terminals and servers is not limited.

[0048] Among them, the terminal 110 can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. A training set can be obtained at the terminal 110 or the server 120, and the training set includes training sub-samples corresponding to multiple different target semantic tasks, and the training sub-samples include training sub-text data and training sub-label data; the training sub-text data corresponding to each target semantic task is input step by step into the multi-task network model of the network structure to be determined, and the sub-prediction results corresponding to each target semantic task are obtained, and the network parameters of the multi-task network model are adjusted according to the difference between the sub-prediction results and the corresponding training sub-label data, until the current target network parameters corresponding to the current network structure are obtained; the search space corresponding to the multi-task network model is obtained to form a differentiable network search space, and a verification set is obtained, and the verification set is adjusted by searching the differentiable network search space. The structural parameters of the multi-task network model corresponding to the current target network parameters are obtained. During the search, the implicit state vector of the multi-task network model is divided into multiple ordered sub-implicit state vectors, and the sub-implicit state vectors corresponding to the current search are obtained in a preset order. The sub-implicit state vectors are input into the corresponding network layer for training to obtain an updated multi-task network model, and the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined is returned step by step until the output result of the multi-task network model on the verification set meets the convergence condition, the target structural parameters are obtained, the network parameters matching the target structural parameters are obtained, and the trained multi-task network model is obtained according to the target structural parameters and the matching network parameters.

[0049] In one embodiment, Figure 2 As shown, a network structure construction method for a multi-task scenario is provided, and the method is applied to Figure 1 The terminal 110 or the server 120 in the example is used as an example to illustrate, and the following steps are included:

[0050] Step 210, obtaining a training set, the training set includes a plurality of training sub-samples corresponding to different target semantic tasks, the training sub-samples include training sub-text data and training sub-label data.

[0051] Among them, the training subsamples corresponding to multiple different target semantic tasks constitute the training set. The target semantic tasks are multiple different types of tasks corresponding to multi-task scenarios. For example, tasks of semantic analysis type include entity recognition, sentence classification, intent recognition, sentence pair similarity and other tasks. The number of target semantic tasks corresponds to the target recognition results of the multi-task network model of the network structure to be determined, wherein the multi-task network model can be a semantic analysis network. If the target recognition results of the semantic analysis network include entity recognition and purpose recognition of the input text, then the target semantic tasks include entity recognition tasks and purpose recognition tasks. For example, when receiving the user's question "How to take metformin", it is necessary to recognize both the entity "metformin" in it and the intention of the sentence, that is, the user wants to ask about the usage and dosage.

[0052] Specifically, different target semantic tasks have corresponding training sub-samples to adapt to multi-task scenarios. For example, the first target semantic task corresponds to the first training sub-sample, and the second target semantic task corresponds to the second training sub-sample. Each training sub-sample includes training sub-text data and training label data, wherein the training sub-label data is the task recognition result corresponding to the training text data whose corresponding task result has been determined. This task recognition result is the training sub-label data corresponding to the target semantic task.

[0053] Step 220, the training sub-text data corresponding to each target semantic task is input step by step into the multi-task network model of the network structure to be determined, and the sub-prediction results corresponding to each target semantic task are obtained. The network parameters of the multi-task network model are adjusted according to the differences between the sub-prediction results and the corresponding training sub-label data until the current target network parameters corresponding to the current network structure are obtained.

[0054] Specifically, the training sub-text data in the training sample is input into the multi-task network model of the network structure to be determined step by step in units of target semantic tasks, wherein the step by step means that the training sub-text data corresponding to the first target semantic task is input first to obtain the first sub-prediction result corresponding to the first target semantic task, and then the training sub-text data corresponding to the second target semantic task is input to obtain the second sub-prediction result corresponding to the second target semantic task, until the training sub-text data corresponding to each target semantic task is input step by step in sequence to obtain the corresponding sub-prediction result. Each sub-prediction result has corresponding training sub-label data, so as to calculate the sub-difference corresponding to each target semantic task, construct a loss function according to each sub-difference, and then back-propagate in the direction of minimizing the loss function, adjust the network parameters of the multi-task network model and continue training until the training end condition is met. By minimizing the training loss, the current target network parameters related to the structure of the multi-task network model, that is, the current optimal weight w, are obtained.

[0055] Step 230, obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, adjust the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set, divide the implicit state vector of the multi-task network model into a plurality of ordered sub-implicit state vectors during the search, obtain the sub-implicit state vectors corresponding to the current search in a preset order, input the sub-implicit state vectors into the corresponding network layer for training, and obtain an updated multi-task network model, return to the step of inputting the training sub-text data corresponding to each target semantic task step by step into the multi-task network model of the network structure to be determined, until the output result of the multi-task network model on the verification set meets the convergence condition, obtain the target structure parameters, obtain the network parameters matching the target structure parameters, and obtain the trained multi-task network model according to the target structure parameters and the matching network parameters.

[0056] Among them, a search space is defined, which contains various operators, including LSTM (Long Short-Term Memory), gated recurrent unit GRU, one-dimensional convolution, multi-head attention, etc. The kernel size of the operator can be 1, 3, 5, etc., and the number of attention heads can be 1, 2, 4, 8, etc.

[0057] Specifically, the multi-task network model is considered to be composed of multiple cells stacked together, and a cell is a directed graph consisting of N ordered nodes connected by directed edges, which continuously relaxes the search space. Each directed edge (i, j) represents an operator, which is considered to be a mixture of all sub-operations and can be realized by superposition of softmax weights. The following is the softmax formula:

[0058]

[0059] Among them, o(x) is the randomly initialized weight of each sub-operation. Due to the need of training, it cannot be limited to a number between 0 and 1. This formula converts it to a number between 0 and 1, so that the weights of all sub-operations add up to 1. The sub-operation mixed weight of the directed edge (i, j) is α (i,j) The dimension is |O|, which is the total number of sub-operations between directed edges (i, j); o() represents the current sub-operation. Update the structural parameters and network parameters, learn the optimal weight parameters, and the optimization goal is a two-level Bi-level optimization problem, that is,

[0060]

[0061]

[0062] The optimization method is cross gradient descent, along L train (w k-1 , α k-1 ) k-1 The gradient of w network parameters is updated once, along L train (w k , α k-1 ) for α k-1 The gradient of α updates the structural parameter α of the multi-task network model once.

[0063] During the search, the hidden state of the multi-task network model is partially linked. For example, if the hidden state vector includes 300 dimensions, the 300 dimensions are divided into 6 ordered sub-implicit state vectors, each of which includes 50 dimensions. Each time the parameters are adjusted, a sub-implicit state vector, that is, 50 dimensions, is selected to perform a one-step differentiable search. Next, another sub-implicit state vector is selected in sequence, and the sub-implicit state vectors are selected in sequence. The sub-implicit state vectors are input into the corresponding network layer for training to obtain an updated multi-task network model, and the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step until the output result of the multi-task network model on the validation set meets the convergence condition, the target structure parameters are obtained, and the network parameters matching the target structure parameters are obtained. The trained multi-task network model is obtained according to the target structure parameters and the matching network parameters. After the optimization is completed, the operator corresponding to the largest weight, that is, the target structure parameter, is activated, and other operators are removed to obtain the trained multi-task network model.

[0064] The above-mentioned network structure construction method for multi-task scenarios has the function of automatically discovering the network architecture that best suits the existing multi-task scenario data set. It can improve the accuracy of the multi-task system without manually trying many different models. It effectively reduces the resource consumption during differentiable search through partial links, and makes the search convergence faster and more stable. While improving the system accuracy, it reduces the manpower and computing resource costs required for system development, improves efficiency and reduces system resource utilization.

[0065] In one embodiment, the differentiable network search space is searched through at least one of the following sharing methods: matrix parameter sharing of multi-head attention in the differentiable network search space; when searching the pooling layer of the multi-task network model, multiple operators based on the capsule network share the parameters of the mapping network; the connection relationship between the nodes of the multi-task network model is obtained, and the nodes with the same starting node are grouped into node sets, and the operators corresponding to the nodes in different node sets share parameters.

[0066] Specifically, the three matrices (W_Q, W_K, W_V) of multi-head attention can share parameters, and the four operators based on the capsule network can share the parameters of a mapping network. For example, the operator 1->2 can be shared with 3->4; the rule for sharing is that the nodes do not share the same starting point, that is, they can be shared, and the nodes with the same starting node are grouped into node sets, and the operators corresponding to the nodes in different node sets share parameters. Parameter sharing effectively reduces resource consumption during differentiable search, and makes the search converge faster and more stable.

[0067] In one embodiment, in step 220, the training sub-text data corresponding to each target semantic task is input step by step into the multi-task network model of the network structure to be determined, and the sub-prediction results corresponding to each target semantic task are obtained, including: segmenting the current training sub-text data corresponding to the current target semantic task, mapping each segmentation to a corresponding vector to form a vector set; extracting semantic features from the vector set through an encoder, and obtaining the sub-prediction results corresponding to the current target semantic task according to the semantic features, wherein the current target semantic task is one of the target semantic tasks.

[0068] Specifically, the current training sub-text data corresponding to the current target semantic task is segmented, and a custom segmentation algorithm can be used, and the segmentation algorithms of different target semantic tasks can be different. Each segmentation is mapped to a corresponding vector, and a custom mapping algorithm can be used. When the current target semantic task is a different target semantic task, the corresponding encoders can be different or the same, so that different semantic features can be extracted for different target semantic tasks, and the sub-prediction result corresponding to the current target semantic task can be obtained according to the semantic features.

[0069] In this embodiment, the current training sub-text data corresponding to the current target semantic task is first segmented and mapped to the corresponding vector to form a vector set, and then the encoder extracts semantic features from the vector set to obtain the sub-prediction result corresponding to the current target semantic task. The diversification of word segmentation and encoders improves the convenience of obtaining corresponding sub-prediction results for each target semantic task, and different word segmentation algorithms and encoders can be flexibly configured for different target semantic tasks.

[0070] In one embodiment, in step 220, the training sub-text data corresponding to each target semantic task is input step by step into the multi-task network model of the network structure to be determined, and the sub-prediction results corresponding to each target semantic task are obtained, including: calculating the similarity between the current training sub-text data corresponding to the current target semantic task and the candidate text in the database, and obtaining similar sub-text data matching the current training sub-text data; inputting the first vector set corresponding to the current training sub-text data into the first encoder to extract semantic features to obtain the first semantic features, and inputting the second vector set corresponding to the similar sub-text data into the second encoder to extract semantic features to obtain the second semantic features; and obtaining the sub-prediction results corresponding to the current target semantic task according to the first semantic features and the second semantic features.

[0071] Specifically, the candidate text in the database can be a relatively standard text. The similar sub-text data corresponding to the training sub-text data can be obtained by similarity search. Because the expression is relatively standard, it is more effective to extract semantic features in the subsequent process. The sub-prediction results corresponding to the current target semantic task are obtained by combining the different semantic features obtained by the two encoders, thereby improving the accuracy of the sub-prediction results. The first encoder can be called a premise encoder, and the second encoder can be called a hypothesis encoder.

[0072] When the current target semantic task is a different target semantic task, the first encoder and the second encoder can be shared. By sharing the encoders, resource utilization is improved and training efficiency is improved. Since one input text forms two input texts, the corresponding target semantic task can also include semantic tasks based on text pairs, such as question-answering sentence tasks, sentence similarity calculation tasks, and probability tasks of one sentence under the condition of another sentence.

[0073] In this embodiment, similar sub-text data that matches the training sub-text data is obtained, and the sub-prediction results corresponding to the current target semantic task are obtained by combining the different semantic features obtained by the two encoders, thereby improving the accuracy of the sub-prediction results and the diversification of the target semantic task forms.

[0074] In one embodiment, the weights of the first encoder and the second encoder are shared.

[0075] Specifically, weight sharing refers to the sharing of convolution kernel parameters, that is, the convolution kernel parameters of the first encoder are the same as those of the second encoder. By reducing the number of parameters through weight sharing, the multi-task system mechanism and the weight sharing of the premise encoder and hypothesis encoder can reduce the memory usage and cost during the deployment of the multi-task system.

[0076] In one embodiment, adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data includes: obtaining the sub-prediction results and training sub-label data corresponding to each target semantic task to obtain the sub-differences corresponding to each target semantic task; obtaining the task weights corresponding to each target semantic task, weighting each sub-difference according to the task weights to obtain statistical sub-differences; and adjusting the network parameters of the multi-task network model according to the statistical sub-differences.

[0077] Specifically, the task weight corresponding to the target semantic task indicates the importance of the target semantic task. The larger the task weight, the higher the importance of the task. For example, for a text, the main task is to identify the entities in the text, and the secondary task is to identify the usage of the entities in the text. Then the task weight corresponding to the entity recognition task is greater than the task weight corresponding to the entity usage recognition task. The sub-differences are weighted by the task weights so that the weight coefficients corresponding to the important tasks are large. Therefore, when adjusting the network parameters of the multi-task network model according to the statistical sub-differences, the important tasks have a higher influence on adjusting the parameters.

[0078] In this embodiment, by configuring corresponding task weights for each target semantic task, the influence of different target semantic tasks on the adjustment of network parameters of the multi-task network model can be flexibly controlled.

[0079] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0080] In one embodiment, Figure 3 As shown, a network structure construction device for a multi-task scenario is provided, including: an acquisition module 310, a network parameter adjustment module 320, and a network structure construction module 330, wherein:

[0081] The acquisition module 310 is used to acquire a training set, where the training set includes training sub-samples corresponding to multiple different target semantic tasks, and the training sub-samples include training sub-text data and training sub-label data.

[0082] The network parameter adjustment module 320 is used to input the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step, obtain the sub-prediction results corresponding to each target semantic task, and adjust the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data until the current target network parameters corresponding to the current network structure are obtained.

[0083] The network structure construction module 330 is used to obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set. During the search, the implicit state vector of the multi-task network model is divided into multiple ordered sub-implicit state vectors, and the sub-implicit state vectors corresponding to the current search are obtained in a preset order. The sub-implicit state vectors are input into the corresponding network layer for training to obtain an updated multi-task network model, and the network parameter adjustment module is returned until the output result of the multi-task network model on the verification set meets the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matching network parameters.

[0084] In one embodiment, the network structure construction module 330 is also used to search the differentiable network search space through at least one of the following sharing methods: sharing matrix parameters of multi-head attention in the differentiable network search space; sharing the parameters of the mapping network based on multiple operators of the capsule network when searching the pooling layer of the multi-task network model; obtaining the connection relationship between the nodes of the multi-task network model, forming a node set with nodes having the same starting node, and sharing parameters of the operators corresponding to the nodes in different node sets.

[0085] In one embodiment, the network parameter adjustment module 320 is also used to segment the current training sub-text data corresponding to the current target semantic task, map each segmented word to a corresponding vector, and form a vector set; the encoder extracts semantic features from the vector set, and obtains the sub-prediction result corresponding to the current target semantic task based on the semantic features, wherein the current target semantic task is one of the target semantic tasks.

[0086] In one embodiment, the network parameter adjustment module 320 is also used to calculate the similarity between the current training sub-text data corresponding to the current target semantic task and the candidate text in the database, and obtain similar sub-text data matching the current training sub-text data; input the first vector set corresponding to the current training sub-text data into the first encoder to extract semantic features to obtain the first semantic features, and input the second vector set corresponding to the similar sub-text data into the second encoder to extract semantic features to obtain the second semantic features; and obtain the sub-prediction result corresponding to the current target semantic task based on the first semantic features and the second semantic features.

[0087] In one embodiment, the weights of the first encoder and the second encoder are shared.

[0088] In one embodiment, the network parameter adjustment module 320 is also used to obtain the sub-prediction results and training sub-label data corresponding to each target semantic task, and obtain the sub-differences corresponding to each target semantic task; obtain the task weights corresponding to each target semantic task, and weight each sub-difference according to the task weights to obtain statistical sub-differences; and adjust the network parameters of the multi-task network model according to the statistical sub-differences.

[0089] For the specific definition of the network structure construction device for multi-task scenarios, please refer to the definition of the network structure construction method for multi-task scenarios above, which will not be repeated here. Each module in the above-mentioned network structure construction device for multi-task scenarios can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0090] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store a training set. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a network structure construction method for a multi-task scenario is implemented.

[0091] Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In some embodiments, the computer device may be a terminal.

[0092] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: obtaining a training set, wherein the training set includes training subsamples corresponding to multiple different target semantic tasks, and the training subsamples include training subtext data and training sublabel data; inputting the training subtext data corresponding to each target semantic task into a multi-task network model of a network structure to be determined step by step, obtaining a sub-prediction result corresponding to each target semantic task, and adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction result and the corresponding training sublabel data until the current target network parameters corresponding to the current network structure are obtained; obtaining a search space corresponding to the multi-task network model to form a differentiable network search space, and obtaining a validation set. The verification set is used to adjust the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set. During the search, the implicit state vector of the multi-task network model is divided into a plurality of ordered sub-implicit state vectors, and the sub-implicit state vectors corresponding to the current search are obtained in a preset order. The sub-implicit state vectors are input into the corresponding network layer for training to obtain an updated multi-task network model, and the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step is returned until the output result of the multi-task network model on the verification set meets the convergence condition, the target structural parameters are obtained, the network parameters matching the target structural parameters are obtained, and the trained multi-task network model is obtained according to the target structural parameters and the matching network parameters.

[0093] In one embodiment, the differentiable network search space is searched through at least one of the following sharing methods: matrix parameter sharing of multi-head attention in the differentiable network search space; when searching the pooling layer of the multi-task network model, multiple operators based on the capsule network share the parameters of the mapping network; the connection relationship between the nodes of the multi-task network model is obtained, and the nodes with the same starting node are grouped into node sets, and the operators corresponding to the nodes in different node sets share parameters.

[0094] In one embodiment, the training sub-text data corresponding to each target semantic task is input step by step into a multi-task network model of a network structure to be determined, and the sub-prediction results corresponding to each target semantic task are obtained, including: segmenting the current training sub-text data corresponding to the current target semantic task, mapping each segmentation to a corresponding vector to form a vector set; extracting semantic features from the vector set through an encoder, and obtaining the sub-prediction results corresponding to the current target semantic task based on the semantic features, wherein the current target semantic task is one of the target semantic tasks.

[0095] In one embodiment, the training sub-text data corresponding to each target semantic task is input step by step into a multi-task network model of a network structure to be determined, and the sub-prediction results corresponding to each target semantic task are obtained, including: calculating the similarity between the current training sub-text data corresponding to the current target semantic task and the candidate text in the database, and obtaining similar sub-text data matching the current training sub-text data; inputting a first vector set corresponding to the current training sub-text data into a first encoder to extract semantic features to obtain a first semantic feature, and inputting a second vector set corresponding to the similar sub-text data into a second encoder to extract semantic features to obtain a second semantic feature; and obtaining the sub-prediction result corresponding to the current target semantic task according to the first semantic feature and the second semantic feature.

[0096] In one embodiment, the weights of the first encoder and the second encoder are shared.

[0097] In one embodiment, adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data includes: obtaining the sub-prediction results and training sub-label data corresponding to each target semantic task to obtain the sub-differences corresponding to each target semantic task; obtaining the task weights corresponding to each target semantic task, weighting each sub-difference according to the task weights to obtain statistical sub-differences; and adjusting the network parameters of the multi-task network model according to the statistical sub-differences.

[0098] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining a training set, the training set including training subsamples corresponding to multiple different target semantic tasks, the training subsamples including training subtext data and training sublabel data; step-by-step inputting the training subtext data corresponding to each target semantic task into a multi-task network model of a network structure to be determined, obtaining a sub-prediction result corresponding to each target semantic task, adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction result and the corresponding training sublabel data, until the current target network parameters corresponding to the current network structure are obtained; obtaining a search space corresponding to the multi-task network model to form a differentiable network search space, obtaining a verification set, and According to the verification set, the structural parameters of the multi-task network model corresponding to the current target network parameters are adjusted by searching the differentiable network search space. During the search, the implicit state vector of the multi-task network model is divided into a plurality of ordered sub-implicit state vectors, and the sub-implicit state vectors corresponding to the current search are obtained in a preset order. The sub-implicit state vectors are input into the corresponding network layer for training to obtain an updated multi-task network model, and the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step is returned until the output result of the multi-task network model on the verification set meets the convergence condition, the target structural parameters are obtained, the network parameters matching the target structural parameters are obtained, and the trained multi-task network model is obtained according to the target structural parameters and the matching network parameters.

[0099] In one embodiment, the differentiable network search space is searched through at least one of the following sharing methods: matrix parameter sharing of multi-head attention in the differentiable network search space; when searching the pooling layer of the multi-task network model, multiple operators based on the capsule network share the parameters of the mapping network; the connection relationship between the nodes of the multi-task network model is obtained, and the nodes with the same starting node are grouped into node sets, and the operators corresponding to the nodes in different node sets share parameters.

[0100] In one embodiment, the training sub-text data corresponding to each target semantic task is input step by step into a multi-task network model of a network structure to be determined, and the sub-prediction results corresponding to each target semantic task are obtained, including: segmenting the current training sub-text data corresponding to the current target semantic task, mapping each segmentation to a corresponding vector to form a vector set; extracting semantic features from the vector set through an encoder, and obtaining the sub-prediction results corresponding to the current target semantic task based on the semantic features, wherein the current target semantic task is one of the target semantic tasks.

[0101] In one embodiment, the training sub-text data corresponding to each target semantic task is input step by step into a multi-task network model of a network structure to be determined, and the sub-prediction results corresponding to each target semantic task are obtained, including: calculating the similarity between the current training sub-text data corresponding to the current target semantic task and the candidate text in the database, and obtaining similar sub-text data matching the current training sub-text data; inputting a first vector set corresponding to the current training sub-text data into a first encoder to extract semantic features to obtain a first semantic feature, and inputting a second vector set corresponding to the similar sub-text data into a second encoder to extract semantic features to obtain a second semantic feature; and obtaining the sub-prediction result corresponding to the current target semantic task according to the first semantic feature and the second semantic feature.

[0102] In one embodiment, the weights of the first encoder and the second encoder are shared.

[0103] In one embodiment, adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data includes: obtaining the sub-prediction results and training sub-label data corresponding to each target semantic task to obtain the sub-differences corresponding to each target semantic task; obtaining the task weights corresponding to each target semantic task, weighting each sub-difference according to the task weights to obtain statistical sub-differences; and adjusting the network parameters of the multi-task network model according to the statistical sub-differences.

[0104] This application can be applied to smart government affairs and smart security, thereby promoting the construction of smart cities.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a network structure for a multi-task scenario, the method comprising: Acquire a training set, wherein the training set includes a plurality of training sub-samples corresponding to different target semantic tasks, and the training sub-samples include training sub-text data and training sub-label data; Inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure step by step, obtaining the sub-prediction results corresponding to each target semantic task, and adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data, until obtaining the current target network parameters corresponding to the current network structure; The method comprises the steps of: obtaining a search space corresponding to the multi-task network model, forming a differentiable network search space, obtaining a verification set, adjusting the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set, dividing the implicit state vector of the multi-task network model into a plurality of ordered sub-implicit state vectors during the search, obtaining the sub-implicit state vectors corresponding to the current search in a preset order, inputting the sub-implicit state vectors into the corresponding network layer for training, and obtaining an updated multi-task network model, returning to the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step, until the output result of the multi-task network model on the verification set meets the convergence condition, obtaining the target structural parameters, obtaining the network parameters matching the target structural parameters, and obtaining the trained multi-task network model according to the target structural parameters and the matching network parameters.

2. The method according to claim 1, characterized in that The search of the differentiable network search space is implemented by at least one of the following sharing methods: Matrix parameter sharing of multi-head attention in the differentiable network search space; The search of the pooling layer of the multi-task network model is based on multiple operators of the capsule network, sharing the parameters of the mapping network; The connection relationship between the nodes of the multi-task network model is obtained, the nodes having the same starting node are grouped into node sets, and the operators corresponding to the nodes in different node sets share parameters.

3. The method according to claim 1, characterized in that The step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure in stages to obtain the sub-prediction results corresponding to each target semantic task includes: Segment the current training subtext data corresponding to the current target semantic task, and map each segmented word to a corresponding vector to form a vector set; The encoder extracts semantic features from the vector set, and obtains a sub-prediction result corresponding to the current target semantic task based on the semantic features, wherein the current target semantic task is one of the target semantic tasks.

4. The method according to claim 1, characterized in that The step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure in stages to obtain the sub-prediction results corresponding to each target semantic task includes: Calculating the similarity between the current training subtext data corresponding to the current target semantic task and the candidate texts in the database, and obtaining similar subtext data matching the current training subtext data; Inputting the first vector set corresponding to the current training subtext data into a first encoder to extract semantic features to obtain a first semantic feature, and inputting the second vector set corresponding to the similar subtext data into a second encoder to extract semantic features to obtain a second semantic feature; A sub-prediction result corresponding to the current target semantic task is obtained according to the first semantic feature and the second semantic feature.

5. The method according to claim 4, characterized in that The weights of the first encoder and the second encoder are shared.

6. The method according to claim 1, characterized in that The adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data includes: Obtain the sub-prediction results and training sub-label data corresponding to each target semantic task, and obtain the sub-differences corresponding to each target semantic task; Obtain the task weights corresponding to each target semantic task, and weight each sub-difference according to the task weights to obtain statistical sub-differences; The network parameters of the multi-task network model are adjusted according to the statistical sub-differences.

7. A network structure construction device for multi-task scenarios, characterized in that: The device comprises: An acquisition module is used to acquire a training set, wherein the training set includes a plurality of training sub-samples corresponding to different target semantic tasks, and the training sub-samples include training sub-text data and training sub-label data; A network parameter adjustment module is used to input the training sub-text data corresponding to each target semantic task into the multi-task network model of the to-be-determined network structure in steps, obtain the sub-prediction results corresponding to each target semantic task, and adjust the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data until the current target network parameters corresponding to the current network structure are obtained; A network structure construction module is used to obtain a search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set. During the search, the implicit state vector of the multi-task network model is divided into a plurality of ordered sub-implicit state vectors, and the sub-implicit state vectors corresponding to the current search are obtained in a preset order. The sub-implicit state vectors are input into the corresponding network layer for training to obtain an updated multi-task network model, and the network parameter adjustment module is returned until the output result of the multi-task network model on the verification set meets the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matching network parameters.

8. The device according to claim 7, characterized in that The network structure building module searches for the differentiable network search space by at least one of the following sharing methods: Matrix parameter sharing of multi-head attention in the differentiable network search space; The search of the pooling layer of the multi-task network model is based on multiple operators of the capsule network, sharing the parameters of the mapping network; The connection relationship between the nodes of the multi-task network model is obtained, the nodes having the same starting node are grouped into node sets, and the operators corresponding to the nodes in different node sets share parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that:When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images