Twin network structure search method and semantic matching method for data intelligence

Abstract

The present invention provides a twin network structure search method and a semantic matching method for data intelligence. The following steps are included: constructing a twin network model according to a target task; wherein the twin network model includes five network structures, and the network structure is composed of a number of operation nodes; defining a search space for the network structure, and the search space is composed of multiple operation nodes; setting the operation constraints of the twin network model; the controller samples the given search space to obtain a network model, trains the network model and the controller, updates the network weight parameters until convergence, and ensures that the network model meets the operation constraints during the training process; according to the updated network weight parameters, determine the target twin network model. This method can adaptively construct the optimal twin network model, effectively reducing the cost and difficulty of manual search, and can be used for text semantic matching.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a twin network structure search method, a semantic matching method, a device and a medium. Background Art

[0002] In the field of natural language processing, twin network models are often used for text semantic matching, such as question-answering systems and search engines. In the past decade, manually designed twin network models have achieved good results, but the optimal twin network model is often different for text semantic matching in different fields, and it takes several days to manually build an optimal network model.

[0003] In order to speed up the above manual search process, neural structure search can automate the network model design process and reduce the labor cost of optimal model design. However, neural structure search still has problems such as low search efficiency and high resource consumption, and research on twin network structure search is still relatively scarce. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a twin network structure search method for data intelligence.

[0005] In order to achieve the purpose of the present invention, the twin network structure search method for data intelligence provided by the present invention includes:

[0006] A twin network model is constructed according to the target task. The network structure of the twin network model includes an input layer, an encoding layer, an aggregation layer, an interaction layer and an output layer. The input layer is used to map a discrete text sequence into a continuous digital vector. A word embedding vector can be obtained from the input layer. The encoding layer is used to combine the semantic information of the text context and the semantic association between text pairs to generate a sentence embedding vector. The aggregation layer is used to aggregate the word embedding vector and the sentence embedding vector obtained by the input layer and the encoding layer to obtain global semantic reasoning information. The interaction layer is used to fuse the sentence semantic reasoning information output by the aggregation layer to generate a feature vector that can identify the semantic relevance between text pairs. The output layer is used to use an activation function to perform the final feature extraction and predict the label of the input sentence pair.

[0007] Defining a search space of the twin network model, wherein the search space includes a plurality of operation nodes;

[0008] Setting constraints for the operation of the twin network model;

[0009] The controller samples the given search space to obtain a twin network model, trains the twin network model and the controller, updates the network weight parameters until convergence, and ensures that the twin network model satisfies the operating constraints during the training process;

[0010] According to the updated network weight parameters, a target twin network model is determined, and the target twin network model is used to match the text semantics.

[0011] Further, in the search space of the network structure, the coding layer and the aggregation layer in the network structure are network structures to be searched, and correspond to different search spaces respectively;

[0012] The operation node sets in the coding layer search space are divided into two categories, self-coding operation node sets and cross-sentence coding operation node sets;

[0013] The autoencoder operation node set includes a convolutional neural network layer, a recurrent neural network layer, a maximum pooling layer, a multi-head attention mechanism layer, a skip connection layer and a zeroing layer;

[0014] The cross-sentence encoding operation node set includes a dot product attention mechanism layer, a concatenation layer, a point-by-point multiplication layer, a point-by-point addition layer, and a point-by-point subtraction layer;

[0015] The set of operation nodes in the aggregation layer search space includes a maximum pooling layer, an average pooling layer, and a self-attention mechanism pooling layer.

[0016] Furthermore, the operation constraints include that the prediction speed of the twin network model and the resources occupied by the operation must meet the set thresholds.

[0017] Furthermore, ensuring that the network model satisfies the operation constraints during the training process includes:

[0018] If the network model cannot satisfy the operation constraint condition, the loss value is set to zero;

[0019] If the network models can all satisfy the operating constraints, the loss value remains unchanged.

[0020] Further, the controller performs sampling on the given search space to obtain a network model, including:

[0021] During the sampling process, the controller selects a plurality of operation nodes in the coding layer search space and connects them, and selects an operation node in the aggregation layer search space.

[0022] Furthermore, in the updating of the network weight parameters, the operation nodes of the same type in the search space share the same network weight parameters; the convolutional neural network layer in the coding layer search space includes a variety of product kernels of different sizes, which share the same network weight parameters, and each network layer in the recurrent neural network layer shares the same network weight parameters. The multi-head attention mechanism layer includes four-head and eight-head attention mechanism layers, and the four-head and eight-head multi-head attention mechanism layers share the same network weight parameters.

[0023] Furthermore, in the sampling process of obtaining the network model by sampling on the given search space, the controller selects a plurality of operation nodes in the coding layer search space and connects them, and selects an operation node in the aggregation layer search space.

[0024] Furthermore, when training the twin network model and the controller, the twin network model not only learns the one-hot true label on the validation set, but also learns the predicted output of the large pre-trained model on the validation set; then the loss value of the twin network model on the validation set includes two parts: the true label score and the predicted output score of the large pre-trained model.

[0025] According to a second aspect of the present invention, a text semantic matching method is provided, comprising:

[0026] Obtain training data for text semantic matching;

[0027] The above twin network structure search method is used for training to obtain a target twin network model for text semantic matching;

[0028] The target twin network model is used to match and predict the text data to be processed to obtain the text semantic matching results.

[0029] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements any of the aforementioned data intelligence-oriented twin network structure search methods or the aforementioned text semantic matching methods.

[0030] The present invention also provides a device, comprising a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, it implements any of the aforementioned data intelligence-oriented twin network structure search methods or the aforementioned text semantic matching methods.

[0031] Compared with the prior art, the present invention has at least the following advantages and technical effects:

[0032] First, the present invention designs a new search space for the twin network model encoding layer, which includes two major types of operation sets, allowing the model to simultaneously select self-encoding and cross-sentence encoding operations. Secondly, the present invention also introduces a search space containing a variety of aggregation operation sets in the twin network aggregation layer, allowing the model to autonomously learn the optimal aggregation operation, so that the method of the present invention can align the semantic information between inputs and provide deeper semantic connections between input pairs. And in order to speed up the process of neural structure search, operations of the same type in the search space share the same network weights. At the same time, the present invention adds twin network model operation constraints during the search. If the performance of the twin network model on the verification set cannot meet the constraints, the loss value fed back to the controller will be set to 0, thereby increasing the speed of the controller's search for a better twin network model and accelerating the matching speed of the semantic matching task. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of the steps of the twin network structure search method for data intelligence in an implementation method.

[0034] Figure 2 Schematic diagram of the network structure of the twin network model described in the present invention.

[0035] Figure 3 Schematic diagram of the cross-sentence encoding operation in the encoding layer of the Siamese network model. DETAILED DESCRIPTION

[0036] The following is a further description of the embodiments of the present invention in conjunction with the examples, but the implementation of the present invention is not limited thereto.

[0037] See also Figure 1 The twin network structure search method for data intelligence provided by the present invention is used for text semantic matching and comprises the following steps:

[0038] S1, build a twin network model according to the target task.

[0039] See also Figure 2The twin network model has five network structures, namely input layer, encoding layer, aggregation layer, interaction layer and output layer. The input layer is used to map discrete text sequences into continuous digital vectors. The word embedding vector can be obtained from the input layer. The encoding layer is used to generate a higher-level sentence vector embedding representation based on the word embedding vector, combining the semantic information of the text context and the semantic association between text pairs. The encoding layer mainly solves the semantic problems of words in the actual context. The aggregation layer is used to aggregate the word embedding vectors and sentence embedding vectors obtained by the input layer and the encoding layer to obtain global semantic reasoning information. The interaction layer is used to use a fully connected neural network to fuse the sentence semantic reasoning information output by the aggregation layer to generate a feature vector that can identify the semantic relevance between text pairs. The output layer is used to input the final feature vector into the classifier to obtain the predicted category label.

[0040] In some embodiments of the present invention, since the commonly used text languages ​​in specific target tasks are Chinese and English, a specific Word2Vec pre-trained model is loaded according to the language type, and the text is vectorized to obtain a vectorized representation of the input.

[0041] According to the number of labels of the specific target task, the classifier of the output layer is determined. In some embodiments of the present invention, if the target task is a binary classification task, the classifier of the output layer is a Sigmoid function; if the target task is a multi-classification task, the classifier of the output layer is a Softmax function.

[0042] S2, defining a search space of the network structure, wherein the search space includes a plurality of operation nodes.

[0043] In some embodiments of the present invention, the coding layer and the aggregation layer are the network structures to be searched of the twin network model, corresponding to different search spaces respectively.

[0044] The coding layer search space is defined as a fully connected directed acyclic graph, including K operation nodes. In some embodiments of the present invention, K is set to 10. The directed edges between the operation nodes<i,j> Indicates that the output of operation node i is the input of operation node j. If an operation node contains multiple directed edges, these inputs will be added point by point. For each operation node in the search space, the controller first determines the type of operation node, which is divided into two types: self-encoding operation node and cross-sentence encoding operation node. Among them, the self-encoding operation node refers to an operation node whose input is the output of the previous layer operation node. The cross-sentence encoding operation node refers to an operation node whose input includes the output of the previous layer operation node and the input of the same layer operation node of the encoding layer, and performs attention mechanism calculation.

[0045] If the operation node is a self-encoding operation node, the optional operation set includes a convolutional neural network layer, a recurrent neural network layer, a maximum pooling layer, a multi-head attention mechanism layer, a jump connection layer, and a zeroing layer. Among them, the convolutional neural network layer includes convolution operations with kernel sizes of 1, 3, and 5. The convolutional neural network layer is used to expand the receptive field of the semantic features of the sentence and improve the ability to capture long-distance dependencies; the recurrent neural network layer includes a long short-term memory network and a gated recurrent network. The recurrent neural network layer is used to capture the semantic information of the sentence context; the maximum pooling layer includes pooling operations with window sizes of 3 and 5. The maximum pooling layer is used to extract the features with the richest semantic information and reduce the dimension of the features. The multi-head attention mechanism layer includes four-head and eight-head attention mechanisms. The multi-head attention mechanism layer is used to capture the semantic information of the long-distance dependency of the sentence. The jump connection layer is used to support residual operations. The zeroing layer sets the output of a certain operation node to 0. Through the zeroing operation, the speed of controller training can be accelerated, and the prediction set loss of the twin network model that does not meet the constraints is set to zero, so that the controller does not learn.

[0046] If the operation node is a cross-sentence encoding operation node, the optional operation set includes a dot product attention mechanism layer, a concatenation layer, a point-by-point multiplication layer, a point-by-point addition layer, and a point-by-point subtraction layer, through which semantic information is aggregated with another sentence. Figure 3 As shown, assuming that the operation node 3 of text one in the coding layer is a cross-sentence coding operation, it will perform a cross-sentence coding operation with the operation node 3 of text two in the coding layer.

[0047] The aggregation layer search space is defined as a set of operation nodes of multiple aggregation operations, including a maximum pooling layer, an average pooling layer, and a self-attention mechanism pooling layer. In addition to selecting one of the aggregation operations at the aggregation layer, the controller also selects the output of several operation nodes in the encoding layer as the input of the aggregation layer. In some embodiments of the present invention, the output of the encoding layer is specifically input to the controller, and the controller outputs the number of the encoding layer.

[0048] S3, setting the operating constraints of the twin network model.

[0049] The controller will preset certain constraints when searching for the twin network model. In some embodiments of the present invention, the set constraints are that the prediction speed of the twin network model on the verification set is greater than the set threshold, and the resources required for the twin network model to run on the verification set are less than the set threshold. In some embodiments of the present invention, the setting of the threshold mainly depends on the requirements of the actual application scenario. For example, for scenarios with high real-time performance, the prediction speed must be specified. It can also be determined based on the average characteristics of the sample. The following STS data set embodiment is based on sample characteristics.

[0050] S4, the controller samples in the given search space to obtain a network model, trains the network model and the controller, updates the network weight parameters until convergence, and ensures that the network model satisfies the constraints during the training process.

[0051] (1) Using a recurrent neural network structure as a controller, in some embodiments of the present invention, a long short-term memory network is selected to perform upsampling in the search space to generate a network model to be trained.

[0052] (2) In each round, the controller performs upsampling in the encoding layer and the aggregation layer to generate a network model. The controller makes three-step judgments for each operation node in the search space of the encoding layer. The first step is to determine whether the operation node is a self-encoding operation node or a cross-sentence encoding operation node. The second step is to determine the specific operation operation. The third step is to determine which operation nodes to establish directed edges with. The controller makes two-step judgments in the search space of the aggregation layer. The first step is to determine which operation node outputs in the encoding layer are selected as the input of the aggregation layer. The second step is to determine which specific aggregation operation is selected. After the controller completes sampling, the generated network model will contain several operation nodes in the encoding layer and the aggregation layer.

[0053] (3) The network model generated by controller sampling is trained on the training set, and the network weight parameters of the operation nodes are updated. In order to speed up the convergence of the network model, the same type of operation nodes in the encoding layer share the same network weight parameters. Convolutional neural network layers with kernel sizes of 1, 3, and 5 all share the same network weight parameters, and the four-head and eight-head multi-head attention mechanism layers share the same network weight parameters.

[0054] (4) The trained network model is predicted and scored on the validation set, and the loss value obtained is used as a reward to update the network weight parameters of the controller.

[0055] (5) Repeat steps (3) to (4) until the network model and controller converge.

[0056] The network model is guaranteed to meet the operating constraints during the training process. If the prediction speed or operating resources of the network model on the validation set do not meet the preset constraints, the loss value is set to 0; if the prediction speed and operating resources of the network model on the validation set both meet the preset constraints, the loss value remains unchanged.

[0057] In some embodiments of the present invention, in order to enable the controller to learn how to generate a better network model more quickly, the network model not only learns the one-hot encoded true label, but also learns the predicted output of the large pre-trained model on the validation set. The large pre-trained model is pre-fine-tuned on the target task dataset. Therefore, the loss value of the twin network model sampled and generated by the controller on the validation set includes two parts: the true label score and the predicted output score of the large pre-trained model. The controller can get more feedback from a single sample.

[0058] Among them, the large-scale pre-trained model adopts the existing model, such as the RoBERTa model used in the following "text semantic matching method" embodiment.

[0059] S5, determine the target twin network model based on the updated network weight parameters.

[0060] After updating the network weight parameters, the controller resamples the network model, and retrains the network model according to the target task data set to obtain the optimal twin network model for the target task.

[0061] The present invention also provides a text semantic matching method. The existing text semantic matching method usually parses, encodes and matches the text input by the user in a preset twin network model. Although good results can be obtained, it can only be used for a single field. If it is used in other fields, the previously preset model may not necessarily achieve good results. The semantic matching method provided by the present invention can make full use of the data features of each field and automatically and quickly search for the optimal twin network model.

[0062] The text semantic matching method provided by the present invention comprises the following steps:

[0063] Step 1: Obtain training data for text semantic matching;

[0064] Step 2: Use the twin network structure search method provided in the above embodiment to perform training to obtain a target twin network model for text semantic matching;

[0065] Step 3: Use the target twin network model for text semantic matching to perform matching prediction on the text pair data to be processed to obtain the text semantic matching result.

[0066] Specifically, in some embodiments of the present invention, taking the text training data of a public data set (such as an STS data set) as a model input to implement text semantic matching as an example, the specific process of the text semantic matching method is as follows:

[0067] M1, obtain training data for text semantic matching. The training data is text training data of the common public dataset STS. The STS dataset is a dataset released by the International Semantic Evaluation Group in 2017, which contains the English dataset used in the STS task between 2012 and 2017.

[0068] M2, based on the text training data obtained by M1, uses the twin network structure search method for training to obtain the target twin network model for text semantic matching;

[0069] M3 uses the target twin network model obtained in M2 to perform matching prediction on the text test data. The test data is the text test data of STS, and obtains the semantic matching results of the test text.

[0070] The above-mentioned M2 can be implemented by using the above-mentioned S1-S5, and in some embodiments of the present invention, specifically includes:

[0071] S1, build a twin network model according to the target task. In this embodiment, the text in the STS is English, and the English Word2Vec pre-trained model is loaded to vectorize the text to obtain a vectorized representation of the input.

[0072] STS divides sentence similarity into six levels from 0 to 5, and determines the classifier of the output layer as the Softmax function.

[0073] S2, defining the search space of the network structure. The encoding layer and aggregation layer of the twin network model are the network structures to be searched, corresponding to different search spaces respectively.

[0074] The coding layer search space is defined as a fully connected directed acyclic graph, including K operation nodes. In this embodiment, K is set to 10. The directed edges between the operation nodes<i,j> It means that the output of operation node i is the input of operation node j. If an operation node contains multiple directed edges, these inputs will be added point by point. For each operation node in the search space, the controller first determines the type of operation node, which is divided into two types: self-encoding operation node and cross-sentence encoding operation node.

[0075] If the operation node is an autoencoder operation node, the optional operation set includes convolutional neural network layer, recurrent neural network layer, maximum pooling layer, multi-head attention mechanism layer, skip connection layer and zeroing layer. Among them, the convolutional neural network layer includes convolution operations with kernel sizes of 1, 3, and 5, the recurrent neural network layer includes long short-term memory network and gated recurrent network, the maximum pooling layer includes pooling operations with window sizes of 3 and 5, the multi-head attention mechanism layer includes four-head and eight-head attention mechanisms, the skip connection layer is used to support residual operations, and the zeroing layer sets the output of a certain operation node to 0.

[0076] If the operation node is a cross-sentence encoding operation node, the optional operation set includes dot product attention mechanism layer, concatenation layer, point-by-point multiplication layer, point-by-point addition layer, and point-by-point subtraction layer. Figure 3 As shown, assuming that the operation node 3 of text one in the coding layer is a cross-sentence coding operation, it will perform a cross-sentence coding operation with the operation node 3 of text two in the coding layer.

[0077] The search space of the aggregation layer is defined as a set of operation nodes of multiple aggregation operations, including the maximum pooling layer, the average pooling layer, and the self-attention mechanism pooling layer. In addition to selecting an aggregation operation in the aggregation layer, the controller also selects the output of several operation nodes in the encoding layer as the input of the aggregation layer.

[0078] S3, setting the operating constraints of the twin network model. The controller will preset certain constraints when searching for the twin network model: in this embodiment, the average length of STS sentences is 10, the maximum length is 56, according to the data characteristics, the prediction speed threshold of the twin network model on the validation set is set to 50 milliseconds per sample, and the resource threshold required for the twin network model to run on the validation set is set to 512MB of GPU video memory.

[0079] S4, the controller samples in the given search space to obtain a network model, trains the network model and the controller, updates the network weight parameters until convergence, and ensures that the network model satisfies the constraints during the training process.

[0080] (1) A recurrent neural network structure is used as a controller. In this embodiment, a long short-term memory network is selected to generate a network model to be trained.

[0081] (2) In each round, the controller samples and generates a network model in the encoding layer and the aggregation layer. The controller makes three-step judgments for each operation node in the encoding layer search space. The first step is to determine whether the operation node is a self-encoding operation node or a cross-sentence encoding operation node. The second step is to determine the specific operation operation. The third step is to determine which operation nodes to establish directed edges with. The controller makes two-step judgments in the aggregation layer search space. The first step is to determine which operation node outputs in the encoding layer are selected as the input of the aggregation layer. The second step is to determine which specific aggregation operation is selected. After the controller completes sampling, the generated network model will contain several operation nodes in the encoding layer and the aggregation layer.

[0082] (3) The network model generated by the controller sampling is trained on the training set, and the network weight parameters of the operation nodes are updated. In order to speed up the convergence of the network model, the operation nodes of the same type in the encoding layer share the same network weight parameters. The convolutional neural network layers with kernel sizes of 1, 3, and 5 all share the same network weight parameters, and the four-head and eight-head multi-head attention mechanism layers share the same network weight parameters.

[0083] (4) The trained network model is predicted and scored on the validation set, and the loss value obtained is used as a reward to update the network weight parameters of the controller.

[0084] (5) Repeat steps (3) to (4) until the network model and controller converge.

[0085] The network model is guaranteed to meet the running constraints during the training process. If the prediction speed of the network model on the validation set cannot reach 50 milliseconds per sample, or the running resources are higher than 512MB of GPU memory, the loss value is set to 0.

[0086] In addition to learning the one-hot encoded true labels, the network model also learns the predicted output of the large pre-trained model on the validation set. In some embodiments of the present invention, RoBERTa is selected as the large pre-trained model. RoBERTa is pre-fine-tuned on the target task dataset. Therefore, the loss value of the sub-model on the validation set consists of two parts: the true label score and the predicted output score of RoBERTa.

[0087] S5, determining the target twin network model according to the updated network weight parameters. After updating the network weight parameters, the controller resamples the network model, and retrains the network model according to the STS text training data to obtain the optimal twin network model for STS.

[0088] Compared with the current semantic matching methods that commonly use pre-trained language models for feature extraction and manually design complex network structures for fine-tuning according to specific scenarios, the solution of the present invention does not require manual design of network structures. Instead, the twin network structure search method automatically searches for the optimal network structure according to specific scenarios, greatly reducing labor costs while also improving the effect of semantic matching.

[0089] An embodiment of the present invention also provides a storage medium, which may be a storage medium such as ROM, RAM, disk, or CD. The storage medium stores one or more programs. When the program is executed by a processor, the data intelligence-oriented twin network structure search method or text semantic matching method provided in the above embodiment is implemented.

[0090] An embodiment of the present invention also provides a device, which may be a desktop computer, a laptop computer, a smart phone, a PDA handheld terminal, a tablet computer or other terminal device with a display function. The device includes a processor and a memory, and the memory stores one or more programs. When the processor executes the program stored in the memory, the data intelligence-oriented twin network structure search method or text semantic matching method provided in the above embodiment is implemented.

[0091] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A twin network structure search method for data intelligence, characterized in that: The following steps are involved: A twin network model is constructed according to the target task. The network structure of the twin network model includes an input layer, an encoding layer, an aggregation layer, an interaction layer and an output layer. The input layer is used to map a discrete text sequence into a continuous digital vector. A word embedding vector can be obtained from the input layer. The encoding layer is used to combine the semantic information of the text context and the semantic association between text pairs to generate a sentence embedding vector. The aggregation layer is used to aggregate the word embedding vector and the sentence embedding vector obtained by the input layer and the encoding layer to obtain global semantic reasoning information. The interaction layer is used to fuse the sentence semantic reasoning information output by the aggregation layer to generate a feature vector that can distinguish the semantic relevance between text pairs. The output layer is used to input the final feature vector into a classifier to obtain a predicted category label. Defining a search space of the twin network model, wherein the search space includes a plurality of operation nodes; Setting operating constraints of the twin network model; The controller samples the given search space to obtain a twin network model, trains the twin network model and the controller, updates the network weight parameters until convergence, and ensures that the twin network model satisfies the operating constraints during the training process; According to the updated network weight parameters, a target twin network model is determined, and the target twin network model is used to match the text semantics.

2. The twin network structure search method for data intelligence according to claim 1 is characterized in that: In the search space of the network structure, the coding layer and the aggregation layer in the network structure are network structures to be searched, and correspond to different search spaces respectively; The operation node sets in the coding layer search space are divided into two categories, including self-coding operation node sets and cross-sentence coding operation node sets; The self-encoding operation node set includes a convolutional neural network layer, a recurrent neural network layer, a maximum pooling layer, a multi-head attention mechanism layer, a skip connection layer and a zeroing layer; the cross-sentence encoding operation node set includes a dot product attention mechanism layer, a concatenation layer, a point-by-point multiplication layer, a point-by-point addition layer, and a point-by-point subtraction layer; The set of operation nodes in the aggregation layer search space includes a maximum pooling layer, an average pooling layer, and a self-attention mechanism pooling layer.

3. The twin network structure search method for data intelligence according to claim 2 is characterized in that: In the updating of network weight parameters, operation nodes of the same type in the search space share the same network weight parameters; the convolutional neural network layer in the encoding layer search space includes product kernels of multiple different sizes, which share the same network weight parameters, and each network layer in the recurrent neural network layer shares the same network weight parameters. The multi-head attention mechanism layer includes four-head and eight-head attention mechanism layers, and the four-head and eight-head multi-head attention mechanism layers share the same network weight parameters.

4. The twin network structure search method for data intelligence according to claim 1 is characterized in that: The operation constraints include that the prediction speed of the twin network model and the resources occupied by the operation must meet the set thresholds.

5. The twin network structure search method for data intelligence according to claim 1 is characterized in that: The step of ensuring that the network model satisfies the operation constraints during the training process includes: If the twin network model cannot meet the operating constraints, the loss value is set to zero; If the network models can all satisfy the operating constraints, the loss value remains unchanged.

6. The twin network structure search method for data intelligence according to claim 1 is characterized in that: In the sampling process of obtaining the network model by sampling on the given search space, the controller selects multiple operation nodes in the coding layer search space and connects them, and selects one operation node in the aggregation layer search space.

7. The twin network structure search method for data intelligence according to any one of claims 1 to 6, characterized in that: When training the twin network model and controller, in addition to learning the one-hot true label on the validation set, the twin network model also learns the predicted output of the large pre-trained model on the validation set; the loss value of the twin network model on the validation set includes two parts: the true label score and the predicted output score of the large pre-trained model.

8. A text semantic matching method, characterized in that: The following steps are involved: Obtain training data for text semantic matching; The twin network structure search method according to any one of claims 1 to 7 is used for training to obtain a target twin network model for text semantic matching; The target twin network model is used to match and predict the text data to be processed to obtain the text semantic matching results.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the data intelligence-oriented twin network structure search method as described in any one of claims 1 to 7 or the text semantic matching method as described in claim 8.

10. A device comprising a processor and a memory for storing a program executable by the processor, characterized in that When the processor executes the program stored in the memory, it implements the data intelligence-oriented twin network structure search method described in any one of claims 1 to 7 or the text semantic matching method described in claim 8.

Images