Pre-training and gating neural network based utterance recognition method, system, device and medium

Abstract

The application provides a pre-training and gate neural network-based sentence step recognition method, system, device and medium, data in a target text section is collected for preprocessing, and is labeled according to a preset sentence step; long, difficult and complex sentences in the target text section are discriminated and split; an ERNIE_AT-GRU sentence step automatic recognition model is built; the split data is input into the ERNIE_AT-GRU model for training, sentence step recognition test is performed on test data after round training, and a sentence step recognition result is obtained; in the specific operation of the pre-training model and the gate neural network-based sentence step recognition method, the ERNIE pre-training model combining large-scale text content and a knowledge graph is used to learn deep text semantics, the disadvantages that a traditional machine learning does not sufficiently mine and utilize the internal relationship and characteristics between words are improved, compared with the prior art, important parts in the text that are beneficial to classification are effectively extracted, so that the model is more simplified, and the efficiency is higher.

Description

Technical Field

[0001] The present invention belongs to the field of text processing technology, and in particular relates to a speech step recognition method, system, device and medium based on pre-training and gated neural network. Background Art

[0002] Abstract step recognition in all paragraphs of academic papers uses concise and clear step words to summarize the abstract sentence, helping readers quickly locate the specific information of the paper. Step classification also has application extensions in artificial intelligence recommendation, realization of library and information science disciplines, information mining, knowledge discovery and knowledge graph construction. It can be seen that step recognition can be used as a basic task in text processing-related research and is an important research. However, in the existing step recognition algorithms, the intrinsic relationship and characteristics between words are not fully explored and utilized, resulting in the need to improve and enhance the performance of such algorithms.

[0003] One difficulty in speech step recognition is that natural language has problems with expression diversity and complexity. Chinese has diverse expressions, multiple meanings of a word, and long and complex sentences formed by nested sentence structures are difficult to split, which are difficult for machines to understand. Another difficulty in speech step recognition is that there is no complete mathematical model to accurately describe the content expressed in natural language, and machines face great challenges in understanding the semantics of natural language.

[0004] Abstract speech step recognition is mainly based on traditional machine learning and deep learning methods. In recent years, with the excellent performance of the BERT pre-training model in various natural language processing tasks, researchers have started to tune and transform the pre-training model. The methods based on traditional machine learning and deep learning for abstract speech step recognition are mostly based on structured summaries or English datasets; however, in rule-based methods, rule formulation is crucial for recognition effect, and rules cannot cover all speech step situations, so the recognition effect needs to be further improved; speech step recognition methods based on traditional machine learning and deep learning rely on the vocabulary, morphology and other features of the text, but machine learning cannot deeply learn its semantics, so the effect is not optimal. Summary of the Invention

[0005] In response to the problems existing in the prior art, the present invention provides a speech step recognition method, system, device and medium based on pre-training and gated neural networks, which can learn the deep semantics of the text based on the pre-training model and combine the gated network with an attention mechanism to perform focused feature recognition, thereby improving the accuracy of semantic recognition.

[0006] The present invention is achieved through the following technical solutions:

[0007] The speech step recognition method based on pre-trained and gated neural networks includes the following steps:

[0008] S1: Collect data from the target text segment, pre-process it, and label it according to the preset steps;

[0009] S2: Identify and split long, difficult and complex sentences in the target text;

[0010] S3: Build an automatic speech step recognition model based on ERNIE_AT-GRU;

[0011] S4: The split data is input into the ERNIE_AT-GRU model for training. After rounds of training, the test data is tested for speech step recognition to obtain the speech step recognition results.

[0012] Preferably, in step S1, the collected target text segment is standardized in text format, and format symbols such as "\n", "\t", and spaces in the text are cleaned to retain the complete text content of the original data.

[0013] Preferably, in step S2, the data of the target text segment is subjected to dependency syntax analysis using the LTP tool, and whether a complex relationship exists in the sentence is identified based on COO; long, difficult and complex sentences are identified and split based on the obtained parallel relationship markers, and single semantic data is obtained, which is divided into training data and test data in a ratio of 8:2.

[0014] Preferably, the LTP tool performs dependency parsing and identifies whether a complex relationship exists in a sentence based on COO, including the following steps:

[0015] A1: The LTP tool performs word segmentation, part-of-speech tagging, and dependency syntax analysis on the target text.

[0016] A2: Integrate the obtained data into a format that is easy to process, S = (segmentation, part-of-speech tag, (segmentation node, parent node, dependency tag));

[0017] A3: Traverse and integrate the data, obtain sentences whose parent node of the word segmentation is the root node and whose dependency is marked as COO, and store the semantically complex sentences that meet the conditions;

[0018] A4: Traverse the semantically complex sentences and split the sentences that meet the conditions at the commas before the parallel relations to obtain single semantic clauses.

[0019] Preferably, building a speech step automatic recognition model in step S3 includes the following steps:

[0020] B1: Build the ERNIE pre-training model and use its Transformer-XL feature processor to integrate the multi-head self-attention mechanism to learn text semantics and obtain the word vector feature matrix integrated with the multi-head self-attention mechanism;

[0021] B2: Build a gated network AT-GRU module with an attention mechanism. Input the word embedding matrix obtained from the pre-trained model into the bidirectional gated network to learn text context features. Connect the attention mechanism to focus on important information for text classification.

[0022] B3: Combine the ERNIE pre-trained model with the AT-GRU module to obtain the ERNIE_AT-GRU model.

[0023] Preferably, the step B1 of building the ERNIE pre-training model includes the following steps:

[0024] C1: Write a pre-training model call interface to load the pre-training parameters and other information required by the pre-training model; the ERNIE pre-training model uses three-segment single-word mask, phrase, and entity-level mask information integration to obtain the complete semantics of words, phrases, and entities;

[0025] C2: word vector X after three-segment masking: {w i1 ,w i2 ,...,w iN} Input Transformer-XL encoder, after word embedding Embedding process x it =W e w it ,t∈[1,N],W e is the weight parameter of the Embedding layer, which converts the high-dimensional sparse word vector matrix into a low-dimensional dense word vector matrix, that is, the word embedding vector for each sentence;

[0026] C3: Three weight matrices W calculated for a single self-attention q 、W k 、W v The matrices Q, K, and V obtained by multiplying the word embedding vector with the three respectively represent the correlation between the current word and other words in the sentence. In order to prevent the result from being too large, divide it by the root mean square of their dimensions d k represents the dimensions of a Q or K matrix vector, where It is a learnable variable used to calculate the relative distance within a range. It is then normalized by the Softmax function to obtain the normalized correlation between each word and other words. It is then multiplied by the V matrix, that is, weighted summation, to obtain the new vector encoding of each word. The formula is as follows:

[0027]

[0028] C4: Merge the Q, K, and V matrices calculated for each single self-attention according to the split head, and multiply the weight W by the dot 0, linearly convert the segmented head into a matrix of the original dimension to obtain the Multi-Head matrix. The process can be expressed as:

[0029] head i =Attention(Q i , K i , V i ), i = 1, ..., h;

[0030] MultiHead(Q,K,V)=Concact(head i ,...,head h )W 0 ;

[0031] C5: The Q, K, and V matrices calculated by Multi-Head are input into the Add&Norm layer for self-attention input and output addition and normalization. They are then processed by the fully connected layer's feedforward neural network and the Add&Norm layer to obtain a word vector feature matrix that integrates the multi-head attention mechanism. The matrix contains the text features learned by the model and the contextual semantic information contained in the text.

[0032] Preferably, the step B2 of building a gated network AT-GRU module with an attention mechanism includes the following steps:

[0033] D1: Write attention layer code based on a single self-attention mechanism formula

[0034]

[0035] D2: The bidirectional gating mechanism BIGRU includes a reset gate and an update gate. The reset gate is used to filter out some of the state information of the previous moment in the candidate state; the update gate determines the amount of information to be retained in the current state. As shown in the formula:

[0036] r t =δ(W r x t +U r h t-1 );

[0037] δ is the activation function, x t is the current input, h t-1 is the hidden layer output at the previous moment, that is, the historical state, W r and U r is the weight matrix;

[0038] D3: Bidirectional gating network has forward and backward training GRU to obtain text context feature information, where the forward operation is to transform the forward hidden state of the i-th sentence and the backward hidden state for:

[0039]

[0040]

[0041] The sentence encoding representation is obtained by the previous and next hidden states:

[0042]

[0043] Connecting the forward and backward training to update the current gate state is the new candidate value u created by the tanh layer. i Decision:

[0044] u i =tanh(W s h i +b s );

[0045] The attention mechanism calculates word weights and adds them to the gated network to form the AT-GRU module:

[0046]

[0047] The attention mechanism is used to obtain the hidden information of clauses that contribute to the semantic representation, and the clause information is aggregated to obtain the representation information of the entire sentence.

[0048] Preferably, in step B3, combining the ERNIE pre-trained model with the AT-GRU module to obtain the ERNIE_AT-GRU model includes the following steps:

[0049] E1: The output dimension of the three-dimensional word vector feature matrix obtained by the pre-training model and integrated with the multi-head attention mechanism is converted into a dimensional shape that can be input into the gating network, and the data is sent to the gating network with the attention mechanism;

[0050] E2: Add a Dropout layer after the gated network layer to randomly ignore a preset number of neurons to prevent the model from overfitting;

[0051] E3: Access the fully connected layer, use Softmax for speech step recognition, and output the classification label.

[0052] 7. The method for speech step recognition based on pre-training and gated neural network according to claim 1, wherein step S4 comprises the following steps:

[0053] F1: Input single semantic data into the model, call the pre-trained model interface to implement text segmentation, and vectorize the segmented words according to the pre-trained model dictionary mapping;

[0054] F2: Mask the text by 0 / 1 using the pad_size length of each sentence in the training and test data according to the batch value. Combine the text word vector content, label numbers, and mask results in 4a and store them as a .pkl file for easy program reading. When loading, use the tabular data structure to load them into a Dataframe tabular data structure.

[0055] F3: The vectorized Dataframe structure data is fed into the ERNIE_AT-GRU model for training in batches. After forward operation and backpropagation, the label of the test data is predicted in each round. The cross entropy is used as the loss function in the backpropagation to optimize the model:

[0056]

[0057] Where D is the size of training data, C is the number of categories, is the text data label, To predict the probability of the model, multiple parameter adjustments are used to determine the label of the optimal prediction result, obtain the speech step recognition effect and classified test label data, that is, the speech step classification result and the loss change analysis model during the model operation.

[0058] A speech step recognition system based on pre-trained and gated neural networks, including:

[0059] The acquisition module is used to collect data from the target text segment for preprocessing and labeling according to the preset steps;

[0060] The processing module is used to identify and split long, difficult and complex sentences in the target text;

[0061] Model building module, used to build an automatic speech recognition model based on ERNIE_AT-GRU;

[0062] The output module is used to input the split data into the ERNIE_AT-GRU model training. After rounds of training, the test data is tested for speech step recognition to obtain the speech step recognition results.

[0063] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the speech step recognition method based on pre-training and gated neural networks are implemented.

[0064] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the speech step recognition method based on pre-training and gated neural networks.

[0065] Compared with the prior art, the present invention has the following beneficial technical effects:

[0066] The present invention provides a speech step recognition method, system, device and medium based on pre-training and gated neural network. The method collects data from a target text segment for pre-processing and labels it according to preset speech steps; identifies and splits long, difficult and complex sentences in the target text segment; builds an ERNIE_AT-GRU speech step automatic recognition model; inputs the split data into the ERNIE_AT-GRU model for training, and performs speech step recognition tests on the test data after rounds of training to obtain speech step recognition results; in specific operation, the speech step recognition method based on the pre-trained model and gated neural network uses the ERNIE pre-trained model that combines large-scale text content with knowledge graphs to learn the deep semantics of the text, improving the disadvantage of traditional machine learning that does not fully explore and utilize the intrinsic relationships and features between words. The downstream gated network AT-GRU model with an attention mechanism performs focused feature learning, focusing on word vectors that are more conducive to text classification, and compensating for the poor classification effect caused by the forgetting problem of long text input in machine learning. Compared with the existing technology, the present invention effectively extracts important parts of the text that are conducive to classification, so the model is more streamlined and more efficient. In addition, it should be noted that the present invention generates a word vector matrix through the dictionary mapping of the pre-trained model, uses a machine-readable digital matrix to represent the text, and undergoes deep semantic learning of the pre-trained model. Therefore, better results can be achieved without a large number of test data samples. The pre-trained model has learned large-scale text data and knowledge graph knowledge, has good transferability, and combines its own training data for semantic learning training, so it is more robust to test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of the implementation of the speech step recognition method based on pre-training and gated neural network of the present invention;

[0068] Figure 2 This is the model diagram for automatic recognition of Chinese unstructured abstract steps in the present invention;

[0069] Figure 3 Schematic diagram of the ERNIE pre-training model of the present invention;

[0070] Figure 4 This is an example diagram of the MASK method of the ERNIE pre-training model of the present invention;

[0071] Figure 5 This is the computational flow of the attention neural network layer of the present invention;

[0072] Figure 6 This is the structural diagram of the BIGRU neural network with attention mechanism of the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further described in detail below with reference to specific embodiments, which are intended to explain the present invention rather than to limit it.

[0074] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0075] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0076] The present invention provides a speech step recognition method based on pre-training and gated neural network, such as Figure 1 As shown, the following steps are included:

[0077] S1: Collect data from the target text segment, pre-process it, and label it according to the preset steps;

[0078] S2: Identify and split long, difficult and complex sentences in the target text;

[0079] S3: Build an automatic speech step recognition model based on ERNIE_AT-GRU;

[0080] S4: The split data is input into the ERNIE_AT-GRU model for training. After rounds of training, the test data is tested for speech step recognition to obtain the speech step recognition results.

[0081] Preferably, in step S1, the collected target text segment is standardized in text format, and format symbols such as "\n", "\t", and spaces in the text are cleaned, retaining the complete text content of the original data; further, in step S2, the data of the target text segment is subjected to dependency syntax analysis using an LTP tool, and whether a complex relationship exists in the sentence is identified based on COO; long, difficult, and complex sentences are identified and split based on the obtained parallel relationship markers to obtain single semantic data, and divided into training data and test data in an 8:2 ratio.

[0082] Preferably, the LTP tool performs dependency parsing and identifies whether a complex relationship exists in a sentence based on COO, including the following steps:

[0083] A1: The LTP tool performs word segmentation, part-of-speech tagging, and dependency syntax analysis on the target text.

[0084] A2: Integrate the obtained data into a format that is easy to process, S = (segmentation, part-of-speech tag, (segmentation node, parent node, dependency tag));

[0085] A3: Traverse and integrate the data, obtain sentences whose parent node of the word segmentation is the root node and whose dependency is marked as COO, and store the semantically complex sentences that meet the conditions;

[0086] A4: Traverse the semantically complex sentences and split the sentences that meet the conditions at the commas before the parallel relations to obtain single semantic clauses.

[0087] Preferably, building a speech step automatic recognition model in step S3 includes the following steps:

[0088] B1: Build the ERNIE pre-training model, such as Figure 3 As shown, the Transformer-XL feature processor is used to integrate the multi-head self-attention mechanism to learn text semantics and obtain the word vector feature matrix integrated with the multi-head attention mechanism;

[0089] B2: Build a gated network AT-GRU module with an attention mechanism. Input the word embedding matrix obtained from the pre-trained model into the bidirectional gated network to learn text context features. Connect the attention mechanism to focus on important information for text classification.

[0090] B3: Combine the ERNIE pre-trained model with the AT-GRU module to obtain the ERNIE_AT-GRU model, as shown in Figure 2 shown.

[0091] Furthermore, the step B1 of building the ERNIE pre-training model includes the following steps:

[0092] C1: Write a pre-training model call interface to load the pre-training parameters and other information required by the pre-training model; the ERNIE pre-training model uses three-segment single-word mask, phrase, and entity-level mask information integration to obtain the complete semantics of words, phrases, and entities;

[0093] C2: word vector X after three-segment masking: {w i1 ,w i2 ,...,w iN} Input Transformer-XL encoder, after word embedding Embedding process x it =W e w it ,t∈[1,N],W e is the weight parameter of the Embedding layer, which converts the high-dimensional sparse word vector matrix into a low-dimensional dense word vector matrix, that is, the word embedding vector for each sentence, such as Figure 4 As shown in the figure, for the input text sentence, the words, Chinese characters and other information in the sentence are first regarded as basic language units and randomly masked by 15% to obtain the basic level mask MASK to obtain the basic level information of the sentence. Then, the second stage entity level MASK is performed to mask entities such as proper nouns in the sentence and predict vacancies in the entities. Finally, phrase MASK is performed to mask the phrases in the sentence and predict all basic units in the same phrase. The phrase information is encoded into the word embedding. The text information of different semantic units is obtained through three-stage masking, and a rich expression of the sentence semantic information is obtained.

[0094] C3: Three weight matrices W calculated for a single self-attention q 、W k 、W v The matrices Q, K, and V obtained by multiplying the word embedding vector with the three respectively represent the correlation between the current word and other words in the sentence. In order to prevent the result from being too large, divide it by the root mean square of their dimensions d k represents the dimensions of a Q or K matrix vector, where It is a learnable variable used to calculate the relative distance within a range. It is then normalized by the Softmax function to obtain the normalized correlation between each word and other words. It is then multiplied by the V matrix, that is, weighted summation, to obtain the new vector encoding of each word. The formula is as follows:

[0095]

[0096] C4: Merge the Q, K, and V matrices calculated for each single self-attention according to the split head, and multiply the weight W by the dot 0, linearly convert the segmented head into a matrix of the original dimension to obtain the Multi-Head matrix. The process can be expressed as:

[0097] head i =Attention(Q i , K i , V i ), i = 1, ..., h;

[0098] MultiHead(Q,K,V)=Concact(head i ,...,head h )W 0 ;

[0099] C5: The Q, K, and V matrices calculated by Multi-Head are input into the Add&Norm layer for self-attention input and output addition and normalization. They are then processed by the fully connected layer's feedforward neural network and the Add&Norm layer to obtain a word vector feature matrix that integrates the multi-head attention mechanism. The matrix contains the text features learned by the model and the contextual semantic information contained in the text.

[0100] Preferably, the step B2 of building a gated network AT-GRU module with an attention mechanism includes the following steps:

[0101] D1: Write attention layer code based on a single self-attention mechanism formula

[0102]

[0103] D2: The bidirectional gating mechanism BIGRU includes a reset gate and an update gate. The reset gate is used to filter out some of the state information of the previous moment in the candidate state; the update gate determines the amount of information to be retained in the current state. As shown in the formula:

[0104] r t =δ(W r x t +U r h t-1 );

[0105] δ is the activation function, x t is the current input, h t-1 is the hidden layer output at the previous moment, that is, the historical state, W r and U r is the weight matrix;

[0106] D3: Bidirectional gating network has forward and backward training GRU to obtain text context feature information, where the forward operation is to transform the forward hidden state of the i-th sentence and the backward hidden state for:

[0107]

[0108]

[0109] The sentence encoding representation is obtained by the previous and next hidden states:

[0110]

[0111] Connecting the forward and backward training to update the current gate state is the new candidate value u created by the tanh layer. i Decision:

[0112] u i =tanh(W s h i +b s );

[0113] The attention mechanism calculates word weights and adds them to the gated network to form the AT-GRU module:

[0114]

[0115] The attention mechanism is used to obtain the hidden information of the clauses that contributes to the semantic representation, and the clause information is aggregated to obtain the representation information of the entire sentence;

[0116] The attention mechanism is added to the gated network to calculate word weights. Figure 5 As shown: Composed of AT-GRU modules, such as Figure 6 As shown: The attention mechanism is used to obtain the hidden information of clauses that contribute to the semantic representation, and the clause information is aggregated to obtain the representation information of the entire sentence.

[0117] Preferably, in step B3, combining the ERNIE pre-trained model with the AT-GRU module to obtain the ERNIE_AT-GRU model includes the following steps:

[0118] E1: The output dimension of the three-dimensional word vector feature matrix obtained by the pre-training model and integrated with the multi-head attention mechanism is converted into a dimensional shape that can be input into the gating network, and the data is sent to the gating network with the attention mechanism;

[0119] E2: Add a Dropout layer after the gated network layer to randomly ignore a preset number of neurons to prevent the model from overfitting;

[0120] E3: Access the fully connected layer, use Softmax for speech step recognition, and output the classification label.

[0121] Preferably, step S4 includes the following steps:

[0122] F1: Input single semantic data into the model, call the pre-trained model interface to implement text segmentation, and vectorize the segmented words according to the pre-trained model dictionary mapping;

[0123] F2: Mask the text by 0 / 1 using the pad_size length of each sentence in the training and test data according to the batch value. Combine the text word vector content, label numbers, and mask results in 4a and store them as a .pkl file for easy program reading. When loading, use the tabular data structure to load them into a Dataframe tabular data structure.

[0124] F3: The vectorized Dataframe structure data is fed into the ERNIE_AT-GRU model for training in batches. After forward operation and backpropagation, the label of the test data is predicted in each round. The cross entropy is used as the loss function in the backpropagation to optimize the model:

[0125]

[0126] Where D is the size of training data, C is the number of categories, is the text data label, To predict the probability of the model, multiple parameter adjustments are used to determine the label of the optimal prediction result, obtain the speech step recognition effect and classified test label data, that is, the speech step classification result and the loss change analysis model during the model operation.

[0127] The present invention provides a speech step recognition system based on pre-trained and gated neural networks, comprising:

[0128] The acquisition module is used to collect data from the target text segment for preprocessing and labeling according to the preset steps;

[0129] The processing module is used to identify and split long, difficult and complex sentences in the target text;

[0130] Model building module, used to build an automatic speech recognition model based on ERNIE_AT-GRU;

[0131] The output module is used to input the split data into the ERNIE_AT-GRU model training. After rounds of training, the test data is tested for speech step recognition to obtain the speech step recognition results.

[0132] In another embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the speech step recognition method based on pre-training and gated neural networks.

[0133] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the speech step recognition method based on pre-training and gated neural networks in the above embodiment.

[0134] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech step recognition method based on pre-training and gated neural network, characterized in that: The following steps are involved: S1: Collect data from the target text segment, pre-process it, and label it according to the preset steps; S2: Identify and split long, difficult and complex sentences in the target text; In step S2, the target text segment data is subjected to dependency syntax analysis using the LTP tool, and whether a complex relationship exists in the sentence is identified based on the COO; long, difficult, and complex sentences are identified and split based on the obtained parallel relationship markers to obtain single semantic data, which is then divided into training data and test data in an 8:2 ratio; The LTP tool performs dependency parsing and identifies whether a sentence contains complex relations based on COO, including the following steps: A1: The LTP tool performs word segmentation, part-of-speech tagging, and dependency syntax analysis on the target text. A2: Integrate the obtained data into a format that is easy to process, S = (segmentation, part-of-speech tag, (segmentation node, parent node, dependency tag)); A3: Traverse and integrate the data, obtain sentences whose parent node of the word segmentation is the root node and whose dependency is marked as COO, and store the semantically complex sentences that meet the conditions; A4: Traverse the semantically complex sentences and split the sentences that meet the conditions at the commas before the parallel relations to obtain single semantic clauses; The step S3 in which the speech step automatic recognition model is constructed includes the following steps: B1: Build the ERNIE pre-training model and use its Transformer-XL feature processor to integrate the multi-head self-attention mechanism to learn text semantics and obtain the word vector feature matrix integrated with the multi-head self-attention mechanism; B2: Build a gated network AT-GRU module with an attention mechanism. Input the word embedding matrix obtained from the pre-trained model into the bidirectional gated network to learn text context features. Connect the attention mechanism to focus on important information for text classification. B3: Combine the ERNIE pre-trained model with the AT-GRU module to obtain the ERNIE_AT-GRU model; Step B1 of building the ERNIE pre-training model includes the following steps: C1: Write a pre-training model call interface to load the pre-training parameter information required by the pre-training model; the ERNIE pre-training model uses three-segment single-word mask, phrase and entity level mask information integration to obtain the complete semantics of words, phrases and entities; C2: word vector X after three-segment masking: {w i1 ,w i2 ,...,w iN } Input Transformer-XL encoder, after word embedding Embedding process x it =W e w it ,t∈[1,N],W e is the weight parameter of the Embedding layer, which converts the high-dimensional sparse word vector matrix into a low-dimensional dense word vector matrix, that is, the word embedding vector for each sentence; C3: Three weight matrices W calculated for a single self-attention q 、W k 、W v The matrices Q, K, and V obtained by multiplying the word embedding vector with the three respectively represent the correlation between the current word and other words in the sentence. In order to prevent the result from being too large, divide it by the root mean square of their dimensions d k represents the dimensions of a Q or K matrix vector, where It is a learnable variable used to calculate the relative distance within a range. It is then normalized by the Softmax function to obtain the normalized correlation between each word and other words. It is then multiplied by the V matrix, that is, weighted summation, to obtain the new vector encoding of each word. The formula is as follows: C4: Merge the Q, K, and V matrices calculated for each single self-attention according to the split head, and multiply the weight W by the dot 0 , linearly convert the segmented head into a matrix of the original dimension to obtain the Multi-Head matrix. The process can be expressed as: head i =Attention(Q i ,K i ,V i ),i=1,...,h; MultiHead(Q,K,V)=Concact(head i ,...,head h )W 0 ; C5: The Q, K, and V matrices calculated by Multi-Head are input into the Add&Norm layer for self-attention input and output addition and normalization. They are then processed by the fully connected layer's feedforward neural network and the Add&Norm layer to obtain a word vector feature matrix that integrates the multi-head attention mechanism. The matrix contains the text features learned by the model and the contextual semantic information contained in the text. S3: Build an automatic speech step recognition model based on ERNIE_AT-GRU; In step B3, the ERNIE pre-trained model is combined with the AT-GRU module to obtain the ERNIE_AT-GRU model, which includes the following steps: E1: The output dimension of the three-dimensional word vector feature matrix obtained by the pre-training model and integrated with the multi-head attention mechanism is converted into a dimensional shape that can be input into the gating network, and the data is sent to the gating network with the attention mechanism; E2: Add a Dropout layer after the gated network layer to randomly ignore a preset number of neurons to prevent the model from overfitting; E3: Accesses the fully connected layer, uses Softmax to perform speech step recognition, and outputs classification labels; S4: The split data is input into the ERNIE_AT-GRU model for training. After rounds of training, the test data is tested for speech step recognition to obtain the speech step recognition results.

2. The speech step recognition method based on pre-training and gated neural network according to claim 1 is characterized in that: In step S1, the collected target text segment is standardized in text format, and the "\n", "\t", and space format symbols in the text are cleaned, so as to retain the complete text content of the original data.

3. The speech step recognition method based on pre-training and gated neural network according to claim 1 is characterized in that: The step B2 of building the AT-GRU module of the gated network with the attention mechanism includes the following steps: D1: Write attention layer code based on a single self-attention mechanism formula D2: The bidirectional gating mechanism BIGRU includes a reset gate and an update gate. The reset gate is used to filter out some of the state information of the previous moment in the candidate state; the update gate determines the amount of information to be retained in the current state. As shown in the formula: r t =δ(W r x t +U r h t-1 ); δ is the activation function, x t is the current input, h t-1 is the hidden layer output at the previous moment, that is, the historical state, W r and U r is the weight matrix; D3: Bidirectional gating network has forward and backward training GRU to obtain text context feature information, where the forward operation is to transform the forward hidden state of the i-th sentence and the backward hidden state for: The sentence encoding representation is obtained by the previous and next hidden states: Connecting the forward and backward training to update the current gate state is the new candidate value u created by the tanh layer. i Decision: u i =tanh(W s h i +b s ); The attention mechanism calculates word weights and adds them to the gated network to form the AT-GRU module: The attention mechanism is used to obtain the hidden information of clauses that contribute to the semantic representation, and the clause information is aggregated to obtain the representation information of the entire sentence.

4. The speech step recognition method based on pre-training and gated neural network according to claim 1 is characterized in that: The step S4 comprises the following steps: F1: Input single semantic data into the model, call the pre-trained model interface to implement text segmentation, and vectorize the segmented words according to the pre-trained model dictionary mapping; F2: Mask the text by 0 / 1 using the pad_size length of each sentence in the training and test data according to the batch value. Combine the text word vector content, label numbers, and mask results in 4a and store them as a .pkl file for easy program reading. When loading, use the tabular data structure to load them into a Dataframe tabular data structure. F3: The vectorized Dataframe structure data is fed into the ERNIE_AT-GRU model for training in batches. After forward operation and backpropagation, the label of the test data is predicted in each round. The cross entropy is used as the loss function in the backpropagation to optimize the model: Where D is the size of training data, C is the number of categories, is the text data label, To predict the probability of the model, multiple parameter adjustments are used to determine the label of the optimal prediction result, obtain the speech step recognition effect and classified test label data, that is, the speech step classification result and the loss change analysis model during the model operation.

5. A speech step recognition system based on pre-trained and gated neural networks, characterized by: The speech step recognition method based on a pre-trained and gated neural network according to any one of claims 1 to 4 comprises: The acquisition module is used to collect data from the target text segment for preprocessing and labeling according to the preset steps; The processing module is used to identify and split long, difficult and complex sentences in the target text; In step S2, the target text segment data is subjected to dependency syntax analysis using the LTP tool, and whether a complex relationship exists in the sentence is identified based on the COO; long, difficult, and complex sentences are identified and split based on the obtained parallel relationship markers to obtain single semantic data, which is then divided into training data and test data in an 8:2 ratio; The LTP tool performs dependency parsing and identifies whether a sentence contains complex relations based on COO, including the following steps: A1: The LTP tool performs word segmentation, part-of-speech tagging, and dependency syntax analysis on the target text. A2: Integrate the obtained data into a format that is easy to process, S = (segmentation, part-of-speech tag, (segmentation node, parent node, dependency tag)); A3: Traverse and integrate the data, obtain sentences whose parent node of the word segmentation is the root node and whose dependency is marked as COO, and store the semantically complex sentences that meet the conditions; A4: Traverse the semantically complex sentences and split the sentences that meet the conditions at the commas before the parallel relations to obtain single semantic clauses; The step S3 in which the speech step automatic recognition model is constructed includes the following steps: B1: Build the ERNIE pre-training model and use its Transformer-XL feature processor to integrate the multi-head self-attention mechanism to learn text semantics and obtain the word vector feature matrix integrated with the multi-head self-attention mechanism; B2: Build a gated network AT-GRU module with an attention mechanism. Input the word embedding matrix obtained from the pre-trained model into the bidirectional gated network to learn text context features. Connect the attention mechanism to focus on important information for text classification. B3: Combine the ERNIE pre-trained model with the AT-GRU module to obtain the ERNIE_AT-GRU model; Step B1 of building the ERNIE pre-training model includes the following steps: C1: Write a pre-training model call interface to load the pre-training parameter information required by the pre-training model; the ERNIE pre-training model uses three-segment single-word mask, phrase and entity level mask information integration to obtain the complete semantics of words, phrases and entities; C2: word vector X after three-segment masking: {w i1 ,w i2 ,...,w iN } Input Transformer-XL encoder, after word embedding Embedding process x it =W e w it ,t∈[1,N],W e is the weight parameter of the Embedding layer, which converts the high-dimensional sparse word vector matrix into a low-dimensional dense word vector matrix, that is, the word embedding vector for each sentence; C3: Three weight matrices W calculated for a single self-attention q 、W k 、W v The matrices Q, K, and V obtained by multiplying the word embedding vector with the three respectively represent the correlation between the current word and other words in the sentence. In order to prevent the result from being too large, divide it by the root mean square of their dimensions d k represents the dimensions of a Q or K matrix vector, where It is a learnable variable used to calculate the relative distance within a range. It is then normalized by the Softmax function to obtain the normalized correlation between each word and other words. It is then multiplied by the V matrix, that is, weighted summation, to obtain the new vector encoding of each word. The formula is as follows: C4: Merge the Q, K, and V matrices calculated for each single self-attention according to the split head, and multiply the weight W by the dot 0 , linearly convert the segmented head into a matrix of the original dimension to obtain the Multi-Head matrix. The process can be expressed as: head i =Attention(Q i ,K i ,V i ),i=1,...,h; MultiHead(Q,K,V)=Concact(head i ,...,head h )W 0 ; C5: The Q, K, and V matrices calculated by Multi-Head are input into the Add&Norm layer for self-attention input and output addition and normalization. They are then processed by the fully connected layer's feedforward neural network and the Add&Norm layer to obtain a word vector feature matrix that integrates the multi-head attention mechanism. The matrix contains the text features learned by the model and the contextual semantic information contained in the text. Model building module, used to build an automatic speech recognition model based on ERNIE_AT-GRU; Combining the ERNIE pre-trained model with the AT-GRU module to obtain the ERNIE_AT-GRU model includes the following steps: E1: The output dimension of the three-dimensional word vector feature matrix obtained by the pre-training model and integrated with the multi-head attention mechanism is converted into a dimensional shape that can be input into the gating network, and the data is sent to the gating network with the attention mechanism; E2: Add a Dropout layer after the gated network layer to randomly ignore a preset number of neurons to prevent the model from overfitting; E3: Accesses the fully connected layer, uses Softmax to perform speech step recognition, and outputs classification labels; The output module is used to input the split data into the ERNIE_AT-GRU model training. After rounds of training, the test data is tested for speech step recognition to obtain the speech step recognition results.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the speech step recognition method based on pre-training and gated neural network as described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the speech step recognition method based on pre-training and gated neural network as described in any one of claims 1 to 4 are implemented.

Images