Spoken language understanding method and device, electronic equipment and storage medium

Abstract

The invention provides a spoken language understanding method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting a voice feature map into a feature extraction module in a spoken language understanding model, and obtaining a shared representation matrix; inputting the shared representation matrix into a modal conversion module in a spoken language understanding model to obtain initial text representation; inputting the initial text representation into a text enhancement module in a spoken language understanding model to obtain an enhanced text representation; and inputting the enhanced text representation into a structured information generation module in the spoken language understanding model to obtain structured information corresponding to the speech to be recognized. Through the method, semantic logic coherence and text element relevance can be effectively enhanced, the semantic consistency learning ability of spoken language understanding is remarkably improved, the accuracy of structured information output and the model generalization ability are greatly improved, and the semantic understanding and intelligent response level of a machine to natural spoken languages is enhanced.

Description

Technical Field

[0001] This invention relates to the field of speech recognition, and more specifically, to a spoken language understanding method, apparatus, electronic device, and storage medium. Background Technology

[0002] The core objective of Spoken Language Understanding (SLU) is to accurately extract specific structured information from natural human spoken speech, thereby improving the machine's semantic understanding and intelligent response capabilities in human language. Common core tasks in SLU include named entity recognition, intent recognition, and sentiment analysis. In practical applications, end-to-end models have become the mainstream technical solution for performing SLU tasks because they do not require splitting the task process and can directly map from speech input to target output.

[0003] However, when end-to-end models face situations where labeled data is sparse (e.g., a lack of spoken language corpus in a specific domain, or limited coverage of labeled samples) or labeled structures are complex (e.g., multiple levels of entity nesting, or cumbersome rules for associating intent with entities), the models struggle to fully learn consistent features at the language level (including semantic logic consistency, entity representation consistency, and consistency between intent and context matching), leading to decreased accuracy and insufficient generalization ability in the model output. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a spoken language understanding method, device, electronic device and storage medium that can effectively enhance semantic logical coherence and text element correlation, significantly improve the semantic consistency learning ability of spoken language understanding, greatly improve the accuracy of structured information output and model generalization ability, and enhance the machine's semantic understanding and intelligent response level to natural spoken language.

[0005] In a first aspect, embodiments of this application provide a spoken language comprehension method, the method comprising: Extract the speech feature map corresponding to the speech to be recognized; The speech feature map is input into the feature extraction module of the spoken language understanding model to obtain a shared representation matrix; the shared representation matrix integrates the acoustic information and text semantics of the speech to be recognized. The shared representation matrix is ​​input into the modality transformation module in the spoken language understanding model to obtain the initial text representation; The initial text representation is input into the text enhancement module in the spoken language understanding model to enhance the semantic logical coherence and the relationship between text elements in the initial text representation, thereby obtaining the enhanced text representation. The enhanced text representation is input into the structured information generation module in the spoken language understanding model to obtain the structured information corresponding to the speech to be recognized.

[0006] In one possible implementation, the step of inputting the speech feature map into the feature extraction module of the spoken language understanding model to obtain a shared representation matrix includes: The speech feature map is input into the feature encoder in the feature extraction module to obtain a context representation; the context representation is a feature matrix that fuses the local acoustic features and global text semantics of the speech to be recognized. The context representation is input into the predictor in the feature extraction module to obtain a text segment representation sequence and the number of text segment representations in the text segment representation sequence. The context representation, the text segment representation sequence, and the number of text segment representations are input into the feature decoder in the feature extraction module to obtain the shared representation matrix.

[0007] In one possible implementation, the step of inputting the shared representation matrix into the modality transformation module of the spoken language understanding model to perform modality transformation and semantic mapping on the shared representation matrix to obtain an initial text representation includes: The shared representation matrix is ​​input into the lexical probability quantization unit in the modality conversion module to obtain the lexical sequence probability distribution; the lexical sequence probability distribution is used to represent the matching probability of the speech segment corresponding to each text segment in the text sequence corresponding to the speech to be recognized with each preset lexical in the preset lexical set. The probability distribution of the word sequence is input into the text feature mapping unit in the modality conversion module to obtain the initial text representation.

[0008] In one possible implementation, inputting the enhanced text representation into the structured information generation module of the spoken language understanding model to obtain the structured information corresponding to the speech to be recognized includes: The enhanced text representation is input into the named entity recognition unit, intent recognition unit, and / or sentiment analysis unit in the structured information generation module to obtain the named entity information, intent information, and / or sentiment information in the structured information corresponding to the speech to be recognized.

[0009] In one possible implementation, the enhanced text representation is input into the named entity recognition unit according to the following steps to obtain named entity information in the structured information corresponding to the speech to be recognized: The enhanced text representation is input into the named entity tagging layer in the named entity recognition unit to obtain a named entity start tag vector and a named entity end tag vector. The named entity start tag vector contains tagging information indicating whether each enhanced lexical feature vector in the enhanced text representation is the start position of a named entity. The named entity end tag vector contains tagging information indicating whether each enhanced lexical feature vector is the end position of a named entity and the named entity type corresponding to the enhanced lexical feature vector of the named entity. The enhanced text representation, the named entity start marker vector, and the named entity end marker vector are input into the named entity generation layer in the named entity recognition unit to generate named entity information in the structured information corresponding to the speech to be recognized.

[0010] In one possible implementation, the enhanced text representation is input into the named entity tagging layer of the named entity recognition unit according to the following steps to obtain a named entity start tagging vector and a named entity end tagging vector: The enhanced text representation is input into the feature splitting block of the named entity tagging layer to obtain the first text representation and the second text representation. The first text representation is input into the first named entity feature extraction block of the named entity tagging layer to obtain the first named entity feature matrix; and the second text representation is input into the second named entity feature extraction block of the named entity tagging layer to obtain the second named entity feature matrix; the second named entity feature matrix is ​​used to provide the basis for determining the start position of a named entity; the second named entity feature matrix is ​​used to provide the joint basis for determining the end position of a named entity and the type of a named entity. The first named entity feature matrix is ​​input into the named entity start tag block of the named entity tag layer to obtain the named entity start tag vector; the second named entity feature matrix is ​​input into the named entity end tag block of the named entity tag layer to obtain the named entity end tag vector.

[0011] In one possible implementation, the enhanced text representation is input into the intent recognition unit or sentiment analysis unit in the structured information generation module according to the following steps to obtain the intent information or sentiment information in the structured information corresponding to the speech to be recognized: The enhanced text representation is input into the first classification feature extraction block in the intent recognition unit or sentiment analysis unit to obtain a word classification feature matrix; the word classification feature includes the classification feature vector of each text segment in the text segment sequence corresponding to the speech to be recognized; The word classification feature matrix is ​​input into the second classification feature extraction block in the intent recognition unit or sentiment analysis unit to obtain a word weight sequence; the word weight sequence contains the importance value of each text segment in the text segment sequence corresponding to the speech to be recognized. The word weight sequence and the word classification feature matrix are input into the classification block of the intent recognition unit or sentiment analysis unit to obtain the intent information or sentiment information in the structured information corresponding to the speech to be recognized.

[0012] Secondly, embodiments of this application also provide a spoken language understanding device, the device comprising: The extraction module is used to extract the speech feature map corresponding to the speech to be recognized; The input module is used to input the speech feature map into the feature extraction module in the spoken language understanding model to obtain a shared representation matrix; the shared representation matrix integrates the acoustic information and text semantics of the speech to be recognized. The input module is also used to input the shared representation matrix into the modality conversion module in the spoken language understanding model to obtain the initial text representation; The input module is also used to input the initial text representation into the text enhancement module in the spoken language understanding model to enhance the semantic logical coherence and the correlation between text elements in the initial text representation, so as to obtain an enhanced text representation. The input module is further configured to input the enhanced text representation into the structured information generation module in the spoken language understanding model to obtain the structured information corresponding to the speech to be recognized.

[0013] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the spoken language understanding method as described in any of the first aspects.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the spoken language understanding method as described in any of the first aspects.

[0015] This application provides a spoken language understanding method, apparatus, electronic device, and storage medium. The method includes: extracting a speech feature map corresponding to the speech to be recognized; inputting the speech feature map into a feature extraction module in a spoken language understanding model to obtain a shared representation matrix; the shared representation matrix fusing acoustic information and text semantics of the speech to be recognized; inputting the shared representation matrix into a modality conversion module in the spoken language understanding model to obtain an initial text representation; inputting the initial text representation into a text enhancement module in the spoken language understanding model to enhance the semantic logical coherence and the correlation between text elements in the initial text representation to obtain an enhanced text representation; and inputting the enhanced text representation into a structured information generation module in the spoken language understanding model to obtain structured information corresponding to the speech to be recognized. The method provided by this application has the following beneficial effects: (1) By extracting speech feature maps and generating a shared representation matrix that integrates acoustic and semantic information, the semantic logical coherence and text element correlation are effectively enhanced through progressive processing of modality conversion, text enhancement and structured generation modules.

[0016] (2) Addressing the performance bottleneck of end-to-end models in scenarios with sparse labeled data and complex structures, it significantly improves the semantic consistency learning ability of spoken language comprehension, greatly enhances the accuracy of structured information output and model generalization ability, and strengthens the machine's semantic understanding and intelligent response level to natural spoken language. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a spoken language comprehension method provided in an embodiment of this application is shown; Figure 2 A flowchart illustrating the naming entity information recognition process provided in an embodiment of this application is shown. Figure 3 This document illustrates a flowchart of the generation process for intent information or emotional information provided in an embodiment of this application. Figure 4 This illustration shows a structural schematic diagram of a spoken language understanding device provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0020] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] To enable those skilled in the art to utilize the content of this application, and in conjunction with the specific application scenario of "speech recognition," the following implementation methods are provided. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is primarily described in the context of "speech recognition," it should be understood that this is merely an exemplary embodiment.

[0022] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0023] The following is a detailed description of a spoken language comprehension method provided by an embodiment of this application.

[0024] Reference Figure 1 The diagram shown is a flowchart illustrating a spoken language comprehension method provided in an embodiment of this application. The exemplary steps of this embodiment are described below: S101. Extract the speech feature map corresponding to the speech to be recognized.

[0025] In the embodiments of this application, the speech to be recognized is speech data for a spoken language comprehension task, uniformly using single-channel 16kHz sampling rate speech data (conforming to the standard input format of general speech processing). The speech feature map extraction process is as follows: first, the original speech data undergoes preprocessing operations such as pre-emphasis, framing, and windowing; then, the preprocessed speech signal is converted into a Mel-spectrogram using a Mel-bank filter. This feature map can be directly used as the input feature of the subsequent speech coding module, i.e., the speech feature map. The speech feature map contains the speech feature vector of the speech frame to be recognized at each time frame.

[0026] S102. Input the speech feature map into the feature extraction module in the spoken language understanding model to obtain a shared representation matrix; the shared representation matrix integrates the acoustic information and text semantics of the speech to be recognized.

[0027] In this embodiment, the shared representation matrix is ​​a cross-modal fusion feature generated by the feature extraction module after encoding the speech feature map. The shared representation matrix includes the shared representation vector of each text segment in the text segment sequence corresponding to the speech to be recognized. The feature dimension of the shared representation matrix is... ,in, Let D be the number of text segments in the sequence of text segments corresponding to the speech to be recognized, and let D be the dimension of the shared representation vector corresponding to each text segment. The acoustic information includes the pitch features, spectral energy features, and speech duration features of the speech to be recognized, covering the physical acoustic properties of the speech; the text semantics includes the contextual semantics corresponding to the speech, which is text-level semantic information obtained based on the speech acoustic feature mapping. The specific encoding process is as follows: Step 1: Input the speech feature map into the feature encoder in the feature extraction module to obtain the context representation; the context representation is a feature matrix that integrates the local acoustic features and global text semantics of the speech to be recognized.

[0028] In this embodiment of the application, the context is represented as a set of temporal feature vectors, the dimension of which is typically [ [×K], where K is the dimension of the context vector corresponding to each time frame in the context representation. Each context vector encompasses both the acoustic details of the corresponding time frame (e.g., phonemes, tonal features) and integrates long-distance semantic associations (e.g., logical connections between preceding and following words). This provides data support for the predictor to achieve "speech-text soft alignment". Therefore, the context representation contains the context vectors corresponding to the speech frames of the speech to be recognized at each time frame. The context vector corresponding to each time frame fuses the local acoustic features and semantic association information of the corresponding speech frame.

[0029] Here, the feature encoder employs a hybrid architecture of "multi-layer self-attention mechanism and convolutional layers". The self-attention mechanism can capture the long-distance dependencies of the speech to be recognized (such as the semantic association between words in a sentence), i.e., global text semantics. The convolutional layers can extract local acoustic features of the speech (such as phoneme and tone details). The combination of the two allows the model to understand the overall semantics while accurately capturing acoustic details.

[0030] The semantic connection between preceding and following words in a sentence refers to the logical link, collocation, or content relevance between words in the preceding and following sentences. For example, in the sentence "The weather is good today": "today" is a time word, and it is usually followed by content describing the state of the day (such as weather or events), which is a logical collocation connection; the word "weather" is likely to be followed by words describing weather conditions such as "good / bad / sunny / rainy," rather than irrelevant words like "eating / running," which is a semantic category connection; conversely, if "good" is preceded by "weather," it clearly points to "good weather conditions"; if it is preceded by "mood," it points to "good emotional state," which is a context-dependent connection.

[0031] Step 2: Input the context representation into the predictor in the feature extraction module to obtain the text segment representation sequence and the number of text segment representations in the text segment representation sequence.

[0032] In this embodiment, the text segment representation sequence includes at least one text segment representation arranged in the order of the speech frames corresponding to the speech to be recognized. Each text segment representation is a token composed of the text content corresponding to one or more speech frames (one speech frame corresponds to one time frame) in the speech to be recognized.

[0033] Specifically, firstly, for each time frame in the context representation, the weight value and boundary value corresponding to the context vector are predicted; then, the context vectors corresponding to all time frames in the context representation are sequentially traversed, and the text segment representation sequence and the number of text segment representations in the text segment representation sequence are output according to the weight value and boundary value corresponding to each context vector.

[0034] The weight values ​​measure the contribution of the context vector corresponding to the current time frame to the text segment representation output; the boundary values ​​measure the probability that the context vector corresponding to the current time frame is the last context vector in the text segment representation. Any deep learning network structure that can achieve this function, such as predicting the weight and boundary values ​​corresponding to the context vector, can be adjusted according to the actual situation and is not limited here.

[0035] Here, the context vectors corresponding to all time frames in the context representation are sequentially traversed. Based on the weight value and boundary value of each context vector, the text segment representation sequence and the number of time frames corresponding to each text segment representation in the text segment representation sequence are output. This includes: for each traversal, calculating the sum of the weight values ​​of all context vectors that have been traversed but have not yet formed a text segment representation; if the sum is greater than or equal to the preset weight value and the boundary value of the current traversed context vector is greater than or equal to the preset boundary value, then all context vectors that have been traversed but have not yet formed a text segment representation are combined into a text segment representation, the text segment representation is output, and the traversal continues; otherwise, the traversal continues.

[0036] For example, the context representation has context vectors for 10 time frames, with weight values ​​of [0.3, 0.4, 0.3, 0.2, 0.5, 0.3, 0.1, 0.4, 0.5, 0.1] and boundary values ​​of [0.3, 0.7, 0.8, 0.6, 0.9, 0.9, 0.1, 0.6, 0.8, 0.2]. The preset weight value is set to 1.0, and the preset boundary value is set to 0.8. The sum of the weights of the context vectors for the first three time frames is 0.3 + 0.4 + 0.3 = 1.0. 1.0 equals the preset weight value of 1.0, and the boundary value of the last context vector, 0.8, equals 0.8. Therefore, the context vectors of the first three time frames are combined to form the first text segment representation. The sum of the weights of the context vectors from the 4th to the 6th time steps is 0.2 + 0.5 + 0.3 = 1.0. Since 1.0 equals the preset weight value of 1.0, and the boundary value of the last context vector (0.9) is greater than 0.8, the context vectors from the 4th to the 6th time steps are combined to form the second text segment representation. Similarly, the context vectors from the 7th to the 9th time steps are combined to form the third text segment representation. Finally, the context vectors from the 10 time frames generate 3 text segment representations, corresponding to 3 tokens.

[0037] Here, since the subsequent non-autoregressive paradigm requires modeling the alignment between speech and text, this embodiment of the application uses a predictor to dynamically achieve soft alignment between the audio feature sequence and the text sequence.

[0038] Step 3: Input the context representation, text segment representation sequence, and number of text segment representations into the feature decoder in the feature extraction module to obtain the shared representation matrix.

[0039] In this embodiment, in addition to outputting a shared representation matrix, the feature decoder can also output a sequence of text segments corresponding to the speech to be recognized. Each text segment sequence contains at least one text segment, which is plain text content that can be directly read by humans. The number of text segments in the text segment sequence is the same as the number of text segment representations. The feature decoder in this embodiment is a non-autoregressive bidirectional Transformer structure, which can be adjusted according to actual conditions.

[0040] Here, the spoken language understanding model provided in this application is an end-to-end model. Traditional end-to-end models directly convert speech into structured information, lacking clear text output and exhibiting insufficient engineering compatibility and interpretability. In this application, the feature extraction module employs an encoder-predictor-decoder architecture, which can accurately output text content and share a representation matrix. This avoids the cascading errors caused by modules in pipeline models and improves engineering compatibility and interpretability.

[0041] S103. Input the shared representation matrix into the modality transformation module in the spoken language understanding model to obtain the initial text representation.

[0042] In this embodiment, the initial text representation includes text feature vectors corresponding to each text segment in the speech to be recognized. This representation is completely detached from the acoustic information of the speech modality. During its generation, the modality conversion module removes acoustic attributes such as pitch, spectral energy, and pronunciation duration from the shared representation matrix, retaining only the text semantic information mapped from the speech. Its feature space is perfectly aligned with the text semantic space and belongs to the same feature space as the semantic features extracted from natural text; therefore, it is a pure text semantic feature. The specific implementation process is as follows: Step 1: Input the shared representation matrix into the lexical probability quantization unit in the modality conversion module to obtain the lexical sequence probability distribution. The lexical sequence probability distribution is used to represent the matching probability of the speech segment corresponding to each text segment in the text sequence to be recognized with each preset lexical in the preset lexical set.

[0043] In this embodiment, the higher the matching probability between a text segment and a preset word, the higher the probability that the text content corresponding to the text segment is the preset word. The word probability quantization unit is a linear layer. Substituting the shared representation matrix into the following formula, the word sequence probability distribution is obtained: ; in, Let [the token sequence] be the probability distribution, and its dimension is []. V represents the number of preset words in the preset word set; For activation functions; For word probability quantization unit The trainable weight matrix; To share the representation matrix; For word probability quantization unit Trainable bias terms.

[0044] Step 2: Input the probability distribution of the word sequence into the text feature mapping unit in the modality conversion module to obtain the initial text representation.

[0045] In this embodiment, the initial text representation includes the text feature vector of each text segment corresponding to the speech segment in the sequence of text segments to be recognized; the text feature mapping unit is a linear layer, and the probability distribution of the word sequence is substituted into the following formula to obtain the initial text representation: ; in, The initial text representation has the following dimensions: [ ], where T is the dimension of each text feature vector in the initial text representation; This is matrix multiplication; Let be the trainable matrix parameters of the text feature mapping unit e, whose dimensions are [ ].

[0046] Here, the shared representation matrix is ​​a cross-modal fusion feature obtained by the feature extraction module after encoding the speech feature map. The acoustic features are the speech physical attribute information implicit in the vector space and have not yet been converted into pure text semantic features. If the cross-modal feature conversion step is skipped and this vector mixed with acoustic information is directly input into the subsequent text enhancement module, the optimization logic loop of the model will be broken. The optimization goal of the structured information generation module is to learn the structured rules of text semantics, while the optimization goal of the feature extraction module is to learn the mapping rules from speech to text. Their optimization directions differ, making joint training and collaborative optimization of parameters impossible. Simultaneously, the speech encoding error of the feature extraction module will be directly transmitted to the text enhancement module, and then superimposed with the semantic processing error of the text enhancement module, forming a cascaded error between modules. This error will gradually accumulate, ultimately reducing the accuracy of the spoken language understanding model in recognizing structured information such as intent and entities. Therefore, in this embodiment, the shared representation matrix is ​​converted into pure text output, i.e., the initial text representation, through a modality conversion module.

[0047] S104. Input the initial text representation into the text enhancement module in the spoken language understanding model to enhance the semantic logical coherence and the relationship between text elements in the initial text representation, and obtain the enhanced text representation.

[0048] In this application embodiment, text representation is enhanced. It contains enhanced lexical feature vectors of the speech segments corresponding to each text segment in the sequence of text segments to be identified, with dimensions [ ]. Where R is the dimension of each enhanced word feature vector.

[0049] The text enhancement module employs a BERT basic model architecture consisting of 12 stacked Transformer encoders (768 hidden layers and 12 heads for multi-head self-attention). Furthermore, before each Transformer encoder's self-attention sub-layer, a self-attention mechanism and Rotary Embedded Encoding (ROPE) technique are introduced. The specific implementation logic is as follows: The input of each Transformer encoder layer is first processed by a self-attention sublayer. The output is then fed into a feedforward network sublayer (containing two linear transformations and a GELU activation function) after passing through a "residual connection + LayerNorm normalization". After another "residual connection + LayerNorm normalization", the output of that layer is obtained. The 12 encoder layers are stacked in sequence to achieve multi-layer semantic enhancement of the text representation.

[0050] (2) Interaction enhancement logic of self-attention mechanism: Each Transformer self-attention sub-layer calculates the "Query, Key, Value" vector for the input text representation sequence, and uses the scaling dot product attention axiom ( ,in, (where the dimension is the key vector), generate attention weights for each text element (i.e., the text feature vector corresponding to each text segment) and all elements within the initial text representation; by weighting and summing the "value" vectors of all elements, the long-distance text element dependencies can be captured, thereby enhancing the semantic interaction ability between text elements within the sequence (for example, in "book a flight from Beijing tomorrow", the association representation between "tomorrow" and "departure", and between "Beijing" and "flight ticket" can be strengthened).

[0051] (3) Position-aware enhancement logic of rotational position encoding: After querying and calculating the key vector in the self-attention sublayer, the vector is encoded by a rotation matrix to inject the relative position information of the text elements into the vector. This method enables the model to accurately distinguish the sequence order of text elements (e.g., avoid confusing the semantic difference between "cancel booking" and "booking cancellation"), thereby enhancing the position-awareness capability of the text enhancement module. After the above processing, the semantic logical coherence and text element association of the initial text representation are strengthened, and the output enhanced text representation can be directly used as the input feature of the downstream structured information generation module.

[0052] S105. Input the enhanced text representation into the structured information generation module in the spoken language understanding model to obtain the structured information corresponding to the speech to be recognized.

[0053] In this embodiment, the structured information includes named entity information, intent information, and / or sentiment information. The structured information generation module includes a named entity recognition unit for generating named entity information, an intent recognition unit for generating intent information, and a sentiment analysis unit for analyzing sentiment information. The user can specify which information from the named entity information, intent information, and sentiment information to recognize. Based on the content of the structured information specified by the user, the enhanced text representation is input into the named entity recognition unit, the intent recognition unit, and / or the sentiment analysis unit. Specifically, the enhanced text representation is input into the named entity recognition unit, the intent recognition unit, and / or the sentiment analysis unit in the structured information generation module to obtain the named entity information, intent information, and / or sentiment information from the structured information corresponding to the speech to be recognized.

[0054] Specifically, refer to Figure 2 The diagram shown is a flowchart of the named entity information recognition process provided in an embodiment of this application: S201. Input the enhanced text representation into the named entity tagging layer in the named entity recognition unit to obtain the named entity start tag vector and the named entity end tag vector.

[0055] In this embodiment, the named entity start marker vector contains marker information indicating whether each enhanced lexical feature vector in the enhanced text representation is the start position of a named entity; the named entity end marker vector contains marker information indicating whether each enhanced lexical feature vector is the end position of a named entity and the named entity type corresponding to the enhanced lexical feature vector of the named entity. The specific implementation process is as follows: Step 1: Input the enhanced text representation into the feature splitting block of the named entity tagging layer to obtain the first text representation and the second text representation.

[0056] In this embodiment, firstly, the enhanced text representation is upsampled; the dimension of each enhanced lexical feature vector in the upsampled enhanced text representation (e.g., 1596) is twice the dimension of each enhanced lexical feature vector in the unupsampled enhanced text representation (e.g., 768); then, the upsampled enhanced text representation is linearly mapped to obtain a first text representation and a second text representation with the same dimension; the first text representation carries the semantic features of the starting boundary of the named entity; the second text representation contains the second text feature vector of each text segment corresponding to the speech segment in the sequence of text segments to be identified, and the second text representation carries the dual semantics of the "ending boundary" and "entity category" of the named entity. The expression of its partitioning formula is as follows: ; in, This is the first text representation (the first half of each enhanced lexical feature vector after linear mapping). This is the second text representation (the latter half of each enhanced lexical feature vector after linear mapping). It is a fully connected layer. The enhanced text representation after upsampling has dimensions of [ ], =2R.

[0057] Among them, fully connected layers are the most basic linear transformation layers in deep learning, and their core function is to perform linear mapping on the input feature vectors.

[0058] Step 2: Input the first text representation into the first named entity feature extraction block of the named entity tagging layer to obtain the first named entity feature matrix; and input the second text representation into the second named entity feature extraction block of the named entity tagging layer to obtain the second named entity feature matrix; the second named entity feature matrix is ​​used to provide the basis for determining the start position of the named entity; the second named entity feature matrix is ​​used to provide the joint basis for determining the end position of the named entity and the named entity type; In this embodiment of the application, the first named entity feature matrix The first named entity feature vector contains the first named entity feature vector for each text segment corresponding to the speech segment in the sequence of text segments to be identified. Each first named entity feature vector contains the textual semantic information and relative position information of the corresponding speech segment. The second named entity feature matrix... It contains a second named entity feature vector for each text segment corresponding to the speech segment in the sequence of text segments to be identified. Each second named entity feature vector contains the textual semantic information and relative position information of the corresponding speech segment, and also carries the associated semantic information of the entity type. The specific formula is as follows: ; in, This is a rotation embedding coding technique.

[0059] Step 3: Input the first named entity feature matrix into the named entity start tag block of the named entity tag layer to obtain the named entity start tag vector; input the second named entity feature matrix into the named entity end tag block of the named entity tag layer to obtain the named entity end tag vector.

[0060] In this embodiment, slots are used to extract structured information from each text segment (one text segment corresponds to one token) in the text segment sequence corresponding to the specified speech to be recognized. This represents a sequence of pointer tags with a pointer tag set of Ω. This pointer sequence corresponds one-to-one with the text segments in the text segment sequence corresponding to the speech to be recognized output by the feature extraction module. This refers to the pointer label representing the i-th text segment in the sequence of text segments corresponding to the speech to be recognized. This application defines two types of pointer label sequences: the starting pointer sequence and the... and end pointer label sequence , This is the (M-1)th named entity type. The former is used to locate the start position of the named entity, and the latter is used to locate the end position of the named entity type.

[0061] Specifically, the first named entity feature matrix is ​​input into the named entity start marker block of the named entity marker layer to obtain the named entity start marker vector, including: determining the start pointer sequence based on the first named entity feature matrix; the start pointer sequence includes the first probability that each text segment in the text segment sequence corresponding to the speech to be recognized is the start position of the named entity; the start pointer sequence is input into the sigmoid activation function to normalize all the first probabilities to obtain the second probability that each text segment is the start position of the named entity; if the second probability is 1 and is greater than the preset start position probability value, then the corresponding text segment in the named entity start marker vector is marked as the named entity start marker value (e.g., 1); otherwise, the corresponding text segment in the named entity start marker vector is marked as a non-named entity start marker value (e.g., 0).

[0062] Here, the starting pointer sequence can be determined using a linear layer based on the first named entity feature matrix.

[0063] Specifically, the second named entity feature matrix is ​​input into the named entity end marker block of the named entity tagging layer to obtain the named entity end marker vector, including: determining the end pointer matrix based on the first named entity feature matrix; the end pointer matrix includes the third probability of the non-end named entity of each text segment in the text segment sequence corresponding to the speech to be recognized and the fourth probability corresponding to each named entity type; for each text segment, the preset marker value of the non-end named entity corresponding to the maximum probability value of the third probability and the fourth probability corresponding to the text segment or the preset marker value corresponding to the named entity type is determined as the marker value of the text segment in the named entity end marker vector.

[0064] Wherein, the number 0 indicates the end position of a non-entity, and the numbers 1, 2, ..., Indicates the end position of different named entity types.

[0065] Here, determining the end pointer matrix can be achieved using a linear layer. "Determining the preset tag value of the ending named entity corresponding to the maximum probability value among the third and fourth probabilities of the text segment, or the preset tag value corresponding to the named entity type, as the tag value of the text segment in the named entity end tag vector" can be achieved using the maximum value algorithm argmax().

[0066] S202. The enhanced text representation, the named entity start marker vector, and the named entity end marker vector are input into the named entity generation layer in the named entity recognition unit to generate named entity information in the structured information corresponding to the speech to be recognized.

[0067] Additionally, refer to Figure 3 The diagram shown is a flowchart illustrating the generation of intent information or sentiment information provided in this application embodiment. Specifically, the enhanced text representation is input into the intent recognition unit or sentiment analysis unit in the structured information generation module according to the following steps to obtain the intent information or sentiment information in the structured information corresponding to the speech to be recognized: S301. Extract the first classification feature block from the enhanced text representation input intent recognition unit or sentiment analysis unit to obtain the word classification feature matrix.

[0068] In this embodiment of the application, the word classification feature matrix contains the classification feature vector of each text segment in the text segment sequence corresponding to the speech to be identified.

[0069] Both the intent recognition unit and the sentiment analysis unit employ Rotation Embedding (ROPE) coding technique to embed relative position information into the enhanced text representation, resulting in a word-level word classification feature matrix. Its formula is expressed as: .

[0070] S302. Input the word classification feature matrix into the second classification feature extraction block in the intent recognition unit or sentiment analysis unit to obtain the word weight sequence.

[0071] In this embodiment of the application, the lexical weight sequence includes the importance value of each text segment in the text segment sequence corresponding to the speech to be recognized.

[0072] Here, the intent recognition unit and the sentiment analysis unit use an improved self-attention pooling mechanism to analyze token-level features. Extract sentence-level classification features, i.e., word weight sequences. The formula for implementing the self-attention pooling mechanism is shown below: ; in, The dimension is [ ]. This is the trainable weight matrix for the second classification feature extraction block. This is the trainable bias term for the second classification feature extraction block.

[0073] S303. Input the word weight sequence and word classification feature matrix into the classification block of the intent recognition unit or sentiment analysis unit to obtain the intent information or sentiment information in the structured information corresponding to the speech to be recognized.

[0074] In this embodiment of the application, the intent information or emotion information in the structured information corresponding to the speech to be recognized is obtained by the following formula: ; in, For the number of intention categories or the number of emotion types, This refers to information of intent or emotion. The trainable weight matrix for the classification blocks. For the trainable bias term of the classification block.

[0075] In addition, the spoken language understanding model proposed in this application calculates three loss functions during training: cross-entropy loss. Mean Absolute Error (MAE) And Minimum Word Error Rate (MWER) In the feature extraction module, cross-entropy and minimizing the word error rate are used to optimize the transcriptional ability of the spoken language understanding model (i.e., the ability to output a sequence of text segments of the speech to be recognized), while the mean absolute error is used to guide the convergence of the predictor in the feature extraction module. The loss function of the feature extraction module is shown below: ,in, The speech to be recognized. The actual text annotation for the speech to be recognized. This is the first preset weight. This represents the loss value of the feature extraction module.

[0076] ; in, The i-th text segment in the output sequence of text segments of the speech to be recognized. Let be the probability predicted by the spoken language comprehension model for the i-th text segment. For the i-th text segment and the actual annotation The word error rate of the i-th text segment. The standard value for word error rate.

[0077] In the structured information generation module, cross-entropy loss is also used to optimize the model's ability to learn structured information. Therefore, the overall loss function of the spoken language comprehension model... for: .

[0078] in, This is the loss function for the structured information generation module. This is the second preset weight.

[0079] Based on the same inventive concept, this application also provides a spoken language understanding device corresponding to the spoken language understanding method. Since the principle of the device in this application is similar to the spoken language understanding method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0080] Reference Figure 4 The diagram shown is a schematic representation of a spoken language understanding device provided in an embodiment of this application. The spoken language understanding device includes: Extraction module 401 is used to extract the speech feature map corresponding to the speech to be recognized; The input module 402 is used to input the speech feature map into the feature extraction module in the spoken language understanding model to obtain a shared representation matrix; the shared representation matrix integrates the acoustic information and text semantics of the speech to be recognized. The input module 402 is further configured to input the shared representation matrix into the modality conversion module in the spoken language understanding model to obtain the initial text representation; The input module 402 is further configured to input the initial text representation into the text enhancement module in the spoken language understanding model to enhance the semantic logical coherence and the correlation between text elements in the initial text representation, thereby obtaining an enhanced text representation. The input module 402 is further configured to input the enhanced text representation into the structured information generation module in the spoken language understanding model to obtain the structured information corresponding to the speech to be recognized.

[0081] In one possible implementation, the input module 402 is specifically used to input the speech feature map into the feature encoder in the feature extraction module to obtain a context representation; the context representation is a feature matrix that fuses the local acoustic features and global text semantics corresponding to the speech to be recognized; the context representation is input into the predictor in the feature extraction module to obtain a text segment representation sequence and the number of text segment representations in the text segment representation sequence; the context representation, the text segment representation sequence, and the number of text segment representations are input into the feature decoder in the feature extraction module to obtain the shared representation matrix.

[0082] In one possible implementation, the input module 402 is specifically used to input the shared representation matrix into the lexical probability quantization unit in the modality conversion module to obtain a lexical sequence probability distribution; the lexical sequence probability distribution is used to represent the matching probability of the speech segment corresponding to each text segment in the text sequence to be recognized with each preset lexical in the preset lexical set; the lexical sequence probability distribution is input into the text feature mapping unit in the modality conversion module to obtain an initial text representation.

[0083] In one possible implementation, the input module 402 is specifically used to input the enhanced text representation into the named entity recognition unit, intent recognition unit, and / or sentiment analysis unit in the structured information generation module to obtain the named entity information, intent information, and / or sentiment information in the structured information corresponding to the speech to be recognized.

[0084] In one possible implementation, the input module 402 is specifically configured to input the enhanced text representation into the named entity recognition unit according to the following steps, to obtain named entity information in the structured information corresponding to the speech to be recognized: The enhanced text representation is input into the named entity tagging layer in the named entity recognition unit to obtain a named entity start tag vector and a named entity end tag vector. The named entity start tag vector contains tagging information indicating whether each enhanced lexical feature vector in the enhanced text representation is the start position of a named entity. The named entity end tag vector contains tagging information indicating whether each enhanced lexical feature vector is the end position of a named entity and the named entity type corresponding to the enhanced lexical feature vector of the named entity. The enhanced text representation, the named entity start marker vector, and the named entity end marker vector are input into the named entity generation layer in the named entity recognition unit to generate named entity information in the structured information corresponding to the speech to be recognized.

[0085] In one possible implementation, the input module 402 is specifically configured to input the enhanced text representation into the named entity tagging layer of the named entity recognition unit according to the following steps, to obtain a named entity start tagging vector and a named entity end tagging vector: The enhanced text representation is input into the feature splitting block of the named entity tagging layer to obtain the first text representation and the second text representation. The first text representation is input into the first named entity feature extraction block of the named entity tagging layer to obtain the first named entity feature matrix; and the second text representation is input into the second named entity feature extraction block of the named entity tagging layer to obtain the second named entity feature matrix; the second named entity feature matrix is ​​used to provide the basis for determining the start position of a named entity; the second named entity feature matrix is ​​used to provide the joint basis for determining the end position of a named entity and the type of a named entity. The first named entity feature matrix is ​​input into the named entity start tag block of the named entity tag layer to obtain the named entity start tag vector; the second named entity feature matrix is ​​input into the named entity end tag block of the named entity tag layer to obtain the named entity end tag vector.

[0086] In one possible implementation, the input module 402 is specifically configured to input the enhanced text representation into the intent recognition unit or sentiment analysis unit in the structured information generation module according to the following steps, to obtain intent information or sentiment information in the structured information corresponding to the speech to be recognized: The enhanced text representation is input into the first classification feature extraction block in the intent recognition unit or sentiment analysis unit to obtain a word classification feature matrix; the word classification feature includes the classification feature vector of each text segment in the text segment sequence corresponding to the speech to be recognized; The word classification feature matrix is ​​input into the second classification feature extraction block in the intent recognition unit or sentiment analysis unit to obtain a word weight sequence; the word weight sequence contains the importance value of each text segment in the text segment sequence corresponding to the speech to be recognized. The word weight sequence and the word classification feature matrix are input into the classification block of the intent recognition unit or sentiment analysis unit to obtain the intent information or sentiment information in the structured information corresponding to the speech to be recognized.

[0087] like Figure 5As shown in the embodiment of this application, an electronic device 500 includes a processor 501, a memory 502, and a bus. The memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device is running, the processor 501 communicates with the memory 502 via the bus, and the processor 501 executes the machine-readable instructions to perform the steps of the spoken language understanding method described above.

[0088] Specifically, the memory 502 and processor 501 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 501 runs the computer program stored in the memory 502, it can execute the above-mentioned spoken language understanding method.

[0089] Corresponding to the above-described spoken language understanding method, this application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described spoken language understanding method.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0091] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0093] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0094] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A spoken language comprehension method, characterized in that, The method includes: Extract the speech feature map corresponding to the speech to be recognized; The speech feature map is input into the feature extraction module of the spoken language understanding model to obtain a shared representation matrix; the shared representation matrix integrates the acoustic information and text semantics of the speech to be recognized. The shared representation matrix is ​​input into the modality transformation module in the spoken language understanding model to obtain the initial text representation; The initial text representation is input into the text enhancement module in the spoken language understanding model to enhance the semantic logical coherence and the relationship between text elements in the initial text representation, thereby obtaining the enhanced text representation. The enhanced text representation is input into the structured information generation module in the spoken language understanding model to obtain the structured information corresponding to the speech to be recognized.

2. The spoken language comprehension method according to claim 1, characterized in that, The step of inputting the speech feature map into the feature extraction module of the spoken language understanding model to obtain a shared representation matrix includes: The speech feature map is input into the feature encoder in the feature extraction module to obtain a context representation; the context representation is a feature matrix that fuses the local acoustic features and global text semantics of the speech to be recognized. The context representation is input into the predictor in the feature extraction module to obtain a text segment representation sequence and the number of text segment representations in the text segment representation sequence. The context representation, the text segment representation sequence, and the number of text segment representations are input into the feature decoder in the feature extraction module to obtain the shared representation matrix.

3. The spoken language comprehension method according to claim 1, characterized in that, The step of inputting the shared representation matrix into the modality transformation module of the spoken language understanding model to obtain the initial text representation includes: The shared representation matrix is ​​input into the lexical probability quantization unit in the modality conversion module to obtain the lexical sequence probability distribution; the lexical sequence probability distribution is used to represent the matching probability of the speech segment corresponding to each text segment in the text sequence corresponding to the speech to be recognized with each preset lexical in the preset lexical set. The probability distribution of the word sequence is input into the text feature mapping unit in the modality conversion module to obtain the initial text representation.

4. The spoken language comprehension method according to claim 1, characterized in that, The step of inputting the enhanced text representation into the structured information generation module of the spoken language understanding model to obtain the structured information corresponding to the speech to be recognized includes: The enhanced text representation is input into the named entity recognition unit, intent recognition unit, and / or sentiment analysis unit in the structured information generation module to obtain the named entity information, intent information, and / or sentiment information in the structured information corresponding to the speech to be recognized.

5. The spoken language comprehension method according to claim 4, characterized in that, The enhanced text representation is input into the named entity recognition unit according to the following steps to obtain the named entity information in the structured information corresponding to the speech to be recognized: The enhanced text representation is input into the named entity tagging layer in the named entity recognition unit to obtain a named entity start tag vector and a named entity end tag vector. The named entity start tag vector contains tagging information indicating whether each enhanced lexical feature vector in the enhanced text representation is the start position of a named entity. The named entity end tag vector contains tagging information indicating whether each enhanced lexical feature vector is the end position of a named entity and the named entity type corresponding to the enhanced lexical feature vector of the named entity. The enhanced text representation, the named entity start marker vector, and the named entity end marker vector are input into the named entity generation layer in the named entity recognition unit to generate named entity information in the structured information corresponding to the speech to be recognized.

6. The spoken language comprehension method according to claim 5, characterized in that, The enhanced text representation is input into the named entity tagging layer of the named entity recognition unit according to the following steps to obtain the named entity start tagging vector and the named entity end tagging vector: The enhanced text representation is input into the feature splitting block of the named entity tagging layer to obtain the first text representation and the second text representation. The first text representation is input into the first named entity feature extraction block of the named entity tagging layer to obtain the first named entity feature matrix; and the second text representation is input into the second named entity feature extraction block of the named entity tagging layer to obtain the second named entity feature matrix; the second named entity feature matrix is ​​used to provide the basis for determining the start position of a named entity; the second named entity feature matrix is ​​used to provide the joint basis for determining the end position of a named entity and the type of a named entity. The first named entity feature matrix is ​​input into the named entity start tag block of the named entity tag layer to obtain the named entity start tag vector; the second named entity feature matrix is ​​input into the named entity end tag block of the named entity tag layer to obtain the named entity end tag vector.

7. The spoken language comprehension method according to claim 4, characterized in that, The enhanced text representation is input into the intent recognition unit or sentiment analysis unit in the structured information generation module according to the following steps to obtain the intent information or sentiment information in the structured information corresponding to the speech to be recognized: The enhanced text representation is input into the first classification feature extraction block in the intent recognition unit or sentiment analysis unit to obtain a word classification feature matrix; the word classification feature matrix contains the classification feature vector of each text segment in the text segment sequence corresponding to the speech to be recognized; The word classification feature matrix is ​​input into the second classification feature extraction block in the intent recognition unit or sentiment analysis unit to obtain a word weight sequence; the word weight sequence contains the importance value of each text segment in the text segment sequence corresponding to the speech to be recognized. The word weight sequence and the word classification feature matrix are input into the classification block of the intent recognition unit or sentiment analysis unit to obtain the intent information or sentiment information in the structured information corresponding to the speech to be recognized.

8. A spoken language comprehension device, characterized in that, The device includes: The extraction module is used to extract the speech feature map corresponding to the speech to be recognized; The input module is used to input the speech feature map into the feature extraction module in the spoken language understanding model to obtain a shared representation matrix; the shared representation matrix integrates the acoustic information and text semantics of the speech to be recognized. The input module is also used to input the shared representation matrix into the modality conversion module in the spoken language understanding model to obtain the initial text representation; The input module is also used to input the initial text representation into the text enhancement module in the spoken language understanding model to enhance the semantic logical coherence and the correlation between text elements in the initial text representation, so as to obtain an enhanced text representation. The input module is further configured to input the enhanced text representation into the structured information generation module in the spoken language understanding model to obtain the structured information corresponding to the speech to be recognized.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the spoken language understanding method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the spoken language understanding method as described in any one of claims 1 to 7.

Images