Streaming semantic understanding method and apparatus

Abstract

The application discloses a streaming semantic understanding method and device. The method comprises the following steps: performing real-time speech recognition on acquired voice data stream to obtain text data stream corresponding to the voice data stream; performing real-time segmentation on the text data stream to obtain at least one text segment contained in the text data stream; and performing real-time semantic understanding on the at least one text segment to obtain semantic understanding result of the at least one text segment. The application can recognize input instructions of a user in real time, and can perform semantic understanding on the recognition result in real time, thereby realizing streaming semantic understanding, greatly reducing the difficulty of whole sentence understanding, and improving user experience.

Description

Technical Field

[0001] This invention relates generally to the field of natural language processing, and more specifically to a streaming semantic understanding method and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence, various smart products are emerging, such as smart home devices and in-vehicle virtual assistants. Voice interaction technology, as one of the most common human-computer interaction methods, has been widely applied to various smart products. As the demand for these smart products continues to increase, their supporting functions and corresponding technologies also need to keep pace with the times. Therefore, how to better understand the commands issued by users to smart products is a technical problem that urgently needs to be solved by those skilled in the art.

[0003] Existing semantic understanding methods mostly adopt the one-by-one approach, which suffers from poor performance, long user waiting time, and a poor user experience.

[0004] Therefore, in order to solve the above problems, a new streaming semantic understanding method and device are needed to address the issues existing in current semantic understanding methods. Summary of the Invention

[0005] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. The summary section of this invention is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0006] According to one aspect of the present invention, a streaming semantic understanding method is provided, the method comprising: performing speech recognition on an acquired speech data stream in real time to obtain a text data stream corresponding to the speech data stream; segmenting the text data stream in real time to obtain at least one text segment contained in the text data stream; and performing semantic understanding on the at least one text segment in real time to obtain a semantic understanding result of the at least one text segment.

[0007] In one embodiment, the method further includes: performing endpoint detection on the acquired speech data stream to obtain a first time interval between the various speech segments contained in the speech data stream.

[0008] In one embodiment, segmenting the text data stream in real time includes: encoding a text sequence of the text data stream in real time to obtain a semantic representation of the text sequence; calculating a predicted pause probability of the text data stream based on the semantic representation of the text sequence; and determining a pause position of the text data stream based on the predicted pause probability, thereby segmenting the text data stream at the pause position.

[0009] In one embodiment, encoding a text sequence of the text data stream in real time to obtain a semantic representation of the text sequence includes: calculating a semantic vector of the text sequence based on the text sequence of the text data stream; calculating an attention weight of the semantic vector and calculating a first context information vector based on the attention weight; and calculating a semantic representation of the text sequence based on the first context information vector.

[0010] In one embodiment, determining the pause position of the text data stream based on the predicted pause probability includes: calculating a predicted second time interval based on the predicted pause probability; and determining the pause position of the text data stream based on the first time interval and the second time interval.

[0011] In one embodiment, the semantic understanding result includes an intent understanding result, which includes intent text. Semantic understanding of the at least one text segment to obtain a semantic understanding result for the at least one text segment includes: encoding a segment sequence of the at least one text segment to obtain a segment vector corresponding to the segment sequence; and calculating the intent text using the segment vector based on an attention mechanism.

[0012] In one embodiment, the intent understanding result further includes an intent type. Semantic understanding of the at least one text segment to obtain a semantic understanding result of the at least one text segment further includes: encoding the word-mixed sequence of the intent text to obtain an intent vector corresponding to the intent text; and obtaining the intent type using the intent vector and the segment vector based on an attention mechanism.

[0013] In one embodiment, the semantic understanding result further includes a slot value understanding result. Semantic understanding of the at least one text segment to obtain a semantic understanding result for the at least one text segment further includes: calculating a second context information vector using the intent vector and the segment vector based on an attention mechanism; calculating a third context information vector using the segment vector based on an attention mechanism; and concatenating the second context information vector, the third context information vector, and the segment vector to obtain a concatenated vector, and calculating the slot value understanding result based on the concatenated vector.

[0014] According to another aspect of the present invention, a streaming semantic understanding apparatus is provided, the apparatus including a memory and a processor, the memory storing a computer program executed by the processor, the computer program causing the processor to perform the streaming semantic understanding method as described above when executed by the processor.

[0015] According to another aspect of the present invention, a computer-readable medium is provided, wherein computer-executable instructions are stored thereon, wherein when executed, the computer-executable instructions perform the streaming semantic understanding method as described above.

[0016] The streaming semantic understanding method and apparatus according to embodiments of the present invention can recognize user input commands in real time and perform semantic understanding on the recognition results in real time, thereby realizing streaming semantic understanding, greatly reducing the difficulty of understanding whole sentences and improving user experience. Attached Figure Description

[0017] The following drawings, which are incorporated herein by reference as part of this invention, are provided for understanding the invention. The drawings illustrate embodiments of the invention and their descriptions, serving to explain the principles of the invention.

[0018] In the attached image:

[0019] Figure 1 An exemplary flowchart of a streaming semantic understanding method according to an embodiment of the present invention is shown; and

[0020] Figure 2 A schematic structural block diagram of a streaming semantic understanding device according to an embodiment of the present invention is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0022] As mentioned above, most existing semantic understanding methods employ a one-by-one approach. This method involves the user sending a query to a smart product, which then understands the query, outputs the semantic understanding result, and responds. Specifically, this involves obtaining the intent and corresponding slot information contained in the query. For example, in the query "Open the car window halfway," the intent is defined as "open the car window," and the slot information is "value: halfway." By combining the intent and slot, the semantic understanding result of the query can be obtained, and a response can be initiated.

[0023] However, in real-world scenarios, most users make several consecutive, multi-intent requests, such as: "Open the car window halfway and play one of my favorite Andy Lau songs," or "Draw the curtains, turn on the air conditioner to 22 degrees Celsius, and turn on the projector to play a movie." Current semantic understanding methods wait until the voice input is complete before treating the entire speech recognition result as a single query command for semantic understanding. In this case, the one-by-one approach has two problems: first, if the user's query command is too long or complex, the difficulty of semantic understanding increases, significantly reducing its effectiveness; second, waiting for all query commands to be fully recognized before semantic understanding results in a long waiting time, leading to a poor user experience.

[0024] Therefore, in order to solve the above-mentioned problems of existing semantic understanding methods, the present invention provides a streaming semantic understanding method, which includes: performing speech recognition on the acquired speech data stream in real time to obtain a text data stream corresponding to the speech data stream; segmenting the text data stream in real time to obtain at least one text segment contained in the text data stream; and performing semantic understanding on the at least one text segment in real time to obtain a semantic understanding result of the at least one text segment.

[0025] The streaming semantic understanding method of the present invention can recognize user input commands in real time and perform semantic understanding on the recognition results in real time, thereby realizing streaming semantic understanding, which greatly reduces the difficulty of understanding whole sentences and improves the user experience.

[0026] The following describes in detail the streaming semantic understanding method and apparatus according to the present invention with reference to specific embodiments.

[0027] First, refer to Figure 1 , Figure 1 An exemplary flowchart of a streaming semantic understanding method 100 according to an embodiment of the present invention is shown. The streaming semantic understanding method 100 of the present invention can be applied to any intelligent device, equipment, product, etc. that requires human-computer interaction, and the present invention does not limit it.

[0028] like Figure 1 As shown, the streaming semantic understanding method 100 may include the following exemplary steps:

[0029] In step S110, the acquired voice data stream is subjected to real-time speech recognition to obtain a text data stream corresponding to the voice data stream;

[0030] In step S120, the text data stream is segmented in real time to obtain at least one text segment contained in the text data stream; and

[0031] In step S130, semantic understanding is performed on at least one text segment in real time to obtain the semantic understanding result of the at least one text segment.

[0032] In step S110, any speech recognition network known in the art can be used to perform speech recognition on the acquired speech data stream. In one embodiment, the speech recognition network may include a pre-trained acoustic model and a language model as a decoding network. The acoustic model can be any acoustic model known in the art, such as GMM-UBM (Gaussian Mixture Model-Universal Background Model), JFA (Joint Factor Analysis) model, GMM-UBM i-vector model, supervised UBMi-vector model, deep neural network i-vector model, etc. The language model can be any language model known in the art, such as N-gram language model, neural probabilistic language model, etc., and the invention is not limited thereto.

[0033] In speech recognition, the decoding network obtains all candidate text paths up to the current moment. During speech recognition, the text path with the highest decoding score is selected from the current decoding results as the current recognized text. When new input speech data is received in real time, the decoding network reselects the text path with the highest decoding score based on the newly received speech data, thus obtaining a new recognized text.

[0034] In one embodiment, during speech recognition, the first time interval Gap1 between different speech segments in the speech data stream can be obtained through an endpoint detection model. This Gap1 is the actual value of the time interval between different speech segments, which can be expressed in milliseconds (ms). It can be assumed that the time interval between different speech segments in the speech data stream should not be less than a threshold time interval Gap0; otherwise, it can be considered a single speech segment. Therefore, only when the time interval of the speech data is greater than this threshold time interval Gap0 will it be considered a single speech segment and sent for the following segmentation.

[0035] In one embodiment, the threshold time interval Gap0 can be preset based on experience, for example, set to 80ms-150ms. In another embodiment, the threshold time interval Gap0 can be 100ms.

[0036] In one embodiment, the real-time segmentation of the text data stream in step S120 can be implemented using a gating network. The gating network may include several neural network layers, such as encoding layers, attention layers, concatenation layers, mapping layers, and decision layers, each used to perform corresponding operations on the text data stream.

[0037] In one embodiment, the real-time segmentation of the text data stream in step S120 may include the following steps:

[0038] In step S1220, the text sequence of the text data stream is encoded in real time to obtain the semantic representation of the text sequence;

[0039] In step S1240, the predicted pause probability of the text data stream is calculated based on the semantic representation of the text sequence; and

[0040] In step S1260, the pause position of the text data stream is determined based on the predicted pause probability, and the text data stream is segmented at the pause position.

[0041] Step S1220 can be implemented by the encoding layer, attention layer and splicing layer of the gating network.

[0042] In one embodiment, step S1220, which involves encoding the text sequence of the text data stream in real time to obtain a semantic representation of the text sequence, includes:

[0043] In step S1222, the semantic vector of the text sequence is calculated based on the text sequence of the text data stream. This step can be implemented in the coding layer.

[0044] In step S1224, attention weights are calculated for the semantic vector, and a first contextual information vector is calculated based on these attention weights. This step can be implemented within the attention layer.

[0045] In step S1226, the semantic representation of the text sequence is calculated based on the first context information vector. This step can be implemented in the concatenation layer.

[0046] In one embodiment, the text sequence of the text data stream can be obtained by vectorizing the text data. Algorithms for vectorizing text data (e.g., word2vec, etc.) are known to those skilled in the art and will not be described in detail here.

[0047] In one embodiment, in the encoding layer, a pre-trained model can be used to encode the text sequence of the text data stream to obtain the semantic vector of the text sequence. Examples include BERT (Transformer-Based Bidirectional Encoder Representation), ALBERT (Lightweight Transformer-Based Bidirectional Encoder Representation), ERNIE (Enhanced Representation with Knowledge Integration), DistilBERT (a distilled version of BERT), and RoBERTa (Robust Optimized BERT Pre-training Method). Other neural network models can also be used to encode the text sequence of the text data stream, such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) models. This invention does not limit the specific models used. In one embodiment, the BERT model is used to encode the text sequence of the text data stream.

[0048] In one embodiment, in the encoding layer, in addition to obtaining the character semantic vector (h1, h2, ..., h) output by the model for each character, it is also possible to obtain the character semantic vector (h1, h2, ..., h) for each character. n In addition to this, it can also obtain the intermediate semantic vector h of the entire sentence. cls The intermediate representation semantic vector can represent all the information of the entire sentence and has learned which characters in the sentence contribute to the semantic representation of the entire sentence. In this paper, both the character semantic vector and the intermediate representation semantic vector are collectively referred to as semantic vectors.

[0049] To accurately capture the contextual information of sentences and the connections between different text segments, attention weights of semantic vectors can be calculated in the attention layer. This invention employs two attention methods: self-attention within a text segment and cross-attention between each word in the current text segment and the entire sentence cls of the previous text segment. After calculating the attention weights, the attention weights are multiplied by the corresponding semantic vectors and concatenated to calculate the first contextual information vector h. att Both attention methods can reflect the semantic representation of the entire sentence, but they have different focuses. Therefore, using both attention methods can better bring in the information from the preceding text as a reference and prevent the problem of text segmentation errors.

[0050] In one embodiment, in the splicing layer, the first context information vector h can be... att With intermediate representation semantic vector h cls The concatenated vectors are used to represent the semantic meaning of the text sequence.

[0051] In one embodiment, step S1240 can be implemented by the output layer of the gating network. In one embodiment, the output layer may include a fully connected layer and a softmax layer. After obtaining the semantic representation of the text sequence, it is fed into the output layer to obtain the predicted pause probability p of the text data stream. This pause probability p can be represented by a value between 0 and 1, where 0 represents that the input sentence should not pause at this point, and 1 represents that the input sentence should pause at this point.

[0052] In one embodiment, step S1260 can be implemented by the mapping layer and the decision layer of the gating network.

[0053] In one embodiment, determining the pause position of the text data stream based on the predicted pause probability in step S1260 may include the following steps:

[0054] The predicted second time interval Gap2 is calculated based on the predicted pause probability; and

[0055] The pause position of the text data stream is determined based on the first time interval Gap1 and the second time interval Gap2.

[0056] The second time interval, Gap2, is the predicted waiting time interval. The smaller Gap2 is, the more complete the semantics of this place; the larger Gap2 is, the less complete the semantics of this place, and the user may want to express more instructions, so they hope to wait a little longer.

[0057] In one embodiment, the second time interval Gap2 in the mapping layer can be calculated using the following formula:

[0058] Gap2 = c1*p + c2*(1-p)

[0059] Where c1 is the lower limit, c2 is the upper limit, and p is the pause probability. c1 and c2 are constants that can be set empirically. In one embodiment, the value of c1 can be 100-200, and the value of c2 can be 500-2000. In another embodiment, the value of c1 is 100, and the value of c2 is 800.

[0060] In one embodiment, in the decision layer, it is determined whether the text data stream should pause, and thus should be truncated, based on a first time interval Gap1 and a second time interval Gap2. Specifically, the conditions for determining whether truncation should occur can be set as follows:

[0061] 1. The first time interval Gap1 is greater than the threshold time interval Gap0;

[0062] 2. If the second time interval Gap2 is less than the first time interval Gap1, then truncate; if the second time interval Gap2 is greater than or equal to the first time interval Gap1, then do not truncate.

[0063] Through the above process, the speech data stream can be segmented into at least one text segment in real time. The resulting text segment is then fed into a semantic understanding model in real time for semantic understanding, thereby obtaining the semantic understanding results for these text segments.

[0064] In one embodiment, the semantic understanding result may include an intent understanding result and a slot value understanding result. For example, for the text fragment "Help me open the car window halfway", the intent understanding result is "open the car window", and the slot value understanding result may be "halfway".

[0065] In one embodiment, the intent understanding result may include intent text and intent type. For example, for the text fragment "Help me open the car window halfway," the intent text is "open the car window," and the intent type is "down." In one embodiment, the obtained intent type may be one of multiple preset intent types, which may include down, up, play, etc., and this invention does not limit this.

[0066] In one embodiment, step S130, which involves performing semantic understanding on at least one text segment in real time to obtain the semantic understanding result of the at least one text segment, may include the following steps:

[0067] Encode a sequence of at least one text segment to obtain a segment vector HH corresponding to the segment sequence; and

[0068] The intent text is calculated using fragment vectors HH based on an attention mechanism.

[0069] In one embodiment, the sequence of text fragments can be obtained by vectorizing the text fragments. Algorithms for vectorizing text fragments (e.g., word2vec, etc.) are known to those skilled in the art and will not be described in detail here.

[0070] In one embodiment, a pre-trained model can be used to encode the sequence of text segments to obtain the segment vector HH of the text segment. Examples include BERT (Transformer-Based Bidirectional Encoder Representation), ALBERT (Lightweight Transformer-Based Bidirectional Encoder Representation), ERNIE (Enhanced Representation with Knowledge Integration), DistilBERT (a distilled version of BERT), and RoBERTa (Robust Optimized BERT Pre-training Method). Other neural network models can also be used to encode the sequence of text segments, such as LSTM (Long Short-Term Memory), BiLSTM (Bidirectional Long Short-Term Memory), and GRU (Gated Recurrent Unit) models. This invention does not limit the specific models used. In one embodiment, a BiLSTM model is used to encode the sequence of text segments.

[0071] In one embodiment, after obtaining the fragment vector HH of the text segment, it can be fed into a self-attention layer and then into a CNN (convolutional neural network) layer. The "half-pointer-half-label" structure is used to predict the beginning and end positions of the intent text, thereby finally obtaining the intent text.

[0072] In one embodiment, step S130, which involves performing semantic understanding on at least one text segment in real time to obtain the semantic understanding result of the at least one text segment, may further include the following steps:

[0073] Encode the word-mixed sequence of the intent text to obtain the intent vector corresponding to the intent text; and

[0074] The intent type is obtained by using intent vectors and fragment vectors based on the attention mechanism.

[0075] In one embodiment, after obtaining the intent text, a pre-trained model can be used to encode the word-mixed sequence of the intent text to obtain the intent vector of the intent text. Examples include BERT (Transformer-Based Bidirectional Encoder Representation), ALBERT (Lightweight Transformer-Based Bidirectional Encoder Representation), ERNIE (Enhanced Representation with Knowledge Integration), DistilBERT (a distilled version of BERT), and RoBERTa (Robust Optimized BERT Pre-training Method). Other neural network models can also be used to encode the word-mixed sequence of the intent text, such as LSTM (Long Short-Term Memory), BiLSTM (Bidirectional Long Short-Term Memory), and GRU (Gated Recurrent Unit) models. This invention does not limit the specific models used. In one embodiment, a BiLSTM model is used to encode the word-mixed sequence of the intent text.

[0076] In one embodiment, after obtaining the intent vector of the intent text, the intent vector of the intent text can be used as a reference to perform attention calculation with the fragment vector HH of the text segment to obtain a second context vector of the same length as the fragment sequence of the text segment, and the intent type can be obtained.

[0077] In one embodiment, step S130, which involves performing semantic understanding on at least one text segment in real time to obtain the semantic understanding result of the at least one text segment, may further include the following steps:

[0078] The second context information vector is calculated using the intention vector and fragment vector based on the attention mechanism;

[0079] The third context information vector is calculated using fragment vectors based on an attention mechanism; and

[0080] The second context information vector, the third context information vector, and the fragment vector are concatenated to obtain a concatenated vector, and the slot value understanding result is calculated based on this concatenated vector.

[0081] In one embodiment, the concatenated vector can be fed into a model such as a CRF (Conditional Random Field) to predict the slot value interpretation result.

[0082] The streaming semantic understanding method of the present invention can recognize user input commands in real time and perform semantic understanding on the recognition results in real time, thereby realizing streaming semantic understanding, which greatly reduces the difficulty of understanding whole sentences and improves the user experience.

[0083] The present invention also provides a streaming semantic understanding device 200. (See reference) Figure 2 , Figure 2A schematic structural block diagram of a streaming semantic understanding device 200 according to an embodiment of the present invention is shown. Figure 2 As shown, the streaming semantic understanding device 200 may include a memory 210 and a processor 220. The memory 210 stores a computer program that is executed by the processor 220. When the computer program is executed by the processor 220, it causes the processor 220 to perform the streaming semantic understanding method 100 described above according to an embodiment of the present invention. Those skilled in the art can understand the specific operation of the streaming semantic understanding device 200 according to an embodiment of the present invention in conjunction with the content described above; for the sake of brevity, it will not be described again here.

[0084] The streaming semantic understanding device of the present invention can recognize user input commands in real time and perform semantic understanding on the recognition results in real time, thereby realizing streaming semantic understanding, which greatly reduces the difficulty of understanding whole sentences and improves the user experience.

[0085] The present invention also provides a computer-readable medium storing computer-executable instructions that, when executed, perform corresponding steps of the streaming semantic understanding method 100 as described above. Any tangible, non-transitory computer-readable medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu-ray discs, etc.), flash memory, and / or the like. These computer-executable instructions may be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to form a machine, such that the instructions, which execute on the computer or other programmable data processing apparatus, can generate means for implementing a specified function. These computer-executable instructions may also be stored in a computer-readable storage medium that can instruct the computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable storage medium can form an article of manufacture, including means for implementing the specified function. The computer-executable instructions may also be loaded onto a computer or other programmable data processing apparatus to perform a series of operational steps on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, can provide steps for implementing the specified function.

[0086] Furthermore, according to embodiments of the present invention, a computer program is also provided, which is executed by a computer or processor to perform corresponding steps of the streaming semantic understanding method 100 of the present invention embodiments.

[0087] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0088] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0089] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0090] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0091] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0092] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0093] The above description is merely a specific embodiment or illustration of the present invention, and the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A streaming semantic understanding method, characterized in that, The method includes: The acquired voice data stream is subjected to real-time speech recognition to obtain a text data stream corresponding to the voice data stream; The text data stream is segmented in real time to obtain at least one text segment contained in the text data stream; and Semantic understanding is performed on the at least one text segment in real time to obtain the semantic understanding result of the at least one text segment; The method further includes: performing endpoint detection on the acquired speech data stream to obtain a first time interval between each speech segment contained in the speech data stream; The real-time segmentation of the text data stream includes: The text sequence of the text data stream is encoded in real time to obtain the semantic representation of the text sequence; The predicted pause probability of the text data stream is calculated based on the semantic representation of the text sequence; and The pause positions of the text data stream are determined based on the predicted pause probabilities, and the text data stream is then segmented at the pause positions. Determining the pause position of the text data stream based on the predicted pause probability includes: A second time interval is calculated based on the predicted pause probability, whereby the second time interval is the predicted waiting time interval; and The pause position of the text data stream is determined based on the first time interval and the second time interval.

2. The method as described in claim 1, characterized in that, The text sequence of the text data stream is encoded in real time to obtain a semantic representation of the text sequence, including: Calculate the semantic vector of the text sequence based on the text data stream; Calculate the attention weights of the semantic vector, and calculate the first contextual information vector based on the attention weights; and The semantic representation of the text sequence is calculated based on the first context information vector.

3. The method as described in claim 1, characterized in that, The semantic understanding result includes an intent understanding result, which includes intent text. Semantic understanding of the at least one text segment to obtain a semantic understanding result for the at least one text segment includes: Encode the sequence of at least one text segment to obtain a segment vector corresponding to the segment sequence; and The intent text is calculated using the fragment vectors based on an attention mechanism.

4. The method as described in claim 3, characterized in that, The intent understanding result further includes intent type, wherein semantic understanding of the at least one text segment to obtain a semantic understanding result of the at least one text segment further includes: Encode the word-mixed sequence of the intent text to obtain the intent vector corresponding to the intent text; and The intent type is obtained using the intent vector and the fragment vector based on the attention mechanism.

5. The method as described in claim 4, characterized in that, The semantic understanding result further includes slot value understanding result, wherein performing semantic understanding on the at least one text segment to obtain the semantic understanding result of the at least one text segment also includes: The second context information vector is calculated based on the attention mechanism using the intent vector and the fragment vector; The third context information vector is calculated using the fragment vector based on the attention mechanism; and The second context information vector, the third context information vector, and the fragment vector are concatenated to obtain a concatenated vector, and the slot value understanding result is calculated based on the concatenated vector.

6. A streaming semantic understanding device, characterized in that, The apparatus includes a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the streaming semantic understanding method as described in any one of claims 1-5.

7. A computer-readable medium, characterized in that, The computer-readable medium stores computer-executable instructions, which, when executed, perform the streaming semantic understanding method as described in any one of claims 1-5.

Images