A semantic-based echo judgment method for continuous human-computer dialogue

Abstract

The invention discloses a method for judging echo based on semantic level in a continuous human-computer dialogue, collecting reply requests and secondary requests, and obtaining integer sequence pairs and phoneme sequence pairs through data processing; after the phoneme sequence pairs are subjected to word embedding layer, LSTM encoding and L2 normalization, two vectors are obtained and then point multiplication is performed to obtain phoneme similarity; after the integer sequence pairs are encoded by word embedding layer, LSTM encoding, vector splicing and phoneme similarity splicing are performed to obtain a new feature vector, which is then passed through two Dense layers and activated by a softmax activation function to obtain an echo prediction result; if the phoneme similarity is greater than the phoneme similarity threshold and the echo prediction result is an echo, the secondary request is an echo and recognition is rejected. The invention judges echo at the semantic level, reduces the situation of voice confusion and misrecognition caused by echo, and improves user experience; at the same time, the threshold of phoneme similarity is strictly limited, and misrecognition of normal requests can also be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for judging echoes based on a semantic level in a continuous human-computer dialogue. Background Art

[0002] With the development of AI technology, more and more intelligent voice devices are used in shopping malls, homes, etc., to provide people with guidance, question-answering and other services. In the service-oriented human-computer dialogue function, the user makes a voice request to the machine, the machine collects voice information through the microphone, encodes it and transmits it to the processor for data processing, and then performs voice recognition, and then replies to this message through the speaker through subsequent semantic understanding. Generally, the above-mentioned voice request and response are carried out separately, but in full-duplex, while the speaker plays the voice reply, the microphone is not turned off. In this case, it is easy to re-collect the speaker's playback, which is mistakenly regarded as new voice information by the subsequent semantic processing module, causing voice confusion and misrecognition. Moreover, if the sound collection function is shielded during playback, firstly, due to the different lengths of the answering voice, it is impossible to set a suitable shielding time, and secondly, the user's secondary real-time voice information and instructions cannot be collected during playback, which greatly reduces the user's experience and satisfaction. Summary of the invention

[0003] The purpose of the present invention is to provide a method for judging echo based on semantic level in continuous human-computer dialogue, which is used to solve the problem in the prior art that the sound played by the speaker is collected by the microphone and mistakenly regarded as a new voice request, resulting in misrecognition of voice.

[0004] The present invention solves the above problems through the following technical solutions:

[0005] A method for judging echo based on semantic level in human-computer continuous dialogue, comprising:

[0006] Step S100, collecting the reply request and the secondary request with a time interval with respect to the reply request within a set time, and performing two types of processing on the collected two voice data after data processing: 1) directly performing sequence conversion to obtain an integer sequence pair; 2) performing pinyin conversion, phoneme conversion, and then performing sequence conversion to obtain a phoneme sequence pair;

[0007] Step S200: After the phoneme sequence pair is subjected to word embedding layer, LSTM encoding and L2 normalization, two vectors are obtained, and the two vectors are point-multiplied to obtain the phoneme similarity;

[0008] After the integer sequence pair is encoded by the word embedding layer and LSTM encoding, two sentence vectors are obtained. The two sentence vectors are integrated by vector concatenation. Then, the phoneme similarity is concatenated with the two sentence vectors through a fully connected layer to obtain a new feature vector. Then, the feature vector is passed through two Dense layers and finally activated by a softmax activation function to obtain an echo prediction result.

[0009] Step S300: compare the phoneme similarity with the set similarity threshold. If the phoneme similarity is greater than the similarity threshold and the echo prediction result is an echo, the secondary request is determined to be the echo of the reply request and recognition is rejected. Otherwise, the secondary request is recognized normally.

[0010] When the integer sequence pair is encoded through the word embedding layer, a mask is added: during forward propagation, the 0s in the integer sequence pair are directly masked; during backward propagation, a meaningful inversion is first performed and then the 0s in the integer sequence pair are masked, and the padding 0 remains unchanged.

[0011] The two collected voice data are processed to remove characters and punctuation marks other than numbers, Chinese and English in the reason data.

[0012] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0013] The present invention determines echoes at the semantic level, based on the phoneme similarity of two sentences and the analysis and classification of their semantics, and the two mutually correct the results and judgments. By not making subsequent voice replies to the results determined as echoes, voice confusion and misrecognition caused by echoes are reduced, and the user experience in continuous conversations is improved; at the same time, the threshold of phoneme similarity is strictly limited, which can also reduce misrecognition of normal requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0015] The present invention is further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0016] Embodiment 1:

[0017] Combined with Figure 1 As shown, a method for judging echo based on semantic level in human-computer continuous dialogue includes:

[0018] 1. Collect and organize echo data, analyze data features, and obtain training corpus

[0019] Filter the data recognized by echo from the user data of the existing continuous conversation to form a sentence pair (the previous round of reply and the current round of request), and analyze the data features. Here, the time difference between the two requests is not considered, and the features of the echo data are analyzed from the text. Check the basic features of the echo data from the data, that is, the pronunciation of this request is highly similar to that of the previous round of reply. Then remove the data that is not echo from the data with similar pronunciation, that is, the relationship between the previous round of reply and the current round of request is "synonymous sentence" or "dialogue", "opposite", etc., such as:

[0020] a. To follow up the previous sentence or answer: "Call me brother" - "Brother", "Do you like me?" - "I like you"

[0021] b. Expresses the opposite meaning of the above sentence: "He is not stupid" - "He is stupid"

[0022] c. Synonymous sentences: "Help me open this" - "Help me open this"

[0023] d. Other conversation situations: “I don’t like this” — “I don’t like this either”

[0024] 2. Data preprocessing

[0025] The sentence pairs are processed uniformly (full-width characters are converted to half-width characters, lowercase characters are converted to uppercase characters, punctuation marks and special characters other than Chinese characters, English characters and numbers are removed), and then the sentence pairs are converted into phonemes (based on the International Phonetic Alphabet). At the same time, the sentences themselves and their phonemes are sequenced. One sentence has two positive number sequences - the sequence composed of phonemes is the phoneme sequence pair, and the sequence composed of characters is the integer sequence pair.

[0026] 3. Build the model, including the phoneme similarity part and the semantic analysis part, input the training corpus, and train the model

[0027] The model consists of two parts. The first part is the judgment of phoneme similarity, which determines whether the pronunciation is similar. After the input phoneme sequence pairs pass through the embedding layer (word embedding), LSTM, L2 normalization, and dot product (Dot), the similarity between them is output. At the same time, this part of the similarity is used as a part of the feature to correct the parsing results of the semantic relevance part; the second part is the analysis of semantic relevance. After the sentence sequence pairs pass through the embedding layer, LSTM, and the fully connected layer, a semantic relevance feature vector is obtained. After this vector is concatenated with the result of phoneme similarity, it passes through the fully connected layer and outputs the final prediction result. The output results of the two parts are combined. If the phoneme similarity is high and the correlation between the sentence pairs is not strong, it is considered to be an echo positive sample.

[0028] For example:

[0029] Phoneme similarity part:

[0030] For a phoneme sequence pair: A = [a0, a1, a2…a n ],B=[b0,b1,b2…b n ], n is the limited sequence length. After the first embedding layer, LSTM encoding and L2 normalization, two vectors can be obtained to represent the phonemes of the sentence:

[0031] X A =(x a0 , x a1 , x a2 …x am ), X B =(x b0 , x b1 , x b2 …x bm )

[0032] And because it has been L2 normalized, we have X A ·X B =cos <X A , X B >, since the subsequent labels are binary, similarity is 1 and dissimilarity is 0. Therefore, after multiplying the two vectors, they are amplified by a set threshold during training:

[0033]

[0034] in, The similarity threshold is set (0.85 in the implementation of the present invention), and finally a phoneme similarity result is obtained.

[0035] Semantic relevance section:

[0036] For integer sequence pairs: A′=[a′0,a′1,q′2…a′ n ], B′=[b′0, b′1, b′2…b′ n ], n is the limited sequence length. Similarly, after the embedding layer and LSTM encoding, two sentence vectors are obtained, and mask masking is added during embedding. The forward pass directly masks 0, and the backward propagation needs to be processed before masking, and the meaningful part is reversed and filled with 0. For example, "I want to watch a movie" - [1 2 3 4 5 0 0 0 0 0], when masking in the backward propagation, the sequence should be changed to [5 4 3 2 10 0 0 0 0] instead of [0 0 0 0 0 5 4 3 2 1], to ensure that 0 represents specific semantic features during bidirectional propagation training:

[0037] X′ A =(x′a0 , x′ a1 , x′ a2 …x′ am ), X′ B =(x′ b0 , x′ b1 , x′ b2 …x′ bm )

[0038] Then, we integrate the two sentence vectors by concatenating the vectors:

[0039] X=(x′ a0 , x′ a1 , x′ a2 …x′ am , x′ b0 , x′ b1 , x′ b2 …x′ bm )

[0040] The dimension is 1×2m, and then the subsequent fully connected layer is used to obtain the preliminary classification result. The judgment result of phoneme similarity is used as a subsequent feature, and it is concatenated with the vector output by the fully connected layer to obtain a new feature vector. This feature vector is then passed through two Dense layers and finally activated by the softmax activation function to obtain the final echo prediction result. The classification model includes a mask layer, which is used to cover and fill meaningless integers 0; at the same time, after the output of the sentence pair through the fully connected layer is integrated, the feature input of phoneme similarity is added to correct the result; the final echo judgment result is judged based on multiple conditions (request time difference, pronunciation similarity, relevance, etc.).

[0041] 4. Two request time limit

[0042] In the prediction part, since there is no hardware sound collection data, it is necessary to limit the time between the two requests. When the machine echo is generated, the time when the sound receiving device obtains the audio should not be much different from the time when the speaker sends a response. This time difference, minus the network transmission delay and the cloud semantic analysis time, is basically equal to the time difference between the two requests obtained by the cloud. Through data analysis, the network transmission delay and the cloud interpretation time of a single request are added, about 600ms, as the time difference. If this time difference is within the set threshold, the next step of echo judgment is performed. If not, it is considered not to be an echo.

[0043] 5. Use the trained model for echo judgment

[0044] Step S100: collect the reply request and the secondary request with a time interval within a set time (such as 600ms) with respect to the reply request, and perform two types of processing after data processing on the two collected voice data:

[0045] 1) Directly perform sequence conversion to obtain integer sequence pairs; for example, "I want to watch a movie" is converted into an integer sequence based on the words, such as: [1,2,3,4,5,0,0,0,0,0].

[0046] 2) Perform pinyin conversion (for example, this can be achieved through the pypinyin package. For some common polyphonetic characters, their pronunciations are marked in the form of a dictionary), and then convert the pinyin into phonemes one by one according to the international phoneme comparison table (segment the pinyin according to phonemes). For example, "I want to watch a movie" is converted to: "uo2 x iang3 k an4 dian4 ing3", and then according to the phonemes, it is converted into a phoneme sequence, such as: [1,2,3,4,5,6,7,8,0,0].

[0047] Step S200, the phoneme similarity judgment part of the phoneme sequence input model, after the word embedding layer, LSTM encoding and L2 normalization, two vectors are obtained, and the two vectors are point-multiplied to obtain the phoneme similarity;

[0048] The semantic relevance analysis part of the integer sequence to the input model is encoded by the word embedding layer and LSTM encoding to obtain two sentence vectors, which are integrated by vector concatenation. Then, the phoneme similarity is concatenated with the two sentence vectors through the fully connected layer to obtain a new feature vector. This feature vector is then passed through two Dense layers and finally activated by the softmax activation function to obtain the echo prediction result.

[0049] Step S300: compare the phoneme similarity with the set similarity threshold. If the phoneme similarity is greater than the similarity threshold and the echo prediction result is an echo, the secondary request is determined to be the echo of the reply request and recognition is rejected. Otherwise, the secondary request is recognized normally.

[0050] Although the present invention is described herein with reference to the illustrative embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A method for judging echo based on semantic level in continuous human-computer dialogue, characterized in that: include: Step S100, collect the reply request and the secondary request with a time interval with the reply request within a set time, and perform two types of processing on the collected two voice data after data processing: 1) directly perform sequence conversion to obtain an integer sequence pair; 2) After performing pinyin conversion and phoneme conversion, sequence conversion is performed to obtain a phoneme sequence pair; Step S200, constructing a model, including a phoneme similarity part and a semantic analysis part, inputting training corpus, and training the model; The model consists of two parts. The first part is to judge whether the pronunciation is similar. After the input phoneme sequence pairs pass through the embedding layer, LSTM, L2 normalization, and dot product, the similarity between them is output. At the same time, this part of the similarity is used as a part of the feature to correct the parsing results of the semantic relevance part. The second part is the analysis of semantic relevance. After the sentence sequence pairs are concatenated through the embedding layer, LSTM, and fully connected layer, and then linearly transformed, a semantic relevance feature vector is obtained. This vector is concatenated with the result of phoneme similarity and then passed through the fully connected layer to output the final prediction result. The output results of the two parts are combined. If the phoneme similarity is high and the correlation between the sentence pairs is not strong, it is considered to be an echo positive sample. Phoneme similarity part: For a phoneme sequence pair: A = [a0, a1, a2…a N ],B=[b0,b1,b2…b n ], n is the limited sequence length. After the first embedding layer, LSTM encoding and L2 normalization, two vectors are obtained to represent the phonemes of the sentence: X A =(x a0 ,x a1 ,x a2 …x am ),X B =(x b0 ,x b1 ,x b2 …x bm ) And because it has been L2 normalized, we have X A ·X B =cos <X A , X B >, since the subsequent labels are binary, similarity is 1 and dissimilarity is 0. Therefore, after multiplying the two vectors, they are amplified by a set threshold during training: in, The similarity threshold is set, and finally a phoneme similarity result is obtained; Semantic relevance section: For integer sequence pairs: A′=[a′0,a′1,a′2…a′ n ], B′=[b′0, b′1, b′2…b′ n ], n is the limited sequence length. Similarly, after the embedding layer and LSTM encoding, two sentence vectors are obtained, and masking is added during embedding. The forward pass directly masks 0, and the backward propagation needs to be processed before masking, and the meaningful inversion is filled with 0 unchanged to ensure that 0 represents specific semantic features during bidirectional propagation training: X′ A =(x′ a0 ,x′ a1 ,x′ a2 …x′ am ),X′ B =(x′ b0 ,x′ b1 ,x′ b2 …x′ bm ) Then, we integrate the two sentence vectors by concatenating the vectors: X=(x′ a0 ,x′ a1 ,x′ a2 …x′ am ,x′ b0 ,x′ b1 ,x′ b2 …x′ bm ) The dimension is 1×2m. Then, it passes through the subsequent fully connected layer to obtain the preliminary classification result. Then, the judgment result of the phoneme similarity is used as a subsequent feature and concatenated with the vector output by the fully connected layer to obtain a new feature vector. Then, this feature vector passes through two Dense layers and is finally activated by the softmax activation function to obtain the final echo prediction result. After the phoneme sequence pair passes through the word embedding layer, LSTM encoding and L2 normalization, two vectors are obtained. The point multiplication of the two vectors is used to obtain the phoneme similarity; After the integer sequence pair is encoded by the word embedding layer and LSTM encoding, two sentence vectors are obtained. The two sentence vectors are integrated by vector concatenation. Then, the phoneme similarity is concatenated with the two sentence vectors through a fully connected layer to obtain a new feature vector. Then, the feature vector is passed through two Dense layers and finally activated by a softmax activation function to obtain an echo prediction result. Step S300: compare the phoneme similarity with the set similarity threshold. If the phoneme similarity is greater than the similarity threshold and the echo prediction result is an echo, the secondary request is determined to be the echo of the reply request and recognition is rejected. Otherwise, the secondary request is recognized normally.

2. The method for judging echo based on semantic level in a continuous human-computer dialogue according to claim 1, characterized in that: When the integer sequence pair is encoded through the word embedding layer, a mask is added: during forward propagation, the 0s in the integer sequence pair are directly masked; during backward propagation, a meaningful inversion is first performed and then the 0s in the integer sequence pair are masked, and the padding 0 remains unchanged.

3. The method for judging echo based on semantic level in a continuous human-computer dialogue according to claim 1, characterized in that: The two collected voice data are processed to remove characters and punctuation marks other than numbers, Chinese and English in the reason data.

Images