End-of-talk detection system and device based on semantic integrity and medium

Abstract

The invention relates to the field of semantic detection algorithms, and discloses an end-of-speech detection system and device based on semantic integrity, and a medium, and the system comprises a voice collection module which is used for collecting a human voice signal in real time; the voice recognition and conversion module is used for extracting a dialogue text in the human voice signal by using an audio recognition method; the semantic detection module is used for processing the dialogue text by adopting a semantic integrity detection model based on large language model optimization to obtain a semantic integrity signal; the decision logic module is used for judging whether the speaking is ended or not according to the human voice signal and the semantic integrity signal, and if the semantic integrity signal is True and the human voice signal disappears for a time exceeding a preset time, determining that the speaking is ended; the method has the beneficial effects that the response accuracy can be improved, the user experience is enhanced, in addition, the detection precision of the end of speaking is improved, and the application scene is wide.

Description

Technical Field

[0001] The present invention relates to the field of semantic detection algorithms, and in particular to a system, device and medium for detecting the end of speech based on semantic integrity. Background Art

[0002] In the field of human-computer interaction, communication is not limited to the transmission of information; it is a complex process of emotional cognition. Everyone has their own unique communication patterns and rhythms. These individual characteristics are reflected not only in intonation, speaking speed, and pauses, but also in vocabulary and sentence structure. This individuality is particularly pronounced for people with mood disorders, who may exhibit more frequent and longer pauses when expressing themselves and are particularly sensitive to interruptions, which can disrupt their flow of thought, trigger unease, or exacerbate anxiety.

[0003] When designing intelligent human-machine dialogue systems, it's crucial to understand and adapt to users' unique communication patterns and rhythms. One of the core challenges is accurately identifying when a user finishes speaking, ensuring the machine responds appropriately.

[0004] Although voice activity detection technology can effectively determine the start and end time of speech by identifying the presence of sound, for patients with depression or anxiety, since they may include long pauses in their speech, traditional voice activity detection algorithms can easily misjudge the end of speech, resulting in inappropriate interruptions and disrupting the user's smooth experience.

[0005] An ideal human-computer dialogue system should be able to understand and adapt to individual differences. This means the machine needs to intelligently sense when the user has completed their thought, even if the speech includes brief pauses or thoughtful silences. Achieving this requires the system to rely not only on the presence or absence of sound but also on a comprehensive assessment of the context. Summary of the Invention

[0006] The purpose of the present invention is to propose a system, device and medium for detecting the end of speech based on semantic integrity, so as to solve the technical problem that the existing end of speech detection method only relies on the unilateral consideration of pause duration, resulting in inaccurate detection.

[0007] Specifically, the present invention provides a system for detecting the end of speech based on semantic integrity, comprising the following: Voice acquisition module, used to collect human voice signals in real time; Speech recognition and conversion module, used to extract the conversation text from the human voice signal using audio recognition method; The semantic detection module is used to process the conversation text using a semantic integrity detection model optimized based on a large language model to obtain a semantic integrity signal; The decision logic module is used to determine whether the speech has ended based on the human voice signal and the semantic integrity signal. If the semantic integrity signal is True and the human voice signal has disappeared for more than a preset time, the speech is deemed to have ended.

[0008] A storage medium stores instructions and data for implementing a speech end detection system based on semantic integrity.

[0009] A device for detecting the end of speech based on semantic integrity comprises: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a system for detecting the end of speech based on semantic integrity.

[0010] The beneficial effects provided by the present invention are: 1. Improved response accuracy: By combining semantic completeness detection with traditional voice activity detection, this method can more accurately determine the end of speech. This is particularly true for users with long pauses during speech (such as those with mood disorders). This method avoids inappropriate interruptions caused by traditional methods, resulting in a more fluid and natural conversation experience.

[0011] 2. Enhanced User Experience: This invention specifically addresses the needs of people with emotional disorders. By understanding and adapting to their unique communication patterns and rhythms, it improves their comfort and satisfaction with intelligent human-computer dialogue systems. This not only supports technological advancement but also promotes a more inclusive design philosophy.

[0012] 3. Technological Innovation: The use of a speech semantics detection model based on a large language model is an innovative extension of existing voice activity detection technology. This demonstrates progress in understanding the complexity and diversity of human language, especially when processing emotionally charged language.

[0013] 4. Wide range of applications: Since this method is not only applicable to specific groups of people with emotional disorders, it can also be widely used in various scenarios that require high-precision end-of-speech detection, such as customer service robots and virtual assistants, improving the practicality and user experience of these applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram of the system structure of the present invention; Figure 2 It is a schematic structural diagram of the EOUDp model and the EOUDb model of the present invention; Figure 3 It is a systematic logical judgment process; Figure 4 The following figure shows the calculation process of the semantic integrity detection model in the present invention; Figure 5 It is a schematic diagram of the working of the hardware device of an embodiment of the present invention. DETAILED DESCRIPTION

[0015] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0016] Before formally explaining the present invention, the scheme of the present invention is first generally explained for easy understanding.

[0017] Please refer to Figure 1 The present invention provides a method for detecting the end of speech based on semantic integrity, comprising the following steps: Voice acquisition module, real-time acquisition of human voice signals; It should be noted that the present invention continuously collects audio data in the environment through a microphone.

[0018] It should be noted that the present invention uses voiceprint detection technology to detect the speaker's voice signal. If there is someone speaking in the audio, a voice presence signal is issued; if no one is speaking, a voice disappearance signal is issued.

[0019] As a more optimal solution, the voice collection module of the present invention also includes a timeout mechanism. If the time for the human voice disappearance signal exceeds the threshold, it is considered to have timed out and the speech end judgment is directly output. Otherwise, continuous monitoring continues and the voice recognition and conversion module is entered.

[0020] Speech recognition and conversion module, used to extract the conversation text from the human voice signal using audio recognition method; It should be noted that the audio recognition method described in the speech recognition and conversion module is as follows: Preprocessing the human voice signal to obtain a processed signal; preprocessing includes but is not limited to operations such as noise reduction and normalization to improve the quality of subsequent processing; The convolutional neural network (CNN) module is used to extract features from the processed signal to obtain feature vectors; these feature vectors capture the key local features in the audio signal and provide a basis for subsequent voice activity detection and semantic integrity analysis.

[0021] The feature vector is input into a pre-trained model based on FunASR to obtain the conversation text. It should be noted that the pre-trained model provided by FunASR is fine-tuned on a dataset of psychotherapy conversations. This dataset contains a large number of representative audio samples and their corresponding annotation information, with a particular focus on the speech characteristics of people with mood disorders. This approach enables the model to better adapt to the needs of the target application scenario and significantly improves the recognition accuracy for specific user groups. Since the audio is a streaming file, the extracted text is also streaming.

[0022] The semantic detection module is used to process the conversation text using a semantic integrity detection model optimized based on a large language model to obtain a semantic integrity signal; It should be noted that the semantic integrity detection model based on large oracle model optimization described in the semantic detection module adopts the EOUDp model or the EOUDb model.

[0023] Please refer to Figure 2 , Figure 2 It is a schematic structural diagram of the EOUDp model and the EOUDb model of the present invention; As an example, the EOUDp model adopts a task transfer learning strategy, adds a lora bypass to each multi-head attention layer, gating layer, and MLP in the large language model, and uses a stop token to limit the output sequence length of the base model to 1.

[0024] During the training process, by constructing prompts in a specific format, the large language model can learn semantically complete features to achieve judgment tasks.

[0025] Specifically, for each input instance, this application integrates the conversation text and label information into a structured prompt string with the structure [input format: spoken text], [output format: True / False], and uses these examples to fine-tune the base model. The result is a semantic integrity detection model that inputs conversation text and outputs True / False.

[0026] In addition, since the output dimension of the inference head is V and the output sequence length is limited to 1, the loss function during training is

[0027] in is the output token in the vocabulary, is the probability of predicting that token.

[0028] On the other hand, if the EOUDb model is used, in addition to adding the basic changes of lora bypass to each multi-head attention layer, gating layer and MLP in the base model for the supervised contrastive learning strategy, a linear layer is used as the classification head instead of the inference output head of the base model.

[0029] During the training process, the semantically complete conversation text is recorded as a positive sample, and its label is represented by 0, and the semantically incomplete conversation text is recorded as a negative sample, and is represented by 1. The model is fine-tuned using positive and negative samples so that it can efficiently complete the binary classification task. Since the output structure is a two-dimensional linear layer, the label , and its loss function is,

[0030] Where P is the probability that the spoken text is semantically complete.

[0031] The specific process of obtaining the semantic integrity signal in the semantic detection module is as follows: Get the content of the spoken text at time t S t ; Convert the spoken text into a token sequence through word segmentation embedding algorithm TS t ; Sequence the tokens TS t Input to the semantic integrity detection model D ( x ), and obtain the semantic integrity signal.

[0032] The decision logic module is used to determine whether the speech has ended based on the human voice signal and the semantic integrity signal. If the semantic integrity signal is True and the human voice signal has disappeared for more than a preset time, the speech is deemed to have ended.

[0033] The logical judgment process of the above system is as follows Figure 3 shown.

[0034] The microphone continuously collects audio data from the environment. The audio detection module analyzes these data streams in real time. Once a human voice is detected, it continuously broadcasts a voice presence signal to the buffer, speech recognition module, and signal gate. Once the buffer receives the presence signal, the audio stream is stored in the buffer through a pipeline.

[0035] When the speaker pauses, the audio detection module broadcasts a voice disappearance signal, the buffer marks the audio with a pause mark, and the speech recognition module recognizes the audio from the beginning to the pause mark as a conversation text and sends it to the semantic integrity detection module.

[0036] The semantic detection module judges the conversation text and sends a signal to the audio detection module to continue waiting when the semantics are incomplete.

[0037] The audio detection module will continue to detect until it times out. If a human voice is detected, it will continue to broadcast the human voice presence signal. If no human voice is detected after the timeout, it will directly determine that the speech has ended. When the semantic detection module determines that the spoken text is semantically complete, it sends a semantic integrity signal to the signal gate.

[0038] Due to the latency between speech recognition and semantic detection, there's a time gap between when the signal gate receives the voice-off signal and when it receives the semantic-complete signal. If the signal gate receives a voice-on signal during this time, it discards the semantic-complete signal and continues detecting the spoken audio. During this time gap, the signal gate only receives the voice-off signal. Upon receiving the semantic-complete signal, it determines the speaker has finished speaking.

[0039] Please refer to Figure 4 , Figure 4 The following shows the calculation process of the semantic integrity detection model in the present invention.

[0040] The spoken text is input into the model with pauses as the rhythm for judgment and classification. False means incomplete and a signal to continue waiting is issued; True means semantically complete and a signal of semantic completeness is issued.

[0041] set up: l ( t ) represents the length of the spoken text at time t.

[0042] S t ,express t At this moment, the content of the spoken text.

[0043] :It is a word segmentation embedding algorithm that converts the spoken text S t Convert to a sequence of tokens TS t .

[0044] D ( x ) is a semantic detection model that accepts the token sequence at time t TS eq , and output True or False.

[0045] T It indicates the time point when the semantic detection model outputs True for the first time, which can be used as a semantic completeness signal.

[0046] During the speaking process, the string length increases with time, i.e. for all , , the spoken text at this time is recorded as .

[0047] Convert the conversation text into a token sequence through word segmentation and embedding methods TS t : TS t = TSeq ( S t ) The semantic detection model first determines whether the current text sequence is semantically complete. If it is incomplete, it calculates the text at the next moment, outputs a waiting signal, and cycles in sequence.

[0048] If the text is semantically complete, stop the loop , output semantically complete signal T.

[0049] See Figure 5 , Figure 5 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a speech end detection device 401 based on semantic integrity, a processor 402 and a storage medium 403.

[0050] An end-of-speech detection device 401 based on semantic integrity: The end-of-speech detection device 401 based on semantic integrity implements the end-of-speech detection system based on semantic integrity.

[0051] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the end-of-speech detection system based on semantic integrity.

[0052] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the end-of-speech detection system based on semantic integrity.

[0053] The beneficial effects of the present invention are: 1. Improved response accuracy: By combining semantic completeness detection with traditional voice activity detection, this method can more accurately determine the end of speech. This is particularly true for users with long pauses during speech (such as those with mood disorders). This method avoids inappropriate interruptions caused by traditional methods, resulting in a more fluid and natural conversation experience.

[0054] 2. Enhanced User Experience: This invention specifically addresses the needs of people with emotional disorders. By understanding and adapting to their unique communication patterns and rhythms, it improves their comfort and satisfaction with intelligent human-computer dialogue systems. This not only supports technological advancement but also promotes a more inclusive design philosophy.

[0055] 3. Technological Innovation: The use of a speech semantics detection model based on a large language model is an innovative extension of existing voice activity detection technology. This demonstrates progress in understanding the complexity and diversity of human language, especially when processing emotionally charged language.

[0056] 4. Wide range of applications: Since this method is not only applicable to specific groups of people with emotional disorders, it can also be widely used in various scenarios that require high-precision end-of-speech detection, such as customer service robots and virtual assistants, improving the practicality and user experience of these applications.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A system for detecting the end of utterance based on semantic integrity, characterized by: The following steps are involved: Voice acquisition module, used to collect human voice signals in real time; Speech recognition and conversion module, used to extract the conversation text from the human voice signal using audio recognition method; The semantic detection module is used to process the conversation text using a semantic integrity detection model optimized based on a large language model to obtain a semantic integrity signal; The decision logic module is used to determine whether the speech has ended based on the human voice signal and the semantic integrity signal. If the semantic integrity signal is True and the human voice signal has disappeared for more than a preset time, the speech is deemed to have ended.

2. The system for detecting the end of utterance based on semantic integrity according to claim 1, wherein: In the voice acquisition module, when collecting human voice signals, the pause length is set. If the pause length exceeds the preset time, it is directly considered that the speech has ended. Otherwise, it enters the voice recognition and conversion module.

3. The method for detecting the end of utterance based on semantic integrity according to claim 1, wherein: The audio recognition method described in the speech recognition and conversion module is as follows: The convolutional neural network (CNN) module is used to extract features from the human voice signal and obtain feature vectors. The feature vector is input into the pre-trained model based on FunASR to obtain the conversation text.

4. The method for detecting the end of utterance based on semantic integrity according to claim 1, wherein: The semantic integrity detection model based on large prediction model optimization described in the semantic detection module adopts the EOUDp model or the EOUDb model.

5. The method for detecting the end of utterance based on semantic integrity according to claim 4, wherein: The EOUDp model adopts a task transfer learning strategy, adds a Lora bypass to each multi-head attention layer, gating layer and MLP in the large language model, and uses a stop token to limit the output sequence length of the base model to 1.

6. The method for detecting the end of utterance based on semantic integrity according to claim 4, wherein: The EOUDb model described in the semantic detection module adopts a supervised contrastive learning strategy. In addition to adding basic changes such as LoRa bypass to each multi-head attention layer, gating layer, and MLP in the large language model, a linear layer is used as the classification head instead of the inference output head of the base model.

7. The method for detecting the end of utterance based on semantic integrity according to claim 1, wherein: The specific process of obtaining the semantic integrity signal in the semantic detection module is as follows: Get t The content of the moment speech text S t ; Convert the spoken text into a token sequence through word segmentation embedding algorithm TS t ; Input the token sequence TSt into the semantic integrity detection model D ( x ), and obtain the semantic integrity signal.

8. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the end-of-speech detection system based on semantic integrity as described in any one of claims 1 to 7.

9. A device for detecting the end of speech based on semantic integrity, characterized by: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a speech end detection system based on semantic integrity as described in any one of claims 1 to 7.

Images