Audio data processing method, human-computer interaction method, device and storage medium

Abstract

Embodiments of the present application provide an audio data processing method, a human-computer interaction method, a device and a storage medium. The method comprises: acquiring a first audio segment, and determining whether the first audio segment and a second audio segment constitute complete semantics according to semantic features of the first audio segment, the second audio segment being collected before the first audio segment. If the first audio segment and the second audio segment constitute complete semantics, the first audio segment and the second audio segment are determined as to-be-processed audio. Finally, the to-be-processed audio is responded. In the above method, since the semantic features of the first audio segment can be obtained after the first audio segment is acquired, it can be immediately determined whether the user has produced audio with complete semantics by using the semantic features, that is, the time required to determine complete semantics is shortened, and then the processing device can immediately respond to the audio with complete semantics, thereby reducing the time delay of human-computer dialogue.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an audio data processing method, a human-computer interaction method, a device, and a storage medium. Background Technology

[0002] A common use case for artificial intelligence technology is human-computer dialogue. For example, dialogue systems can be used to engage in voice conversations with users. These dialogue systems can be integrated into hardware interaction devices, such as smart speakers and mobile terminals. Dialogue systems can also be deployed in the cloud as online intelligent customer service to meet users' needs for information inquiries and feedback.

[0003] During a voice conversation, the dialogue system can collect audio data generated by the user in real time. After determining that the audio data has complete semantics, it further performs semantic recognition on the audio data and then responds based on the semantics, thus completing a round of dialogue. The time between the dialogue system collecting the audio data and the dialogue system responding to the audio data is the latency of the human-computer dialogue.

[0004] Latency in human-computer interaction can obviously have a significant impact on the user experience. Therefore, reducing latency in human-computer interaction has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an audio data processing method, a human-computer interaction method, a device, and a storage medium to reduce the latency of human-computer dialogue and improve the fluency of human-computer dialogue.

[0006] In a first aspect, embodiments of the present invention provide an audio data processing method, comprising:

[0007] Get the first audio segment;

[0008] Based on the semantic features of the first audio segment, it is determined whether the first audio segment and the second audio segment constitute a complete semantic unit, and the second audio segment is collected before the first audio segment;

[0009] If the first audio segment and the second audio segment constitute a complete semantic unit, then the first audio segment and the second audio segment are determined as audio to be processed;

[0010] Respond to the audio to be processed.

[0011] Secondly, embodiments of the present invention provide an audio data processing method, including:

[0012] Obtain the semantic features of the training audio segment and the reference type of the training audio segment, wherein the training audio segment is any audio segment in the training audio;

[0013] The semantic features of the training audio segment are used as training data, and the reference type is used as supervision information to train the classification model. The classification result output by the classification model is used to determine whether the training audio segment and the audio segment played before the training audio segment constitute complete semantics.

[0014] Loss is calculated for the predicted type and the reference type of the training audio segment output by the classification model;

[0015] The classification model is optimized based on the loss calculation results.

[0016] Thirdly, embodiments of the present invention provide a human-computer interaction method, including:

[0017] In response to user input, capture the first audio segment generated by the user;

[0018] The output is a response result of the audio to be processed with complete semantics, wherein whether the semantics of the audio to be processed is complete is determined according to the semantic features of the first audio segment, and the audio to be processed includes the first audio segment and a second audio segment acquired before the first audio segment.

[0019] Fourthly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the audio data processing method in the first or second aspect described above, or the human-computer interaction method in the third aspect described above. The electronic device may also include a communication interface for communicating with other devices or communication systems.

[0020] Fifthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, wherein when the executable code is executed by a processor of an electronic device, the processor is able to implement at least the audio data processing method as described in the first or second aspect above, or the human-computer interaction method as described in the third aspect above.

[0021] The audio data processing method provided in this embodiment of the invention can acquire a first audio segment and determine whether the first audio segment and a second audio segment constitute a complete semantic unit based on the semantic features of the first audio segment. The second audio segment is acquired before the first audio segment. If the first and second audio segments constitute a complete semantic unit, then both audio segments are identified as audio to be processed. Finally, a response is generated for the audio to be processed.

[0022] In the above method, since the semantic features of the first audio segment can be obtained immediately, it is possible to determine whether the user has produced semantically complete audio, thus shortening the time required to determine semantic completeness. The processing device can then immediately respond to the complete audio, thereby reducing the latency of human-computer dialogue. On the other hand, compared to acoustic features, semantic features contain richer semantic information. Therefore, using semantic features can obviously more accurately identify whether the user has produced semantically complete audio and can respond to it, thereby improving the fluency of human-computer dialogue. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart of an audio data processing method provided in an embodiment of the present invention;

[0025] Figure 2 A flowchart of another audio data processing method provided in an embodiment of the present invention;

[0026] Figure 3 A flowchart of a model training method provided in an embodiment of the present invention;

[0027] Figure 4 A flowchart of another audio data processing method provided in an embodiment of the present invention;

[0028] Figure 5 A flowchart of another model training method provided in an embodiment of the present invention;

[0029] Figure 6 A flowchart illustrating yet another model training method provided in this embodiment of the invention;

[0030] Figure 7 A schematic diagram of the structure of the jointly trained model provided in an embodiment of the present invention;

[0031] Figure 8 A flowchart of another audio data processing method provided in an embodiment of the present invention;

[0032] Figure 9 A flowchart of a human-computer interaction method provided in an embodiment of the present invention;

[0033] Figure 10 This is a structural diagram of a human-computer interaction scenario provided in an embodiment of the present invention;

[0034] Figure 11 This is a schematic diagram of the user interface provided in an embodiment of the present invention;

[0035] Figure 12 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of the present invention;

[0036] Figure 13 This is a schematic diagram of another audio data processing device provided in an embodiment of the present invention;

[0037] Figure 14 This is a schematic diagram of the structure of a human-computer interaction device provided in an embodiment of the present invention;

[0038] Figure 15 A schematic diagram of the structure of an electronic device provided in the embodiments of the invention;

[0039] Figure 16 A schematic diagram of the structure of another electronic device provided in the embodiments of the invention;

[0040] Figure 17 A schematic diagram of the structure of another electronic device provided in the embodiments of the invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0043] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0044] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to identification.” Similarly, depending on the context, the phrases “if determination” or “if identification (of the condition or event of the statement)” can be interpreted as “when determination” or “in response to determination” or “when identification (of the condition or event of the statement)” or “in response to identification (of the condition or event of the statement).”

[0045] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0046] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0047] Before describing in detail the audio data processing method and human-computer interaction method provided in the following embodiments of the present invention, the relevant concepts involved in the following embodiments can be explained first:

[0048] Voice activity detection (VAD), also known as voice endpoint detection, is a front-end module used to process long audio files. It uses acoustic features to distinguish between silent and spoken audio segments. The classification results can be used for subsequent downstream tasks, such as speech recognition (converting audio to text) and sensitive word detection.

[0049] Audio segment: This refers to the result of segmenting audio into frames; one frame of audio constitutes one audio segment. The length of an audio segment can be 10ms. Audio segments can include semantic speech segments as well as silent audio segments without semantic meaning.

[0050] Acoustic features: Acoustic features refer to the acoustic characteristics of various speech audio signals. These features are generated by acoustic effects during the generation, transmission, and reception of speech signals. Furthermore, since acoustic features are obtained by performing a Fourier transform on an audio segment, they can also be understood as the low-level features of an audio segment.

[0051] Semantic features: Semantic features can be textual information obtained by feature extraction from audio segments. In addition, they can include other information such as pauses, volume, pitch, and audio frequency. In other words, the semantic features of an audio segment contain rich information and can be understood as the high-level features of the audio segment.

[0052] Before describing in detail the audio data processing method and human-computer interaction method provided in the following embodiments of the present invention, the application scenarios and significance of the methods provided by the present invention can be explained first:

[0053] In human-computer dialogue scenarios, intelligent robots such as service robots, greeting robots, and self-propelled vending robots can integrate dialogue systems. In addition, intelligent terminals such as mobile terminals, smart home appliances, and smart wearable devices can also integrate dialogue systems. All of these devices can be considered hardware interaction devices. Furthermore, dialogue systems can be deployed in the cloud to provide users with services such as online consultation and customer service follow-up. Generally speaking, any dialogue system that supports human-computer dialogue with users via voice, whether deployed on hardware interaction devices or in the cloud, can receive and respond to audio data generated by users.

[0054] In human-computer dialogue scenarios, the methods mentioned in the following embodiments of the present invention can be used to reduce latency and ensure smoothness of dialogue. Additionally, in scenarios requiring speech recognition of long speech segments, such as meeting minutes, real-time captions, and voice notes, the methods mentioned in the following embodiments of the present invention can also be used to ensure the accuracy of speech recognition.

[0055] Based on the above description, some embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0056] Figure 1This is a flowchart illustrating an audio data processing method provided in an embodiment of the present invention. This audio data processing method can be executed by a processing device providing voice dialogue functionality. The processing device can be a physical device or a virtual device, and can be part of a dialogue system. Figure 1 As shown, the method may include the following steps:

[0057] S101, Obtain the first audio segment.

[0058] Users can generate audio data. Optionally, this audio data can be generated in real time during human-computer interaction between the user and the processing device, or it can be obtained by the user recording meeting content or teaching content in real time. Optionally, the audio data can also be pre-collected, such as pre-recorded meeting content or teaching content.

[0059] The processing device can acquire continuously generated audio data from the user in real time and obtain audio segments. Then, the processing device can perform frame-by-frame processing on the acquired audio data to obtain further audio segments. The most recently acquired audio segment at the current moment can be considered the first audio segment. Optionally, the length of the first audio segment can be a preset length, such as 10ms.

[0060] For example, in a human-computer interaction scenario, such as a user-controlled cleaning robot, the user could generate the following audio data for the robot: "Start cleaning the bedroom floor." In a meeting recording scenario, a smart terminal with audio capture capabilities could collect the following audio data during a meeting: "May 1st, the boss is organizing a team-building trip to Beijing for employees." This audio data can then be divided into multiple audio segments according to a preset length.

[0061] S102, based on the semantic features of the first audio segment, determine whether the first audio segment and the second audio segment constitute a complete semantic unit, and the second audio segment is collected before the first audio segment.

[0062] S103, if the first audio segment and the second audio segment constitute a complete semantic, then the first audio segment and the second audio segment are determined as audio to be processed.

[0063] S104, responding to audio to be processed.

[0064] Then, the processing device can extract features from the first audio segment to obtain the semantic features of the first audio segment in the form of vectors. As the name implies, compared with the acoustic features of the audio segment, the semantic features contain richer semantic information. At this time, in an optional manner, the processing device can calculate the similarity between the semantic features of the first audio segment and the feature vectors in the preset feature vector library to determine that the first audio segment and the second audio segment obtained before the first audio segment form a complete semantics. Among them, the number of the second audio segments is at least one. The preset feature vector library stores vectors of common words (such as "le", "de", "ne") that appear at the end of a sentence.

[0065] If the similarity is greater than or equal to the preset similarity, it can be determined that the first audio segment and the second audio segment form a complete semantics. The processing device can also determine the first audio segment and the second audio segment as the audio to be processed, and further respond to the audio to be processed.

[0066] If the similarity is less than the preset similarity, it can be determined that the first audio segment and the second audio segment cannot form a complete semantics. At this time, the processing device can continue to collect the audio segment generated by the user at the next moment of the current moment. After that, the processing device can continue to determine, according to the semantic features of the audio segment collected at the next moment, that the audio segment collected at the next moment, and the above-mentioned first audio segment and second audio segment form a complete semantics.

[0067] Optionally, as mentioned above in the human-computer dialogue scenario and the meeting record scenario, the audio to be processed can include the audio generated by the user and the processing device during the human-computer dialogue or the audio that needs to be speech-transcribed. Correspondingly, for the audio to be processed in different scenarios, the response methods can also be different. For example, if the audio to be processed is the audio generated in the human-computer interaction scenario, the processing device can perform semantic recognition on the audio to be processed and play the corresponding response audio of the audio to be processed. If the audio to be processed is the audio generated in the meeting record scenario, the processing device can perform speech recognition on the audio to be processed and display the text content corresponding to the audio to be processed.

[0068] It should be noted that after the processing device determines that the first audio segment and the second audio segment have a complete semantics, it can determine the current moment when the first audio segment is collected as a segmentation point. The processing device can segment the real-time collected audio data according to this segmentation point to obtain the audio to be processed with complete semantics. And while performing the audio data segmentation, the processing device can continue to collect the audio segments generated by the user after the first audio segment.

[0069] In this embodiment, the processing device acquires a first audio segment and, based on the semantic features of the first audio segment, determines whether the first audio segment and a second audio segment constitute a complete semantic unit. The second audio segment is acquired before the first audio segment. If the first and second audio segments constitute a complete semantic unit, then both the first and second audio segments are identified as audio to be processed. Finally, a response is generated for the audio to be processed.

[0070] In the above method, since the semantic features of the first audio segment can be obtained immediately, it is possible to determine whether the user has produced semantically complete audio, thus shortening the time required to determine semantic completeness. The processing device can then immediately respond to the complete audio, thereby reducing the latency of human-computer dialogue. On the other hand, compared to acoustic features, semantic features contain richer semantic information. Therefore, using semantic features can obviously more accurately identify whether the user has produced semantically complete audio and can respond to it, thereby improving the fluency of human-computer dialogue.

[0071] In addition, for Figure 1 The method provided in the illustrated embodiment, when specifically applied in a human-computer dialogue scenario, can achieve the following technical effects:

[0072] The dialogue system used in human-computer dialogue scenarios can collect audio segments generated by the user in real time and use VAD (Voice over Audio) to detect whether silent audio segments appear. If a preset number of silent audio segments appear consecutively, i.e., the silence lasts for a certain duration, it is determined that the user has generated a semantically complete audio segment before the silent audio segment was collected, meaning the user has finished expressing all the content they wanted to convey. Finally, the dialogue system can perform subsequent processing such as speech recognition on this semantically complete audio. However, when using the above method, the latency of human-computer dialogue is the sum of the processing time required for the dialogue system to perform subsequent processing on the semantically complete audio and the duration of silence, which greatly increases the latency of human-computer dialogue.

[0073] Compared to the methods described above that determine the semantic completeness of audio based on the duration of silence or the acoustic features of audio segments, this embodiment determines whether the user has produced a semantically complete audio segment based on its semantic features. That is, the processing device can immediately determine the semantic completeness of an audio segment after acquiring it and respond immediately to semantically complete audio, without waiting for a period of time, thereby reducing latency in human-computer dialogue. Furthermore, compared to acoustic features, semantic features contain richer semantic information; therefore, using semantic features can obviously more accurately identify whether the user has produced semantically complete audio. Further, based on this reliable recognition result, the dialogue system can respond to audio with complete speech, thereby improving the fluency of human-computer dialogue.

[0074] In addition, for Figure 1 The methods provided in the illustrated embodiments, when specifically applied in meeting recording scenarios, can accurately segment semantically complete sentences contained in long speech recordings and accurately perform speech recognition on the meeting records. When specifically applied in real-time captioning scenarios, the methods provided in the present invention can accurately segment semantically complete sentences contained in long speech recordings with low latency and display the speech recognition results of semantically complete sentences, i.e., captions.

[0075] Figure 1 The illustrated embodiment discloses a method for determining whether a first audio segment and a second audio segment constitute complete semantics based on semantic features. To further improve the accuracy of determining semantic completeness, the processing device may optionally add a step of determining the silence duration before identifying the first and second audio segments as audio to be processed. The silence duration is equal to the product of the length of a silent audio segment and the number of silent audio segments.

[0076] Specifically, after determining that the first and second audio segments possess complete semantic meaning based on semantic features, the processing device does not directly identify them as audio to be processed and respond to them. Instead, it continues to collect a preset number of third audio segments generated after the first audio segment. The processing device can further perform speech activity detection on the collected third audio segments. If the detection result shows that the preset number of third audio segments are all silent audio segments, it indicates that the silence duration after the first audio segment meets the first preset duration, meaning that the user has not generated any speech audio segments within a certain period after generating the first audio segment. At this point, it can be considered that the user has finished expressing what they wanted to say, and the processing device can identify the first and second audio segments as audio to be processed. Since the semantic features can accurately and reliably determine that the first and second audio segments can constitute complete semantic meaning, the first preset duration can be set to a small value, such as 300ms.

[0077] In this embodiment, after determining that the first and second audio segments constitute complete semantics using the semantic features of the audio segments, the processing device can further utilize the acoustic features of the audio segments to detect whether a third audio segment generated after the first audio segment is a silent audio segment and its silence duration, and determine whether the user has expressed complete semantics based on the silence duration. This multi-dimensional feature-based approach can more accurately determine whether the first and second audio segments constitute complete semantics, thereby improving the accuracy of subsequent speech recognition of the audio to be processed.

[0078] Figure 1The illustrated embodiment already mentions determining whether the first and second audio segments constitute complete semantics by comparing feature vectors. Alternatively, classification can also be used to determine whether the first and second audio segments constitute complete semantics. Figure 2 A flowchart of another audio data processing method provided in an embodiment of the present invention is shown below. Figure 2 As shown, the method may include the following steps:

[0079] S201, Obtain the first audio segment.

[0080] For the specific implementation process of step S201 above, please refer to [link / reference]. Figure 1 The specific descriptions of the relevant steps in the illustrated embodiments will not be repeated here.

[0081] S202, Perform speech activity detection on the first audio segment to obtain the first type to which the first audio segment belongs.

[0082] Based on the first audio segment obtained in step S201, the processing device can perform speech activity detection on the first audio segment to determine the first type to which the first audio segment belongs. The first type is obtained by classifying the first audio segment using acoustic features, and may specifically include: a silent audio segment or a speech audio segment. Optionally, for speech activity detection, it can be determined whether the audio segment is a silent audio segment or a speech audio segment by detecting the signal strength; alternatively, it can be determined using a neural network model.

[0083] S203, perform semantic breakpoint classification on the first audio segment based on its semantic features to obtain the second type to which the first audio segment belongs.

[0084] Then, the processing device can extract features from the first audio segment to obtain its semantic features. The processing device can then classify the first audio segment based on its semantic features using semantic breakpoints to determine a second type to which the first audio segment belongs. The second type, obtained by classifying the first audio segment using semantic features, can specifically include either semantic breakpoints or non-semantic breakpoints. Compared to the second classification model in subsequent embodiments, the output of the first classification model is actually a single type.

[0085] Semantic breakpoints indicate that the moment the first audio segment is captured is a segmentation point, meaning the processing device can determine that audio segments captured before this segmentation point are audio with complete semantics to be processed. Non-semantic breakpoints indicate that the moment the first audio segment is captured is not a segmentation point, meaning the processing device can determine that audio segments captured before the segmentation point cannot yield audio with complete semantics to be processed.

[0086] Regarding the determination of the second type, one optional approach is for the processing device to perform semantic breakpoint classification on the first audio segment based on its semantic features and a classification algorithm, thereby obtaining the second type to which the first audio segment belongs. The classification algorithm may include any one of the following: Naive Bayes algorithm, K-Nearest Neighbors algorithm, and Support Vector Machine algorithm.

[0087] Alternatively, the processing device can input the semantic features of the first audio segment into a first classification model, so that the first classification model can output the second type to which the first audio segment belongs. Optionally, the first classification model can include any model with classification function, such as any of the following models: Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and Decision Tree.

[0088] S204, determine whether the first audio segment and the second audio segment constitute a complete semantic based on the first type and the second type, wherein the second audio segment was acquired before the first audio segment.

[0089] S205, if the first audio segment and the second audio segment constitute a complete semantic, then the first audio segment and the second audio segment are determined as audio to be processed.

[0090] Next, the processing device can comprehensively analyze, based on the first type and the second type, whether the first audio segment is a silent audio segment and whether the first audio segment has semantic breakpoints, to determine whether the first audio segment and the second audio segment constitute complete semantics. The second audio segment is captured before the first audio segment.

[0091] In one scenario, if the first type indicates that the first audio segment is a silent audio segment and the second type indicates that the first audio segment has a semantic breakpoint, it suggests that the user is unlikely to produce any more speech audio segments within a certain period after generating the first audio segment. In this case, it can be assumed that the user has finished expressing their intended message, and the first and second audio segments are considered to constitute a complete semantic unit. Optionally, in this case, the processing device can directly identify the first and second audio segments as the audio to be processed.

[0092] The processing device may also, as mentioned in the above embodiments, add a step of determining the silence duration before determining the first audio segment and the second audio segment as the audio to be processed. If it is determined that the silence duration after the first audio segment meets the second preset duration, the processing device can determine the first audio segment and the second audio segment as the audio to be processed. Since it can be determined relatively accurately and reliably that the first audio segment and the second audio segment can constitute complete semantics based on semantic features, the second preset duration can also be set to a small value, such as 300ms.

[0093] In another scenario, if the first type indicates that the first audio segment is a silent audio segment and the second type indicates that the first audio segment has a non-semantic breakpoint, it means that the user is likely to generate another audio segment within a certain period of time after generating the first audio segment. In this case, it can be assumed that the user has not finished expressing what they want to say. The processing device can then continue to collect the fourth audio segment generated after the first audio segment and perform speech activity detection on the fourth audio segment.

[0094] If the detection results show that the preset number of fourth audio segments are all silent audio segments, it indicates that the silence duration after the first audio segment meets the third preset duration, that is, the user has not generated any voice audio segments within a certain period of time after generating the first audio segment. At this time, it can be assumed that the user has finished expressing what he wanted to say, and the processing device can determine the first audio segment and the second audio segment as the audio to be processed.

[0095] Optionally, non-semantic breakpoints can be either non-punctuation marks or non-ending punctuation marks. Non-ending punctuation marks can be commas or pause marks. During continuous speech, users may experience short pauses, which may occur within the same semantic group or between different semantic groups within a sentence.

[0096] When the first audio segment contains non-punctuation, it indicates that the user may have paused while expressing the same thought group. When the first audio segment contains no ending punctuation, it indicates that the user may have paused between expressing different thought groups. Therefore, compared to a non-ending punctuation mark, the user is more likely to continue speaking after the non-punctuation mark. Thus, when the first audio segment contains non-punctuation, the third preset duration can be set to a larger value, such as 700ms. When the first audio segment contains no ending punctuation, the third preset duration can be set to a smaller value, such as 400ms.

[0097] S206, responding to audio to be processed.

[0098] For the specific implementation process of step 206 above, please refer to [link / reference]. Figure 1 The specific descriptions of the relevant steps in the illustrated embodiments will not be repeated here.

[0099] In this embodiment, the processing device initially determines whether the first audio segment is a silent audio segment based on the first type to which it belongs, and further determines whether the first audio segment has semantic breakpoints based on the second type to which it belongs. That is, by comprehensively analyzing the results of the two classifications, the processing device can accurately determine whether the first audio segment and the second audio segment constitute complete semantics.

[0100] Figure 2 The illustrated embodiment has mentioned one method for determining whether a first audio segment and a second audio segment constitute complete semantics through classification. Alternatively, another method... Figure 3 A flowchart of another audio data processing method provided in an embodiment of the present invention is shown below. Figure 3 As shown, the method may include the following steps:

[0101] S301, acquire the first audio segment.

[0102] For the specific implementation process of step S301 above, please refer to [link / reference]. Figure 1 The specific descriptions of the relevant steps in the illustrated embodiments will not be repeated here.

[0103] S302, input the semantic features of the first audio segment into the second classification model, so that the second classification model can output the composite type to which the first audio segment belongs.

[0104] Optionally, the processing device can extract features from the first audio segment to obtain its semantic features. Then, the processing device can input the semantic features of the first audio segment into a second classification model, which will output the composite type to which the first audio segment belongs. Optionally, the second classification model can include any model with classification capabilities, such as a CNN model, an LSTM model, or a decision tree model. The second classification model can have the same or different model structure as the first classification model. The composite type includes any of the following: the first audio segment is a silent audio segment with semantic breaks; the first audio segment is a silent audio segment with non-semantic breaks; the first audio segment is a speech audio segment with semantic breaks; and the first audio segment is a speech audio segment with non-semantic breaks.

[0105] S303, if the composite type reflects that the first audio segment is a silent audio segment and the first audio segment has a semantic breakpoint, then it is determined that the first audio segment and the second audio segment constitute a complete semantic, wherein the second audio segment was collected before the first audio segment.

[0106] S304, if the first audio segment and the second audio segment constitute a complete semantic, then the first audio segment and the second audio segment are determined as audio to be processed.

[0107] Optionally, in one case, if the composite type reflects that the first audio segment is a silent audio segment and the first audio segment has a semantic breakpoint, indicating that the user has not produced any speech audio segments for a period of time after producing the first audio segment, it can be assumed that the user has finished expressing what they wanted to say, and then it is determined that the first audio segment and the second audio segment constitute a complete semantic.

[0108] Optionally, in this case, the processing device can directly determine the first audio segment and the second audio segment as the audio to be processed. The processing device can also, as mentioned in the above embodiments, add a step of determining the silence duration before determining the first audio segment and the second audio segment as the audio to be processed. For details, please refer to the descriptions in the relevant embodiments above, which will not be repeated here.

[0109] In another scenario, if the composite type reflects one of the three cases other than the first audio segment being a silent audio segment or the first audio segment having a semantic breakpoint, it indicates that the user is likely to generate another audio segment within a certain period of time after generating the first audio segment. In this case, it can be assumed that the user has not finished expressing what they want to say. The processing device can then continue to collect a fourth audio segment after the first audio segment and determine whether to identify the first and second audio segments as audio segments with complete semantics based on the detection results of the speech activity detection of the fourth audio segment.

[0110] It should be noted that the second classification model used in this embodiment can be considered as classifying audio segments into speech segments or silent audio segments, while also classifying them into semantic breakpoints. In other words, the second classification model can be considered a classification model that integrates VAD detection and semantic breakpoint classification functions. In contrast, the first classification model only has semantic breakpoint classification functionality.

[0111] S305 responds to audio to be processed.

[0112] For the specific implementation process of step S305 above, please refer to [link / reference]. Figure 2 The specific descriptions of the relevant steps in the illustrated embodiments will not be repeated here.

[0113] In this embodiment, the processing device can perform semantic breakpoint classification on the first audio segment using a second classification model to obtain the composite type to which the first audio segment belongs. Based on the result reflected by this composite type and the manually set endpoint labels, that is, by combining multiple semantic judgment methods, it can more accurately determine whether the first audio segment and the second audio segment constitute complete semantics. In addition, the contents not described in detail in this embodiment and the technical effects that can be achieved can be referred to the relevant descriptions in the above embodiments, and will not be repeated here.

[0114] It should also be noted that, Figure 2 The first classification model mentioned in the illustrated embodiment and Figure 3 The second classification model mentioned in the illustrated embodiment can be used alone or in combination. Optionally, when the first classification model and the second classification model are used in combination, if the first classification model and / or the second classification model determine that the first audio segment is a silent audio segment and has semantic breakpoints, then the first audio segment and the second audio segment are determined to be the audio to be processed.

[0115] According to the above Figures 1 to 3 As described in the illustrated embodiment, the processing device can extract the semantic features of the first audio segment. Specifically, the processing device can first extract the acoustic features of the first audio segment through processing methods such as Fourier transform and windowing, that is, extract the low-level features of the first audio segment. Then, the processing device can extract the semantic features of the first audio segment from the acoustic features, which can be considered as the high-level features of the first audio segment.

[0116] Optionally, the processing device can utilize a feature extraction model to extract semantic features. This feature extraction model may include one of the following: a Bidirectional Encoder Representation from Transformers (BERT) model, a CNN model, or an LSTM model.

[0117] According to the above Figures 1 to 3 As can be seen from the usage process of the processing device described in the illustrated embodiment, when determining whether the first audio segment and the second audio segment constitute complete semantics based on semantic features, the processing device uses at least one of a first classification model, a second classification model, and a feature extraction model. It should be noted that the above models can be deployed on interactive devices or in the cloud.

[0118] The training process for each model is described below. Furthermore, each of the following embodiments can be executed by a training device. Optionally, the training device can be the processing device mentioned in the above embodiments, or it can be another device besides the processing device, which can be deployed in the cloud.

[0119] To ensure that the primary classification model outputs more accurate classification results, the training device can use supervised training to train the primary classification model. Figure 4 This is a flowchart illustrating a model training method provided in an embodiment of the present invention. This embodiment can be executed by the aforementioned training device. Figure 4 As shown, the method may include the following steps:

[0120] S401, Obtain the semantic features of the first training audio segment and the reference type of the first training audio segment.

[0121] Optionally, the semantic features and reference type of the first training audio segment can be obtained through preprocessing. That is, the training device can acquire the first training audio segment from a pre-collected set of training audio segments and perform feature extraction on the acquired first training audio segment to obtain the semantic features corresponding to the first training audio segment. The process of extracting semantic features can be found in the description of the use of the processing device in the above embodiments. The reference type of the first training audio segment can be obtained through manual annotation, and the reference type can include semantic breakpoints or non-semantic breakpoints.

[0122] S402, use the semantic features of the first training audio segment as training data and the reference type as supervision information to train the first classification model.

[0123] S403, calculate the loss for the predicted type of the first training audio segment output by the first classification model and the reference type of the first training audio segment.

[0124] S404, optimize the first classification model based on the loss calculation results.

[0125] Based on the semantic features and reference type of the first training audio segment obtained in step S401, the training device can use the semantic features of the first training audio segment as training data and the reference type corresponding to the first training audio segment as supervision information to train the first classification model.

[0126] Next, the training device can calculate the loss between the predicted type of the first training audio segment output by the first classification model and the reference type corresponding to the first training audio segment, and optimize the first classification model based on the obtained loss calculation results. Optionally, the loss function used in the loss calculation process may include one of the following: cross-entropy loss function, log loss function, and squared loss function.

[0127] In this embodiment, a supervised training method is used, that is, the first classification model is trained with the reference type corresponding to the first training audio segment, so that the training effect of the first classification model is better. Furthermore, the predicted type of the first training audio segment output by the first classification model is also more accurate during the model usage phase.

[0128] To ensure the second classification model outputs more accurate classification results, similar to the process of training the first classification model, the training device can also use supervised training to train the second classification model. This embodiment can also be executed by the aforementioned training device. Figure 5 A flowchart illustrating another model training method provided in an embodiment of the present invention. For example... Figure 5 As shown, the method may include the following steps:

[0129] S501, Obtain the semantic features of the training audio segment and the reference composite type of the training audio segment.

[0130] Optionally, the semantic features of the training audio segments and the reference composite type of the training audio segments can be obtained through preprocessing. The acquisition process can be found in [link to documentation]. Figure 4 The relevant descriptions in step S401 of the embodiment will not be repeated here.

[0131] The reference composite type of any training audio segment may include any one of the following: the training audio segment is a silent audio segment and the training audio segment has semantic breaks; the training audio segment is a silent audio segment and the training audio segment has non-semantic breaks; the training audio segment is a speech audio segment and the training audio segment has semantic breaks; and the training audio segment is a speech audio segment and the training audio segment has non-semantic breaks.

[0132] S502 uses the semantic features of the training audio segments as training data and the reference composite type as supervision information to train the second classification model.

[0133] S503, calculate the loss for the predicted composite type of the training audio segment output by the second classification model and the reference composite type of the training audio segment.

[0134] S504, optimize the second classification model based on the loss calculation results.

[0135] The training process for the second classification model is similar to that of the first model; for details, please refer to [link / reference]. Figure 4 The relevant descriptions in the illustrated embodiments.

[0136] In this embodiment, a supervised training method is used, that is, the second classification model is trained with reference composite type, which makes the second classification model perform better. Furthermore, the predicted type of the training audio segment output by the second classification model is also more accurate during the model usage phase.

[0137] Since the accuracy of the classification result is related to the input of the classification model, namely the semantic features, the classification accuracy can be improved by using an additional speech recognition model and the aforementioned feature extraction model. Furthermore, the speech recognition results from the speech recognition model can also improve the feature extraction capability of the feature extraction model. In this case, the training device can jointly train the classification model, the speech recognition model, and the feature extraction model. The classification model here can include the first classification model and / or the second classification model mentioned in the above embodiments.

[0138] but Figure 6 This is a flowchart illustrating another model training method provided in an embodiment of the present invention. This embodiment can also be executed using the aforementioned training equipment. Figure 6 As shown, the method may include the following steps:

[0139] S601, obtain the semantic features of the second training audio segment, the reference type of the second training audio segment, and the reference speech recognition result of the second training audio segment.

[0140] S602, the semantic features of the second training audio segment are used as training data, and the reference type is used as supervision information to train the classification model. The classification model includes a first classification model and / or a second classification model.

[0141] Optionally, the semantic features of the second training audio segment, the reference type of the second training audio segment, and the reference speech recognition result of the second training audio segment can all be obtained through preprocessing. That is, the training device can obtain the second training audio segment from the pre-collected training audio segment set and perform feature extraction on the obtained second training audio segment to obtain the semantic features corresponding to the second training audio segment. The process of extracting semantic features can be found in the description of the use of the processing device in the above embodiments.

[0142] The reference type for the second training audio segment can be obtained through manual annotation, and the reference type corresponds to the classification model. Specifically, if the classification model includes the first classification model, the reference type can include: semantic breakpoint or non-semantic breakpoint.

[0143] If the classification model includes a second classification model, the reference type may include any one of the following: the second training audio segment is a silent audio segment and the second training audio segment has a semantic breakpoint; the second training audio segment is a silent audio segment and the second training audio segment has a non-semantic breakpoint; the second training audio segment is a speech audio segment and the second training audio segment has a semantic breakpoint; and the second training audio segment is a speech audio segment and the second training audio segment has a non-semantic breakpoint.

[0144] If the classification model includes a first classification model and a second classification model, then the reference type may include at least one of all the types mentioned above.

[0145] The reference speech recognition result for the second training audio segment can be determined manually. The speech recognition result may include the text information corresponding to the second training audio segment.

[0146] Next, the training device can use the semantic features of the second training audio segment as training data and the reference type corresponding to the second training audio segment as supervision information to train the classification model.

[0147] S603, the semantic features of the second training audio segment are used as training data, and the reference speech recognition result is used as supervision information to train the speech recognition model.

[0148] Based on the semantic features of the second training audio segment and the reference speech recognition result obtained in step S601, the training device can also use the semantic features of the second training audio segment as training data for the speech recognition model and use the reference speech recognition result as supervision information to train the speech recognition model.

[0149] S604. Determine the first loss value based on the predicted type of the second training audio segment output by the classification model and the reference type of the second training audio segment.

[0150] S605, determine the second loss value based on the predicted speech recognition result of the second training audio segment output by the speech recognition model and the reference speech recognition result.

[0151] S606 optimizes the classification model, speech recognition model, and feature extraction model based on the first loss value, the second loss value, and their respective weight parameters.

[0152] Next, the training device can determine a first loss value based on the predicted type of the second training audio segment output by the classification model and the reference type of the second training audio segment; and determine a second loss value based on the predicted speech recognition result of the second training audio segment output by the speech recognition model and the reference speech recognition result. The first and second loss values ​​can be calculated using a loss function.

[0153] Ultimately, the training device can optimize the classification model, speech recognition model, and feature extraction model based on the first loss value, the second loss value, and their respective weight parameters. Since the method provided by this invention focuses on using the classification results obtained from the classification model to determine whether the first and second audio segments constitute complete semantics, the weight parameter of the first loss value corresponding to the classification model can be set to a larger value. Of course, different methods can be adopted for setting the weight parameters according to actual needs.

[0154] The joint training process of the above multiple models can also be combined Figure 7 The training device first obtains the semantic features of the second training audio segment through a feature extraction model. Then, these semantic features are input into both a speech recognition model and a classification model. The speech recognition model outputs a predicted speech recognition result, and the classification model outputs a predicted type. The predicted type can include predicted semantic breakpoint type and / or predicted composite type. During this process, the first loss value generated by the speech recognition model can be used to further train the feature extraction model to better extract the semantic features of the first audio segment. In other words, accurate semantic features can be obtained through the speech recognition model and feature extraction model. Inputting these accurate semantic features into the classification model can further improve the classification accuracy.

[0155] In this embodiment, on the one hand, utilizing the speech recognition results of the speech recognition model can improve the feature extraction capability of the feature extraction model, resulting in semantic features extracted by the feature extraction model having richer and more accurate semantic information. On the other hand, based on the accurate semantic features extracted by the speech recognition model and the feature extraction model, jointly training the classification model, the speech recognition model, and the feature extraction model can further improve the classification accuracy of the classification model.

[0156] Figure 8 This is a flowchart illustrating another audio data processing method provided in an embodiment of the present invention. The audio data processing method provided in this embodiment is essentially a training method for a classification model, which can be executed by a training device. Optionally, the training device can be the processing device mentioned in the above embodiments, or it can be another device besides the processing device, which can be deployed in the cloud. Figure 7 As shown, the method may include the following steps:

[0157] S801, obtain the semantic features of the training audio segment and the reference type of the training audio segment, where the training audio segment is any audio segment in the training audio.

[0158] S802, the semantic features of the training audio segment are used as training data, and the reference type is used as supervision information to train the classification model. The classification result output by the classification model is used to determine whether the training audio segment and the audio segment played before the training audio segment constitute complete semantics.

[0159] S803 calculates the loss for the predicted type and reference type of the training audio segments output by the classification model.

[0160] S804, optimize the classification model based on the loss calculation results.

[0161] The classification model in this embodiment can be either the first classification model or the second classification model mentioned in the above embodiments. The training process of the classification model can be found in [reference needed]. Figure 4 or Figure 5 The embodiments shown are not described in detail here.

[0162] Furthermore, any content not described in detail in this embodiment, as well as the technical effects that can be achieved, can be found in the relevant descriptions of the above embodiments, and will not be repeated here.

[0163] Figures 1 to 8 The illustrated embodiment details the processing procedure of the processing device for audio data. Based on this, a human-computer interaction method can also be proposed, starting from the user's interaction flow. Figure 9 This is a flowchart illustrating a human-computer interaction method provided in an embodiment of the present invention. This method can also be provided to users as a SaaS service. Therefore, the executing entity of this method can specifically be a service platform. Figure 9 As shown, the method may include the following steps:

[0164] S901, in response to user input, captures the first audio segment generated by the user.

[0165] S902, output the response result of the audio to be processed with complete semantics, wherein whether the semantics of the audio to be processed is complete is determined according to the semantic features of the first audio segment, and the audio to be processed includes the first audio segment and the second audio segment acquired before the first audio segment.

[0166] Optionally, the service platform can respond to a user's touch operation or voice command to acquire the first audio segment generated by the user. For example, when a user touches a control on the service platform, they can start recording a meeting. In a meeting recording scenario, the user can upload the audio to be transcribed to the service platform by touching the controls on the interface provided by the service platform. The service platform can perform frame-by-frame processing on the audio data uploaded by the user to obtain audio segments. The latest audio segment acquired by the service platform at the current moment can be considered the first audio segment.

[0167] In a cleaning robot scenario, users can issue voice commands to the cleaning robot. The cleaning robot can collect the audio data generated by the user in real time to obtain the latest audio segment at the current moment, i.e., the first audio segment.

[0168] Then, the service platform can output a response result with complete semantics of the audio to be processed. Optionally, the audio to be processed can include audio generated during the human-computer dialogue between the user and the service platform, or audio that needs to be transcribed into speech. Correspondingly, the response method can also differ depending on the audio to be processed in different scenarios.

[0169] For example, if the audio to be processed is generated in a human-computer interaction scenario, the processing device can perform semantic recognition on the audio and play the corresponding response audio. If the audio to be processed is generated in a meeting recording scenario, the processing device can perform speech recognition on the audio and display the corresponding text content.

[0170] The semantic completeness of the audio to be processed is determined based on the semantic features of the first audio segment. The audio to be processed includes the first audio segment and a second audio segment acquired before the first audio segment. Furthermore, the process of acquiring the first audio segment and determining the complete semantics can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0171] Furthermore, the contents not described in detail in this embodiment and the technical effects that can be achieved can be found in the relevant descriptions of the above embodiments, and will not be repeated here.

[0172] To facilitate understanding, the specific implementation process of the audio data processing methods provided in the above embodiments will be illustrated by example in the following scenario.

[0173] The specific implementation process of the audio data processing method and human-computer interaction method provided in the above embodiments in the human-computer interaction scenario can be combined with Figure 10 Understood. In human-computer interaction scenarios, such as user-controlled cleaning robots, users can generate audio data for the robot. Simultaneously, the cleaning robot can collect this audio data in real time during the user's interaction to obtain the audio segment generated at time T1, i.e., the first audio segment.

[0174] Then, the cleaning robot can extract features from the acquired first audio segment to obtain its semantic features. Next, the cleaning robot can comprehensively analyze whether the first and second audio segments constitute complete semantics based on the semantic features of the first audio segment and the classification results of the classification model. The second audio segment is acquired before the first audio segment. Optionally, the classification model may include a first classification model and / or a second classification model.

[0175] After determining that the first and second audio segments have complete semantics, the cleaning robot can obtain a piece of audio to be processed with complete semantics, such as "Start cleaning the bedroom floor". Finally, the cleaning robot can perform semantic recognition on the obtained audio to be processed and play the corresponding response audio, such as "Okay, start cleaning now".

[0176] In addition, while performing semantic recognition on the audio to be processed, the cleaning robot can also continue to collect audio segments generated by the user at time T2.

[0177] Furthermore, the processes for extracting semantic features, determining complete semantics, training the classification model, and achieving the desired technical effects can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0178] In meeting recording scenarios, users can utilize services provided by the platform, such as... Figure 11 The interface shown uploads the audio to be transcribed to the service platform. The service platform can then perform frame-by-frame processing on the user-uploaded audio data to obtain audio segments.

[0179] The service platform can sequentially process each audio segment as follows: For any audio segment, the service platform can extract features from the acquired audio segment to obtain its semantic features. Then, based on the semantic features of the audio segment and the classification results of the classification model, the service platform can comprehensively analyze whether the audio segment and the preceding audio segments in the audio data constitute complete semantics. Optionally, the classification model may include a first classification model and / or a second classification model.

[0180] Once the service platform determines that multiple audio segments have complete semantics, it can perform speech recognition on these audio segments with complete semantics. The speech recognition results can be displayed on the operation interface provided by the service platform, which is to say, the text content of the audio to be processed is displayed on the interface.

[0181] Furthermore, the processes for extracting semantic features, determining complete semantics, training the classification model, and achieving the desired technical effects can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0182] The audio data processing apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that these audio data processing apparatuses can all be configured using commercially available hardware components through the steps taught in this invention.

[0183] Figure 12 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of the present invention, as shown below. Figure 12 As shown, the device includes:

[0184] Acquisition module 11 is used to acquire the first audio segment.

[0185] The complete semantic determination module 12 is used to determine whether the first audio segment and the second audio segment constitute complete semantics based on the semantic features of the first audio segment, wherein the second audio segment is collected before the first audio segment.

[0186] The audio to be processed determination module 13 is used to determine the first audio segment and the second audio segment as audio to be processed if the first audio segment and the second audio segment constitute a complete semantics.

[0187] Response module 14 is used to respond to the audio to be processed.

[0188] Optionally, the device further includes: an activity detection module 15, used to perform speech activity detection on at least one third audio segment acquired after the first audio segment.

[0189] The audio to be processed determination module 13 is further configured to determine the first audio segment and the second audio segment as audio to be processed if the detection result indicates that the third audio segment is a silent audio segment.

[0190] Optionally, the complete semantic determination module 12 is used to perform speech activity detection on the first audio segment to obtain a first type to which the first audio segment belongs; to perform semantic breakpoint classification on the first audio segment according to the semantic features of the first audio segment to obtain a second type to which the first audio segment belongs; and to determine whether the first audio segment and the second audio segment constitute complete semantics based on the first type and the second type.

[0191] Optionally, the complete semantic determination module 12 is used to determine that the first audio segment and the second audio segment constitute complete semantics if the first type reflects that the first audio segment is a silent audio segment and the second type reflects that the first audio segment has a semantic breakpoint.

[0192] Optionally, the complete semantic determination module 12 is configured to perform speech activity detection on at least one fourth audio segment collected after the first audio segment if the first type reflects that the first audio segment is a silent audio segment and the second type reflects that the first audio segment has a non-semantic breakpoint; and if the fourth audio segment is a silent audio segment, determine that the first audio segment and the second audio segment constitute complete semantics.

[0193] Optionally, the complete semantic determination module 12 is used to input the semantic features of the first audio segment into a first classification model so that the first classification model outputs a second type of the first audio segment.

[0194] Optionally, the apparatus further includes: a training module 16, configured to acquire semantic features of a first training audio segment and a reference type of the training audio segment; train the first classification model using the semantic features of the first training audio segment as training data and the reference type as supervision information; perform loss calculation on the predicted type of the first training audio segment and the reference type of the first training audio segment output by the first classification model; and optimize the first classification model based on the loss calculation result.

[0195] Optionally, the complete semantic determination module 12 is used to input the semantic features of the first audio segment into the second classification model, so that the second classification model outputs the composite type to which the first audio segment belongs; if the composite type reflects that the first audio segment is a silent audio segment and the first audio segment is a semantic breakpoint, then it is determined that the first audio segment and the second audio segment constitute a complete semantic.

[0196] Optionally, the device further includes an acoustic feature extraction module 17 and a semantic feature extraction module 18.

[0197] The acoustic feature extraction module 17 is used to extract features from the first audio segment to obtain the acoustic features of the first audio segment.

[0198] The semantic feature extraction module 18 is used to input the acoustic features into the feature extraction model so that the feature extraction model outputs the semantic features of the first audio segment.

[0199] Optionally, the training module 16 is configured to: acquire the semantic features of the second training audio segment, the reference type of the second training audio segment, and the reference speech recognition result of the second training audio segment; train a classification model using the semantic features of the second training audio segment as training data and the reference type as supervised information, wherein the classification model includes the first classification model and / or the second classification model; train a speech recognition model using the semantic features of the second training audio segment as training data and the reference speech recognition result as supervised information; determine a first loss value based on the predicted type of the second training audio segment output by the classification model and the reference type of the second training audio segment; determine a second loss value based on the predicted speech recognition result of the second training audio segment output by the speech recognition model and the reference speech recognition result; and optimize the classification model, the speech recognition model, and the feature extraction model based on the first loss value, the second loss value, and their respective weight parameters.

[0200] Figure 12 The device shown can perform Figures 1 to 7 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figures 1 to 7 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figures 1 to 7 The descriptions in the illustrated embodiments will not be repeated here.

[0201] Figure 13 This is a schematic diagram of another audio data processing device provided in an embodiment of the present invention, as shown below. Figure 13 As shown, the device includes:

[0202] The training audio segment acquisition module 21 is used to acquire the semantic features of the training audio segment and the reference type of the training audio segment, wherein the training audio segment is any audio segment in the training audio.

[0203] The classification model training module 22 is used to train a classification model by using the semantic features of the training audio segment as training data and the reference type as supervision information. The classification model is used to determine whether the training audio segment is a semantic breakpoint in the training audio.

[0204] The loss calculation module 23 is used to calculate the loss of the predicted type and the reference type of the training audio segment output by the classification model.

[0205] The optimization module 24 is used to optimize the classification model based on the loss calculation results.

[0206] Figure 13 The device shown can perform Figure 8For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figure 8 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figure 8 The descriptions in the illustrated embodiments will not be repeated here.

[0207] Figure 14 This is a schematic diagram of the structure of a human-computer interaction device provided in an embodiment of the present invention, as shown below. Figure 14 As shown, the device includes:

[0208] Acquisition module 31 is used to acquire the first audio segment generated by the user in response to the user's input operation.

[0209] Output module 32 is used to output a response result of the audio to be processed with complete semantics, wherein whether the semantics of the audio to be processed is complete is determined according to the semantic features of the first audio segment, and the audio to be processed includes the first audio segment and a second audio segment collected before the first audio segment.

[0210] Figure 14 The device shown can perform Figure 9 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figure 9 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figure 9 The descriptions in the illustrated embodiments will not be repeated here.

[0211] In one possible design, the audio data processing methods provided in the above embodiments can be applied to an electronic device, such as... Figure 15 As shown, the electronic device may include a first processor 41 and a first memory 42. The first memory 42 is used to store data supporting the electronic device in performing the above-described actions. Figures 1 to 7 In the audio data processing method program provided in the illustrated embodiment, the first processor 41 is configured to execute the program stored in the first memory 42.

[0212] The program includes one or more computer instructions, wherein the one or more computer instructions, when executed by the first processor 41, can perform the following steps:

[0213] Get the first audio segment;

[0214] Based on the semantic features of the first audio segment, it is determined whether the first audio segment and the second audio segment constitute a complete semantic unit, and the second audio segment is collected before the first audio segment;

[0215] If the first audio segment and the second audio segment constitute a complete semantic unit, then the first audio segment and the second audio segment are determined as audio to be processed;

[0216] Respond to the audio to be processed.

[0217] Optionally, the first processor 41 is also used to perform the aforementioned Figures 1 to 7 All or part of the steps in the illustrated embodiments.

[0218] The structure of the electronic device may also include a first communication interface 43 for the electronic device to communicate with other devices or communication systems.

[0219] In addition, embodiments of the present invention provide a computer storage medium for storing computer software instructions used by the aforementioned electronic device, which includes instructions for executing the above-mentioned... Figures 1 to 7 The audio data processing method shown involves the following procedures.

[0220] In one possible design, the audio data processing methods provided in the above embodiments can be applied to another electronic device, such as... Figure 16 As shown, the electronic device may include a second processor 51 and a second memory 52. ​​The second memory 52 is used to store data supporting the electronic device in performing the above-described actions. Figure 8 In the audio data processing method program provided in the illustrated embodiment, the second processor 51 is configured to execute the program stored in the second memory 52.

[0221] The program includes one or more computer instructions, wherein the one or more computer instructions, when executed by the second processor 51, can perform the following steps:

[0222] Obtain the semantic features of the training audio segment and the reference type of the training audio segment, wherein the training audio segment is any audio segment in the training audio;

[0223] The semantic features of the training audio segment are used as training data, and the reference type is used as supervision information to train the classification model. The classification result output by the classification model is used to determine whether the training audio segment and the audio segment played before the training audio segment constitute complete semantics.

[0224] Loss is calculated for the predicted type and the reference type of the training audio segment output by the classification model;

[0225] The classification model is optimized based on the loss calculation results.

[0226] Optionally, the second processor 51 is also used to perform the aforementioned Figure 8 All or part of the steps in the illustrated embodiments.

[0227] The electronic device may also include a second communication interface 53 for communicating with other devices or communication systems.

[0228] In addition, embodiments of the present invention provide a computer storage medium for storing computer software instructions used by the aforementioned electronic device, which includes instructions for executing the above-mentioned... Figure 8 The audio data processing method shown involves the following procedures.

[0229] In one possible design, the human-computer interaction methods provided in the above embodiments can be applied to yet another electronic device, such as... Figure 17 As shown, the electronic device may include a third processor 61 and a third memory 62. The third memory 62 is used to store data supporting the electronic device in performing the aforementioned operations. Figure 9 In the human-computer interaction method program provided in the illustrated embodiment, the third processor 61 is configured to execute the program stored in the third memory 62.

[0230] The program includes one or more computer instructions, wherein the one or more computer instructions, when executed by the third processor 61, can perform the following steps:

[0231] In response to user input, capture the first audio segment generated by the user;

[0232] The output is a response result of the audio to be processed with complete semantics, wherein whether the semantics of the audio to be processed is complete is determined according to the semantic features of the first audio segment, and the audio to be processed includes the first audio segment and a second audio segment acquired before the first audio segment.

[0233] Optionally, the third processor 61 is also used to perform the aforementioned Figure 9 All or part of the steps in the illustrated embodiments.

[0234] The structure of the electronic device may also include a third communication interface 63 for the electronic device to communicate with other devices or communication systems.

[0235] In addition, embodiments of the present invention provide a computer storage medium for storing computer software instructions used by the aforementioned electronic device, which includes instructions for executing the above-mentioned... Figure 9 The program involved in the human-computer interaction method shown.

[0236] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio data processing method, characterized in that, include: Get the first audio segment; Based on the semantic features of the first audio segment, it is determined whether the first audio segment and the second audio segment constitute a complete semantic unit, and the second audio segment is collected before the first audio segment; If the first audio segment and the second audio segment constitute a complete semantic unit, then the first audio segment and the second audio segment are determined as audio to be processed; Responding to the audio to be processed; The step of determining whether the first audio segment and the second audio segment constitute a complete semantic unit based on the semantic features of the first audio segment includes: Speech activity detection is performed on the first audio segment to determine the first type to which the first audio segment belongs; Based on the semantic features of the first audio segment, the first audio segment is classified by semantic breakpoints to obtain the second type to which the first audio segment belongs; Based on the first type and the second type, determine whether the first audio segment and the second audio segment constitute a complete semantic unit.

2. The method according to claim 1, characterized in that, Before determining the first audio segment and the second audio segment as the audio to be processed, the method further includes: Speech activity detection is performed on at least one third audio segment acquired after the first audio segment; If the detection result indicates that the third audio segment is a silent audio segment, then the first audio segment and the second audio segment are determined as the audio to be processed.

3. The method according to claim 1, characterized in that, The step of determining whether the first audio segment and the second audio segment constitute complete semantics based on the first type and the second type includes: If the first type indicates that the first audio segment is a silent audio segment and the second type indicates that the first audio segment has a semantic breakpoint, then it is determined that the first audio segment and the second audio segment constitute a complete semantic unit.

4. The method according to claim 1, characterized in that, The step of determining whether the first audio segment and the second audio segment constitute complete semantics based on the first type and the second type includes: If the first type indicates that the first audio segment is a silent audio segment and the second type indicates that the first audio segment has a non-semantic breakpoint, then speech activity detection is performed on at least one fourth audio segment acquired after the first audio segment. If the fourth audio segment is a silent audio segment, then the first audio segment and the second audio segment are determined to constitute a complete semantic unit.

5. The method according to claim 1, characterized in that, The step of classifying the first audio segment based on the semantic features includes: The semantic features of the first audio segment are input into the first classification model, so that the first classification model outputs the second type of the first audio segment.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the semantic features of the first training audio segment and the reference type of the training audio segment; The semantic features of the first training audio segment are used as training data, and the reference type is used as supervision information to train the first classification model. Loss is calculated for the predicted type of the first training audio segment output by the first classification model and the reference type of the first training audio segment. The first classification model is optimized based on the loss calculation results.

7. The method according to claim 5, characterized in that, The method further includes: Feature extraction is performed on the first audio segment to obtain the acoustic features of the first audio segment; The acoustic features are input into a feature extraction model, which then outputs the semantic features of the first audio segment.

8. The method according to claim 7, characterized in that, The method further includes: Obtain the semantic features of the second training audio segment, the reference type of the second training audio segment, and the reference speech recognition result of the second training audio segment; The semantic features of the second training audio segment are used as training data, and the reference type is used as supervised information to train a classification model, wherein the classification model includes the first classification model; The semantic features of the second training audio segment are used as training data, and the reference speech recognition result is used as supervision information to train the speech recognition model. The first loss value is determined based on the predicted type of the second training audio segment output by the classification model and the reference type of the second training audio segment. The second loss value is determined based on the predicted speech recognition result of the second training audio segment output by the speech recognition model and the reference speech recognition result; The classification model, the speech recognition model, and the feature extraction model are optimized based on the first loss value, the second loss value, and their respective weight parameters.

9. A human-computer interaction method, characterized in that, Applications to service platforms include: In response to user input, capture the first audio segment generated by the user; The system outputs a response result of the audio to be processed with complete semantics. Whether the semantics of the audio to be processed are complete is determined based on the semantic features of the first audio segment. The audio to be processed includes the first audio segment and a second audio segment acquired before the first audio segment. Determining whether the first audio segment and the second audio segment constitute complete semantics based on the semantic features of the first audio segment includes: performing speech activity detection on the first audio segment to obtain a first type to which the first audio segment belongs; performing semantic breakpoint classification on the first audio segment based on its semantic features to obtain a second type to which the first audio segment belongs; and determining whether the first audio segment and the second audio segment constitute complete semantics based on the first type and the second type.

10. An electronic device, characterized in that, include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the audio data processing method as described in any one of claims 1 to 8, or performs the human-computer interaction method as described in claim 9.

11. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the audio data processing method as described in any one of claims 1 to 8, or causes the processor to perform the human-computer interaction method as described in claim 9.

Images