Audio stream processing methods and apparatus, storage media and electronic devices

CN117672188BActive Publication Date: 2026-11-17HAIER YOUJIA INTELLIGENT TECH (BEIJING) CO LTD +2
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202211058664.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-31
Publication Date
2026-11-17
Estimated Expiration
2042-08-31

AI Technical Summary

Technical Problem

[0005]本发明实施例提供了一种音频流处理方法及装置、存储介质及电子装置,以至少解决确定音频流的音频流结束时刻的准确性较低的问题

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117672188B_ABST
    Figure CN117672188B_ABST
Patent Text Reader

Abstract

This application discloses an audio stream processing method, apparatus, storage medium, and electronic device, relating to the field of smart home technology. The audio stream processing method includes: performing acoustic detection on audio segments acquired within multiple consecutive preset time intervals to obtain acoustic attributes of the audio segments; wherein the audio segments are audio segments included in an audio stream obtained by continuous audio acquisition when the target device is awakened; performing speech recognition on the audio segments acquired within multiple consecutive preset time intervals to obtain recognized text of the audio segments; and determining the audio stream end time based on the acoustic attributes and / or recognized text, wherein the audio stream end time is the time when the target device stops audio acquisition. By employing the above technical solution, the problem of low accuracy in determining the audio stream end time is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to an audio stream processing method and apparatus, a storage medium and an electronic device. Background Technology

[0002] With the mature development of IoT technology, smart homes connect various devices in the home through IoT technology. In human-computer interaction, the voice module is generally a basic component of smart home devices, enabling them to "hear" sounds. When a user uses a smart voice device, the device needs to detect the last frame of the audio stream to determine whether the user's voice input has ended. However, current last frame detection relies on speech recognition results, and the rules and constraints are biased towards judging the completeness of speech and the accuracy of recognition, resulting in the timing of the last frame determination of the audio stream being later than the actual end time of the user's voice input.

[0003] There is currently no effective solution to the problem of low accuracy in determining the end time of an audio stream in related technologies.

[0004] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0005] The present invention provides an audio stream processing method, apparatus, storage medium, and electronic device to at least solve the problem of low accuracy in determining the end time of an audio stream.

[0006] According to one aspect of the present invention, an audio stream processing method is provided, comprising: performing acoustic detection on audio segments acquired respectively within a plurality of consecutive preset time intervals to obtain acoustic attributes of a plurality of audio segments; wherein the audio segments are audio segments included in an audio stream obtained by a target device through audio acquisition; performing speech recognition on the audio segments acquired respectively within the plurality of consecutive preset time intervals to obtain recognized text of the plurality of audio segments; and determining the audio stream end time of the audio stream based on the acoustic attributes and / or the recognized text.

[0007] According to another aspect of the present invention, an audio stream processing apparatus is also provided, comprising: a first detection module, configured to perform acoustic detection on audio segments acquired within a plurality of consecutive preset time intervals to obtain acoustic properties of the plurality of audio segments; wherein the audio segments are audio segments included in an audio stream obtained by a target device through audio acquisition; a second detection module, configured to perform speech recognition on the audio segments acquired within the plurality of consecutive preset time intervals to obtain recognized text of the plurality of audio segments; and a determination module, configured to determine the audio stream end time of the audio stream based on the acoustic properties and / or the recognized text.

[0008] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the above-described audio stream processing method at runtime.

[0009] According to another aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the audio stream processing method described above through the computer program.

[0010] This invention performs acoustic detection and speech recognition on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic attributes and recognized text of the audio segments. The audio segments are those included in an audio stream obtained through continuous audio acquisition when the target device is activated. The end time of the audio stream is then determined based on the acoustic attributes and / or the recognized text. This technical solution improves the accuracy of determining the end time of an audio stream and solves the problem of low accuracy in determining the end time of an audio stream. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the hardware environment for an audio stream processing method according to an embodiment of this application;

[0014] Figure 2This is a flowchart of an audio stream processing method according to an embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram of a scenario for an audio stream processing method according to an embodiment of the present invention (I);

[0016] Figure 4 This is a schematic diagram (II) of an audio stream processing method according to an embodiment of the present invention;

[0017] Figure 5 This is a schematic diagram (iii) of an audio stream processing method according to an embodiment of the present invention;

[0018] Figure 6 This is a schematic diagram of a scenario for an audio stream processing method according to an embodiment of the present invention (IV);

[0019] Figure 7 This is a schematic diagram (five) of an audio stream processing method according to an embodiment of the present invention;

[0020] Figure 8 This is a structural block diagram of an audio stream processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] According to one aspect of the embodiments of this application, an audio stream processing method is provided. This audio stream processing method is widely used in whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligencehouse ecosystems. Optionally, in this embodiment, the above-mentioned audio stream processing method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0024] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0025] To address the aforementioned issues, this embodiment provides an audio stream processing method, including but not limited to applications in cloud servers or terminals.

[0026] It should be noted that if the terminal has chip computing power, storage space, and other configurations, meaning the terminal can detect the last frame of the audio stream, then the technical solution of this application embodiment can be applied to the terminal. If the terminal has a lower configuration and does not have the ability to detect the last frame of the audio stream, then the technical solution of this application embodiment can be applied to the cloud server. The cloud server will use the last frame detection to determine the user's voice input status. When it is determined that the user's input status is finished, it needs to notify the terminal device to turn off the microphone switch and instruct the terminal device to respond to the interaction result of the cloud server. When it is determined that the user's input status is in progress, it needs to continue to receive audio and record the voice recognition result.

[0027] Specifically, Figure 2 This is a flowchart of an audio stream processing method according to an embodiment of the present invention, the process including the following steps:

[0028] Step S202: Perform acoustic detection on the audio segments acquired in multiple consecutive preset time intervals to obtain the acoustic properties of the multiple audio segments; wherein, the audio segments are the audio segments included in the audio stream obtained by continuous audio acquisition when the target device is woken up;

[0029] In an exemplary embodiment, step S202 can be implemented as follows: during the acquisition of the audio stream, acoustic detection is performed on the preset number of audio segments acquired within the preset number of preset time intervals at preset intervals.

[0030] It should be noted that the above preset number can be one or more, and the preset time interval between the preset number of presets must be less than the sixth preset threshold.

[0031] In a preferred embodiment, the preset number is 1, meaning that acoustic detection is performed on the audio segments acquired within the preset time interval at preset time intervals. It should be noted that the preset time interval can be set to 350ms, meaning that an audio segment is obtained every 350ms, and acoustic detection is performed on the audio segments acquired within the first 350ms.

[0032] In an exemplary embodiment, step S202 above can be implemented by inputting each audio segment into an acoustic detection model for acoustic detection to obtain one or more combinations of the following acoustic properties contained in each audio segment: human voice confidence, noise confidence, silence confidence, audio start time, and audio end time.

[0033] In a preferred embodiment, at preset time intervals, audio segments acquired within the preset time interval are input into the acoustic detection model for detection to obtain the voice confidence level, noise confidence level, silence confidence level, audio start time, and audio end time of each audio segment. It should be noted that the acoustic attributes of each audio segment include: voice confidence level, noise confidence level, silence confidence level, audio start time, and audio end time.

[0034] It should be noted that for an audio segment, the sum of its corresponding confidence scores for human voice, noise, and silence approaches 1.

[0035] Step S204: Perform speech recognition on the audio segments acquired within multiple preset time intervals to obtain the recognized text of the multiple audio segments;

[0036] It should be noted that steps S202 and S204 are executed asynchronously. In an exemplary example, step S204 can be implemented as follows: during the acquisition of the audio stream, speech recognition is performed on the preset number of audio segments acquired within the preset number of preset time intervals.

[0037] Step S206: Determine the audio stream end time of the audio stream based on the acoustic attributes and / or the identified text, wherein the audio stream end time is the time when the target device stops collecting audio.

[0038] Through the above steps, acoustic detection and speech recognition are performed on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic attributes and recognized text of the audio segments. The audio segments are those included in the audio stream obtained by continuous audio acquisition when the target device is awakened. Then, the end time of the audio stream is determined based on the acoustic attributes and / or the recognized text. This technical solution improves the accuracy of determining the end time of the audio stream and solves the problem of low accuracy in determining the end time of the audio stream.

[0039] In an exemplary embodiment, step S204 above can be implemented by the following steps S11-S12:

[0040] Step S11: Determine the segment type of each audio segment in the N audio segments and the total duration of some or all audio segments in the N audio segments according to the acoustic properties, where N is a positive integer;

[0041] It should be noted that during the processing of the audio stream, N preset time intervals have passed from the moment the audio stream is first received to the current moment, that is, N audio segments have been acquired. Then, the segment type of each audio segment and the total duration of some or all of the audio segments in the N audio segments can be determined based on the acoustic properties of each audio segment.

[0042] Optionally, the above-mentioned determination of the segment type of each audio segment among N audio segments based on the acoustic properties can be achieved in the following ways: if the confidence level of human voice in the acoustic properties of the audio segment is greater than or equal to a first preset threshold, the audio segment is determined to be a human voice audio segment; if the confidence level of noise in the acoustic properties of the audio segment is greater than or equal to a second preset threshold, the audio segment is determined to be a noise audio segment; if the confidence level of silence in the acoustic properties of the audio segment is greater than or equal to a third preset threshold, the audio segment is determined to be a silence audio segment.

[0043] It should be noted that the first preset threshold can be 0.7, the second preset threshold can be 0.3, and the third preset threshold can be 0.3. For an audio segment, the sum of its corresponding human voice confidence, noise confidence, and silence confidence infinitely approaches 1.

[0044] Step S12: Determine the audio end time of the audio stream based on the segment type of each audio segment and the total duration of all audio segments in the N audio segments, or determine the audio end time of the audio stream based on the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the identified text.

[0045] Optionally, the above-mentioned determination of the audio end time of the audio stream based on the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the recognized text can be achieved in the following way: when the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the recognized text satisfy a third preset condition, the audio end time of the Nth audio segment is determined as the audio stream end time, wherein the third preset condition includes: in the N audio segments, the Mth audio segment is a human voice audio segment, the audio segments after the Mth audio segment are all silent audio segments, and the (M+1)th audio segment is a human voice audio segment. The total audio duration of the first to the Mth audio segments is greater than or equal to a sixth preset threshold; the total duration of the first to the Mth audio segments without any consecutive silent or noisy audio segments is greater than or equal to a fourth preset threshold; the total duration of the first to the Mth audio segments without any consecutive silent or noisy audio segments is greater than or equal to a fifth preset threshold; and the total duration of the first to the Mth audio segments without any consecutive silent audio segments following a human voice audio segment is greater than or equal to the sixth preset threshold; the recognition text corresponding to the first to the Mth audio segments does not contain any related words corresponding to the target device; M is greater than or equal to 1 and less than N; and N is a positive integer.

[0046] Furthermore, in the third preset condition, the total audio duration of the M+1th audio segment to the N-1th audio segment is less than the sixth preset threshold.

[0047] To better understand, the following explanation is provided: If a user does not speak for a duration of the sixth preset threshold after finishing speaking, it can be determined that the user has finished speaking, and at this time the device can be controlled to stop audio acquisition.

[0048] It should be noted that the sixth preset threshold is the scene silence threshold, which is optional and can be between 350ms and 500ms.

[0049] To better understand, such as Figure 3 As mentioned above, in Figure 3 In this context, the end time of f2 is the end time of the Mth audio segment, meaning that after f2, the total duration of f3 is greater than or equal to the sixth preset threshold. That is, the user ends speaking at the end time of f2.

[0050] It should be noted that, in Figures 3 to 7 f1, f2, and f3 in the text include one or more audio segments. Figure 3 In f2, there is a silent audio segment, but the total duration of this silent audio segment is less than the first preset threshold. This can be interpreted as a pause by the user, not the end of the user's speech. It should be noted that there are no related words for the target device in the recognized text corresponding to f1 and f2.

[0051] Furthermore, in one scenario, the user wakes up the device but does not immediately engage in voice input, for example, such as... Figure 4 As shown, f1 and f2 are both continuous silent audio segments, but the total duration of f1 and f2 does not exceed the fourth preset threshold. Therefore, it cannot be determined that the user's voice input has ended, and audio acquisition needs to continue. However, the total duration of the continuous silent audio segments after f3 (after the human voice audio segment) is greater than or equal to the sixth preset threshold. Therefore, it is determined that the user's speech ends at the end of f3. It should be noted that the aforementioned fourth and fifth preset thresholds are both greater than or equal to the sixth preset threshold. There are no related words for the target device in the recognition text corresponding to f3.

[0052] In addition, Figure 5In this scenario, during user voice input, machine noise or other background sounds occur, meaning there are continuous noise audio segments in f2. However, the total duration of these continuous noise audio segments is less than the fourth preset threshold, making it impossible to determine whether the user's voice input has ended. Continuous audio acquisition is necessary. Then, the total duration of the continuous silent audio segments following f2 (after the human voice audio segment) is greater than or equal to the sixth preset threshold, thus determining that the user's speech has ended at the end of f2. It should be noted that there are no related words for the target device in the recognition text corresponding to f1 and f2.

[0053] Optionally, the above-mentioned determination of the audio end time of the audio stream based on the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the identified text can also be implemented in the following way: when the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the identified text satisfy a fourth preset condition, the audio end time of the Nth audio segment is determined as the audio stream end time, wherein the fourth preset condition includes: the identified text corresponding to the Nth audio segment contains a word associated with the target device, or the combined identified text corresponding to the Nth audio segment and the audio segments preceding the Nth audio segment contains a word associated with the target device; the total duration of no continuous silent audio segments or continuous noise audio segments between the 1st audio segment and the Nth audio segment is greater than or equal to a fourth preset threshold; the total duration of no continuous silent audio segments and noise audio segments is greater than or equal to a fifth preset threshold, and the total duration of no continuous silent audio segments following the human voice audio segment is greater than or equal to a sixth preset threshold; and N is a positive integer.

[0054] It should be noted that when users control smart devices with their voice, they will use certain fixed phrases to express their control intentions in the application scenario. Keywords in the phrases are extracted according to their frequency of occurrence in the scenario to form high-frequency related words. For example, in the air conditioning application scenario, common phrases include "turn on the air conditioner" and "turn off the air conditioner". When the voice recognition text matches the high-frequency phrase keywords, it is determined to be the end.

[0055] It should be noted that, during the acquisition of the audio stream, an audio segment is determined at preset time intervals. As a result, the recognition text of an audio segment may contain complete related words, and the recognition texts corresponding to multiple consecutive audio segments may also contain complete related words. Therefore, as long as the recognition text corresponding to the Nth audio segment contains a related word corresponding to the target device, or the combined recognition text of the Nth audio segment and the audio segments preceding the Nth audio segment contains a related word corresponding to the target device, the audio stream can be determined to have ended.

[0056] Optionally, the audio end time of the audio stream can be determined based on the segment type of each audio segment and the total duration of all audio segments in the N audio segments. This can be achieved as follows: when the segment type of each audio segment and the total duration of all audio segments in the N audio segments meet a first preset condition, the audio end time of the Nth audio segment is determined as the end time of the audio stream. The first preset condition includes: all N audio segments are silent audio segments, the total audio duration of the N audio segments is greater than or equal to a fourth preset threshold, and N is a positive integer.

[0057] It should be noted that the first preset condition also includes: in the N audio segments, the total audio duration from the first audio segment to the (N-1)th audio segment is less than the fourth preset threshold.

[0058] It should be noted that if the user remains silent for more than the fourth preset threshold after waking up the device, the recording will be considered to have ended. Figure 6 As shown, Figure 5 If the total audio duration of f1-f5 is greater than or equal to the fourth preset threshold, then the end time of f5 can be determined as the end time of the audio stream.

[0059] Optionally, determining the audio end time of the audio stream based on the segment type of each audio segment and the total duration of all audio segments in the N audio segments can be achieved in the following way: when the segment type of each audio segment and the total duration of all audio segments in the N audio segments meet a second preset condition, the audio end time of the Nth audio segment is determined as the audio stream end time, wherein the second preset condition includes: the N audio segments are composed of noise audio segments or the N audio segments are composed of both silent audio segments and noise audio segments, the total audio duration of the N audio segments is greater than or equal to a fifth preset threshold, and N is a positive integer.

[0060] It should be noted that the second preset condition mentioned above also includes: in N audio segments, the total audio duration from the first audio segment to the (N-1)th audio segment is less than the fifth preset threshold.

[0061] It should be noted that the following scenario exists: after the device is woken up, the user does not speak, but there is intermittent noise in the environment. Therefore, the detected audio segment type is either a noise audio segment, or a combination of continuous silence and noise audio segments. Specifically, such as... Figure 7 As mentioned above, in Figure 7 If the total audio duration of f1-f5 is greater than or equal to the fifth preset threshold, then the end time of f5 can be determined as the end time of the audio stream.

[0062] Obviously, the embodiments described above are merely some embodiments of the present invention, and not all embodiments. To better understand the above method, the following description, in conjunction with embodiments, illustrates the process, but is not intended to limit the technical solutions of the embodiments of the present invention. Specifically:

[0063] In this embodiment of the application, taking the above solution applied to a cloud server as an example, by sorting out the user interaction scenario and the cloud speech recognition processing flow, the algorithm and rule constraints required for cloud server tail frame detection are determined, the terminal tail endpoint detection effect is compared, and algorithm evaluation indicators and rule calculation strategies are established.

[0064] The cloud server-based speech end-frame detection solution addresses the following five scenarios:

[0065] 1. There are silent segments in the human voice. For example, when the user inputs voice, there are pauses or poor sound pickup. Specifically, such as... Figure 3 As shown, in this scenario: for audio segment f1, the voice confidence score (p_score) is higher than the threshold of 0.85, and the speech recognition text did not match the keywords; for audio segment f2, the voice confidence score is in the range of 0.7 to 0.85, the silence confidence score is less than 0.3, and the noise confidence score is less than 0.3; for audio segment f3, the silence time is greater than the scene's silence threshold. Based on the rule constraints and the end-frame judgment strategy, the speech end point is calculated as the end time of segment f2. It should be noted that the speech end point is used to indicate the time when the user ends speaking. After the cloud server determines the speech end point, it immediately instructs the device to stop audio acquisition.

[0066] 2. Silent segments appear before or after the user's voice. For example, the user does not immediately input voice after being woken up, and there are long pauses in the input during the voice expression. Figure 4 As shown, in this scenario, audio segments f1 and f2 are both continuous silences with a silence confidence score (q_score) greater than 0.75, and their frequency of consecutive occurrence is less than the frequency of silences in the scene. For audio segment f3, the voice confidence score is greater than 0.85, and the speech recognition text did not match any keywords. For audio segment f4, the silence duration is greater than the scene's silence threshold. Based on the rule constraints and the end-frame judgment strategy, the speech termination point is calculated as the end time of segment f3.

[0067] 3. The human voice contains noisy segments. For example, during the user's voice input, there may be machine noises or other background sounds. Figure 5As shown, in this scenario, audio segment f1 has a voice confidence score greater than 0.85, and the speech recognition text did not match the keywords; audio segment f2 has a voice confidence score less than 0.7 and a noise confidence score greater than 0.3; audio segment f3 has a silence time greater than the scene's silence threshold. Based on the rule constraints and the end-frame judgment strategy, the speech end point is calculated as the end time of segment f2.

[0068] 4. Continuous periods of silence. For example: after the user wakes up, there is no voice input for an extended period or abnormal sound reception. Figure 6 As shown, in this scenario, audio segments f1-f5 are all silent segments, and their frequency of occurrence exceeds the scene's silence frequency threshold. Based on the rule constraints and the end-frame judgment strategy, the end point of the speech is calculated as the end time of segment f5.

[0069] 5. Continuous occurrence of silence and noise. For example: After the speaker is activated by the user, a loud background noise appears. Figure 7 As shown, in this scenario, audio segments f1-f5 consist of consecutive occurrences of silence and noise, with the duration exceeding the scene noise threshold. Based on the rule constraints and the end-frame determination strategy, the speech termination point is calculated as the end time of segment f5.

[0070] It should be noted that during the process of acquiring the audio stream sent by the target device, the cloud server will detect the audio segments acquired within each preset time interval. This involves inputting the audio segments into both the acoustic model and the speech recognition model for detection. The acoustic model then outputs: start time, end time, voice confidence score, noise confidence score, and silence confidence score. The speech recognition model outputs intermediate results: word text, start time, and end time. Furthermore, for multiple preset time intervals, the speech recognition module will output complete results: word text, word start time, and word end time.

[0071] It should be noted that the following rules apply:

[0072] (1) High-frequency dialogue keyword list (equivalent to the related words in the above embodiment): When users control smart devices with voice, they will use some fixed dialogues to express their control intentions in the application scenario. The keywords in the dialogues are extracted according to the frequency of occurrence in the scenario to form a high-frequency keyword list. For example, in the air conditioning application scenario, the more common dialogues are "turn on the air conditioner" and "turn off the air conditioner";

[0073] (2) Scene silence threshold: The range of silence time before and after voice input after the user wakes up;

[0074] (3) Scene silence frequency: If silence occurs N times in a row, it is considered invalid voice input and the recording should be stopped;

[0075] (4) Scene noise threshold: If silence or noise occurs continuously after the user wakes up, it is considered invalid voice input and the recording should be stopped.

[0076] End frame determination strategy:

[0077] (1) When the speech recognition text matches a high-frequency utterance keyword, the process is considered complete. For example: play music, turn on the air conditioner, etc.

[0078] (2) When a silence occurs, if it is continuous and the duration exceeds (scene silence threshold * scene silence frequency), it is determined to be the end;

[0079] (3) When the duration of silence or noise exceeds the scene noise threshold, it is determined to end;

[0080] (4) When a human voice appears and then the silence time exceeds the scene's silence threshold, the process is considered to end.

[0081] It should be noted that the confidence level for human voice detection is 0.7–0.85, the confidence level for noise detection is 0.3–0.75, the confidence level for silence detection is 0.3–0.75, and the scene silence threshold range is 350ms–500ms.

[0082] It should be noted that, through the cloud-based voice tail frame detection in this application embodiment, users can obtain the same voice interaction experience as high-configuration devices even when using some lower-configuration devices (without a terminal voice detection module). Furthermore, compared to the updates and version iterations of terminal acoustic model algorithms, the cloud can update the algorithm model without the user's awareness and adjust the rule constraint strategies according to the device usage scenario.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0084] This embodiment also provides an audio stream processing device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0085] Figure 8 This is a structural block diagram of an audio stream processing apparatus according to an embodiment of the present invention, the apparatus comprising:

[0086] The first detection module 82 is used to perform acoustic detection on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic properties of the multiple audio segments; wherein, the audio segments are audio segments included in the audio stream obtained by the target device through audio acquisition;

[0087] The second detection module 84 is used to perform speech recognition on the audio segments obtained in the multiple consecutive preset time intervals to obtain the recognized text of the multiple audio segments.

[0088] The determining module 86 is used to determine the audio stream end time of the audio stream based on the acoustic properties and / or the identified text.

[0089] Using the aforementioned device, acoustic detection and speech recognition are performed on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic attributes and recognized text of the audio segments. The audio segments are those included in an audio stream obtained through continuous audio acquisition when the target device is awakened. Then, the end time of the audio stream is determined based on the acoustic attributes and / or the recognized text. This technical solution improves the accuracy of determining the end time of the audio stream and solves the problem of low accuracy in determining the end time of an audio stream.

[0090] In an exemplary embodiment, the first detection module 82 is configured to perform acoustic detection on a preset number of audio segments acquired within a preset number of preset time intervals during the process of acquiring the audio stream sent by the target device.

[0091] In an exemplary embodiment, the first detection module 84 is further configured to input each of the audio segments into the acoustic detection model for acoustic detection, so as to obtain one or more combinations of the following acoustic properties contained in each audio segment: human voice confidence, noise confidence, silence confidence, audio start time, and audio end time.

[0092] In an exemplary embodiment, the determining module 86 is configured to determine the segment type of each of the N audio segments and the total duration of some or all of the N audio segments based on the acoustic properties, wherein N is a positive integer; determine the audio end time of the audio stream based on the segment type of each audio segment and the total duration of all the audio segments in the N audio segments, or determine the audio end time of the audio stream based on the segment type of each audio segment, the total duration of some of the audio segments in the N audio segments, and the identified text.

[0093] In an exemplary embodiment, the determining module 86 is configured to determine the audio segment as a human voice audio segment if the confidence level of human voice in the acoustic properties of the audio segment is greater than or equal to a first preset threshold; determine the audio segment as a noise audio segment if the confidence level of noise in the acoustic properties of the audio segment is greater than or equal to a second preset threshold; and determine the audio segment as a silent audio segment if the confidence level of silence in the acoustic properties of the audio segment is greater than or equal to a third preset threshold.

[0094] In an exemplary embodiment, the determining module 86 is configured to determine the audio end time of the Nth audio segment as the audio stream end time when the segment type of each audio segment and the total duration of all audio segments in the N audio segments meet a first preset condition, wherein the first preset condition includes: all N audio segments are silent audio segments, the total audio duration of the N audio segments is greater than or equal to a fourth preset threshold, and N is a positive integer.

[0095] In an exemplary embodiment, the determining module 86 is configured to determine the audio end time of the Nth audio segment as the audio stream end time when the segment type of each audio segment and the total duration of all audio segments in the N audio segments meet a second preset condition. The second preset condition includes: the N audio segments are composed of noise audio segments or the N audio segments are composed of both silent audio segments and noise audio segments; the total audio duration of the N audio segments is greater than or equal to a fifth preset threshold; and N is a positive integer.

[0096] In an exemplary embodiment, the determining module 86 is configured to determine the audio end time of the Nth audio segment as the audio stream end time when the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the identified text meet a third preset condition. The third preset condition includes: in the N audio segments, the Mth audio segment is a human voice audio segment; all audio segments after the Mth audio segment are silent audio segments; and the total audio duration from the (M+1)th audio segment to the Nth audio segment is greater than or equal to the sixth audio segment. The following conditions are met: the total duration of the absence of consecutive silent audio segments or consecutive noisy audio segments between the first audio segment and the Mth audio segment is greater than or equal to a fourth preset threshold; the total duration of the absence of consecutive silent audio segments and noisy audio segments is greater than or equal to a fifth preset threshold; and the total duration of the absence of consecutive silent audio segments following a human voice audio segment is greater than or equal to a sixth preset threshold. Furthermore, the recognition text corresponding to the first audio segment to the Mth audio segment does not contain any associated words corresponding to the target device. M is greater than or equal to 1 and less than N, where N is a positive integer.

[0097] In an exemplary embodiment, the determining module 86 is configured to determine the audio end time of the Nth audio segment as the audio stream end time when the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the identified text satisfy a fourth preset condition. The fourth preset condition includes: the identified text corresponding to the Nth audio segment contains a word associated with the target device, or the combined identified text corresponding to the Nth audio segment and the audio segments preceding the Nth audio segment contains a word associated with the target device; the total duration of no consecutive silent audio segments or consecutive noise audio segments between the 1st and Nth audio segments is greater than or equal to a fourth preset threshold; the total duration of no consecutive silent audio segments and noise audio segments is greater than or equal to a fifth preset threshold, and the total duration of no consecutive silent audio segments following a human voice audio segment is greater than or equal to a sixth preset threshold; and N is a positive integer.

[0098] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0099] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0100] S1, perform acoustic detection on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic properties of the audio segments; wherein, the audio segments are audio segments included in the audio stream obtained by the target device during audio acquisition; and

[0101] S2, perform speech recognition on the audio segments obtained in the multiple consecutive preset time intervals to obtain the recognized text of the multiple audio segments;

[0102] S3, determine the audio stream end time of the audio stream based on the acoustic properties and / or the identified text.

[0103] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0104] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0105] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0106] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0107] S1, perform acoustic detection on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic properties of the audio segments; wherein, the audio segments are audio segments included in the audio stream obtained by the target device during audio acquisition; and

[0108] S2, perform speech recognition on the audio segments obtained in the multiple consecutive preset time intervals to obtain the recognized text of the multiple audio segments;

[0109] S3, determine the audio stream end time of the audio stream based on the acoustic properties and / or the identified text.

[0110] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0111] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0112] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0113] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An audio stream processing method, characterized in that, include: Acoustic detection is performed on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic properties of the audio segments; wherein, the audio segments are audio segments included in the audio stream obtained by the target device during audio acquisition; and Speech recognition is performed on the audio segments acquired within the multiple consecutive preset time intervals to obtain the recognized text of the multiple audio segments; The audio stream end time is determined based on the acoustic properties and / or the identified text; Determining the audio stream end time based on the acoustic properties and / or the recognized text includes: Based on the acoustic properties, determine the segment type of each of the N audio segments and the total duration of some or all of the N audio segments, where N is a positive integer; The audio end time of the audio stream can be determined based on the segment type of each audio segment and the total duration of all audio segments in the N audio segments, or based on the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the recognized text. The process of determining the audio end time of the audio stream based on the segment type of each audio segment and the total duration of all audio segments in the N audio segments includes: If the segment type of each audio segment and the total duration of all audio segments in the N audio segments meet the first preset condition, the audio end time of the Nth audio segment is determined as the audio stream end time. The first preset condition includes: all N audio segments are silent audio segments, the total audio duration of the N audio segments is greater than or equal to a fourth preset threshold, and N is a positive integer.

2. The method according to claim 1, characterized in that, Acoustic detection is performed on audio segments acquired within multiple consecutive preset time intervals, including: During the acquisition of the audio stream, acoustic detection is performed on the preset number of audio segments acquired within the preset number of preset time intervals.

3. The method according to claim 1, characterized in that, Acoustic detection is performed on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic properties of the audio segments, including: Each audio segment is input into an acoustic detection model for acoustic detection to obtain one or more combinations of the following acoustic attributes contained in each audio segment: human voice confidence, noise confidence, silence confidence, audio start time, and audio end time.

4. The method according to claim 1, characterized in that, Based on the acoustic properties, determine the segment type of each of the N audio segments, including: If the confidence level of human voice in the acoustic properties of the audio segment is greater than or equal to a first preset threshold, the audio segment is determined to be a human voice audio segment. If the noise confidence level in the acoustic properties of the audio segment is greater than or equal to a second preset threshold, the audio segment is determined to be a noisy audio segment. If the silence confidence level in the acoustic properties of the audio segment is greater than or equal to a third preset threshold, the audio segment is determined to be a silent audio segment.

5. The method according to claim 1, characterized in that, Determining the audio end time of the audio stream based on the segment type of each audio segment and the total duration of all audio segments in the N audio segments also includes: If the segment type of each audio segment and the total duration of all audio segments in the N audio segments meet the second preset condition, the audio end time of the Nth audio segment is determined as the audio stream end time. The second preset condition includes: the N audio segments are composed of noise audio segments or the N audio segments are composed of both silent audio segments and noise audio segments; the total audio duration of the N audio segments is greater than or equal to a fifth preset threshold; and N is a positive integer.

6. The method according to claim 1, characterized in that, Determining the audio end time of the audio stream based on the segment type of each audio segment, the total duration of some audio segments among the N audio segments, and the recognized text includes: If the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the identified text satisfy the third preset condition, the audio end time of the Nth audio segment is determined as the audio stream end time. The third preset condition includes: among the N audio segments, the Mth audio segment is a human voice audio segment; all audio segments after the Mth audio segment are silent audio segments; the total audio duration from the (M+1)th audio segment to the Nth audio segment is greater than or equal to a sixth preset threshold; between the 1st audio segment and the Mth audio segment, there is no continuous silent audio segment or continuous noise audio segment with a total duration greater than or equal to a fourth preset threshold; there is no continuous silent audio segment or noise audio segment with a total duration greater than or equal to a fifth preset threshold; and there is no continuous silent audio segment after the human voice audio segment with a total duration greater than or equal to the sixth preset threshold; the identified text corresponding to the 1st audio segment to the Mth audio segment does not contain any related words corresponding to the target device; M is greater than or equal to 1 and less than N; and N is a positive integer.

7. The method according to claim 1, characterized in that, Determining the audio end time of the audio stream based on the segment type of each audio segment, the total duration of some audio segments among the N audio segments, and the recognized text includes: When the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the identified text satisfy the fourth preset condition, the audio end time of the Nth audio segment is determined as the audio stream end time. The fourth preset condition includes: the identified text corresponding to the Nth audio segment contains a word associated with the target device, or the combined identified text corresponding to the Nth audio segment and the audio segments preceding the Nth audio segment contains a word associated with the target device; the total duration of no continuous silent audio segments or continuous noise audio segments between the 1st audio segment and the Nth audio segment is greater than or equal to the fourth preset threshold; the total duration of no continuous silent audio segments and noise audio segments is greater than or equal to the fifth preset threshold, and the total duration of no continuous silent audio segments following the human voice audio segment is greater than or equal to the sixth preset threshold; and N is a positive integer.

8. An audio stream processing device, characterized in that, include: The first detection module is used to perform acoustic detection on audio segments acquired within multiple consecutive preset time intervals to obtain the acoustic properties of the multiple audio segments; wherein, the audio segments are audio segments included in the audio stream obtained by the target device through audio acquisition; The second detection module is used to perform speech recognition on the audio segments obtained in the multiple consecutive preset time intervals to obtain the recognized text of the multiple audio segments. A determining module is used to determine the audio stream end time of the audio stream based on the acoustic properties and / or the identified text; The determining module is further configured to determine the segment type of each audio segment among the N audio segments and the total duration of some or all audio segments among the N audio segments based on the acoustic properties, wherein N is a positive integer; The audio end time of the audio stream can be determined based on the segment type of each audio segment and the total duration of all audio segments in the N audio segments, or based on the segment type of each audio segment, the total duration of some audio segments in the N audio segments, and the recognized text. The determining module is further configured to determine the audio end time of the Nth audio segment as the audio stream end time when the segment type of each audio segment and the total duration of all audio segments in the N audio segments meet a first preset condition. The first preset condition includes: all N audio segments are silent audio segments, the total audio duration of the N audio segments is greater than or equal to a fourth preset threshold, and N is a positive integer.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 7 through the computer program.

Citation Information

Patent Citations

  • Voice endpoint judgment method and device, equipment, storage medium and product

    CN114495981A