Voice information processing method, related method, device and storage medium

Abstract

The invention discloses a voice information processing method, a related method, a related device and a storage medium, and the method comprises the steps: obtaining a sound pressure characterization value of an original audio frame, and obtaining a mask characterization value of the original audio frame based on an output mask of a mask layer in a voice model when the original audio frame is processed through the voice model; wherein the output mask comprises mask values of the original audio frame at a plurality of frequency points, and the mask values are first numerical values representing noise or second numerical values representing voice; and based on the sound pressure characterization value and the mask characterization value, determining whether the original audio frame is used as a target audio frame to be transmitted to the opposite communication end. According to the scheme, whether each frequency point in the original audio frame belongs to noise or voice is distinguished through the voice model, and the target audio frame to be transmitted to the opposite communication end is determined in combination with the sound pressure characterization value and the mask characterization value, so that the accuracy and efficiency of voice communication awakening can be improved on the premise of realizing non-inductive voice communication awakening.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech information processing method and related methods, devices and storage media. Background Art

[0002] In a voice communication system, when a user has a communication need, he or she often needs to press and hold an activation button on a call device (eg, Push-To-Talk on a walkie-talkie) to wake up the device and then conduct voice communication. Otherwise, communication is not possible.

[0003] However, in real-life scenarios, key operations may not be possible in some situations, such as when the driver is driving or when the service personnel are providing services. In view of this, it is urgent to realize non-sensory voice communication wake-up, while at the same time ensuring the accuracy and efficiency of voice communication wake-up as much as possible. In view of this, how to improve the accuracy and efficiency of voice communication wake-up while realizing non-sensory voice communication wake-up has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a voice information processing method and related methods, devices and storage media, which can improve the accuracy and efficiency of voice communication wake-up under the premise of realizing seamless voice communication wake-up.

[0005] In order to solve the above-mentioned technical problems, the first aspect of the present application provides a speech information processing method, including: obtaining the sound pressure representation value of the original audio frame, and obtaining the mask representation value of the original audio frame based on the output mask of the mask layer in the speech model when processing the original audio frame based on the speech model; wherein the output mask includes the mask values ​​of the original audio frame at several frequency points, and the mask value is a first value representing the noise or a second value representing the speech; based on the sound pressure representation value and the mask representation value, determining whether to use the original audio frame as the target audio frame to be transmitted to the communication counterpart.

[0006] To solve the above-mentioned technical problems, the second aspect of the present application provides a communication wake-up method, including: obtaining each audio frame collected by the first communication device as an original audio frame respectively; determining whether the original audio frame is a target audio frame to be transmitted to the communication counterpart; wherein, the original audio frame is determined to be a target audio frame based on the voice information processing method of the first aspect above; based on whether the original audio frame is the target audio frame, determining whether to send the original audio frame to a second communication device that is communicatively connected to the first communication device.

[0007] In order to solve the above-mentioned technical problems, the third aspect of the present application provides a speech information processing device, including an acquisition module and a determination module. The acquisition module is used to obtain the sound pressure representation value of the original audio frame, and obtain the mask representation value of the original audio frame based on the output mask of the mask layer in the speech model when processing the original audio frame based on the speech model; wherein the output mask includes the mask values ​​of the original audio frame at several frequency points, and the mask value is a first value representing the noise or a second value representing the speech; the determination module is used to determine whether to use the original audio frame as the target audio frame to be transmitted to the communication counterpart based on the sound pressure representation value and the mask representation value.

[0008] To solve the above-mentioned technical problems, the fourth aspect of the present application provides a communication wake-up device, including an acquisition module and a judgment module, the acquisition module is used to obtain each audio frame collected by the first communication device as an original audio frame; the judgment module is used to determine whether the original audio frame is a target audio frame to be transmitted to the communication counterpart; wherein the original audio frame is determined to be a target audio frame based on the voice information processing device in the above-mentioned third aspect; the sending module is used to determine whether to send the original audio frame to the second communication device connected to the first communication device based on whether the original audio frame is the target audio frame.

[0009] In order to solve the above-mentioned technical problems, the fifth aspect of the present application provides an electronic device, including a processor and a memory, the memory is coupled to the processor, and a computing program is stored in the memory to implement the voice information processing method of the first aspect or the communication wake-up method of the second aspect.

[0010] In order to solve the above technical problems, the sixth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program is processed and executed to implement the voice information processing method of the first aspect or the communication wake-up method of the second aspect.

[0011] In the above scheme, the sound pressure characterization value of the original audio frame is obtained, and the mask characterization value of the original audio frame is obtained based on the output mask of the mask layer in the speech model when the original audio frame is processed based on the speech model, and the output mask includes the mask values ​​of the original audio frame at several frequency points, and the mask value is a first value representing the noise or a second value representing the speech. Then, based on the sound pressure characterization value and the mask characterization value, it is determined whether to use the original audio frame as the target audio frame to be transmitted to the communication peer. Therefore, on the one hand, since it is automatically determined whether to transmit the original audio frame to the communication peer without manually pressing a button to confirm whether to conduct voice communication, it can achieve non-sensitive voice communication wake-up. On the other hand, The mask characterization value is obtained by the output mask, and the output mask contains a mask value that characterizes the frequency point as noise or speech. Therefore, the mask characterization value and the sound pressure characterization value are combined to jointly determine whether to transmit the original audio frame to the communication counterpart, which can minimize the false wake-up of voice communication caused by background noise, burst noise and other noises, and help improve the accuracy of voice communication wake-up. On the other hand, because the above-mentioned joint determination process can automatically determine whether to transmit the original audio frame to the communication counterpart, the communication device can enter the wake-up state and transmit the audio frame only when communication transmission is required, and the transmission of invalid information can be reduced as much as possible, which helps to improve the efficiency of voice communication wake-up. Therefore, the accuracy and efficiency of voice communication wake-up can be improved under the premise of realizing non-perceptible voice communication wake-up. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of an embodiment of the voice information processing method of the present application;

[0013] Figure 2 This is a flow chart of an embodiment of the communication wake-up method of the present application;

[0014] Figure 3 It is a schematic diagram of the framework of an embodiment of a voice information processing device of the present application;

[0015] Figure 4 It is a schematic diagram of the framework of an embodiment of the communication awakening device of the present application;

[0016] Figure 5 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0017] Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0019] It should be understood that the reference to "embodiment" in this application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0020] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. The technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity, specific order or primary and secondary relationship of the indicated technical features.

[0021] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for processing voice information of the present application. The method comprises the following steps:

[0022] Step S11: Obtain the sound pressure characterization value of the original audio frame, and obtain the mask characterization value of the original audio frame based on the output mask of the mask layer in the speech model when processing the original audio frame based on the speech model; wherein the output mask includes the mask values ​​of the original audio frame at several frequency points, and the mask value is a first value characterizing noise or a second value characterizing speech.

[0023] It should be noted that the audio frame is the basic unit for processing continuous audio signals, and the original audio frame can be an audio signal divided into a series of short time periods by framing and windowing the target audio information. The target audio information can be any audio information that needs to be processed. In a specific implementation scenario, the target audio information is the audio information collected by a device, and the target audio information is divided into multiple segments of fixed length, each of which is an audio frame. Windowing is to reduce the discontinuity between audio frames, and a window function is used to smooth the boundaries of audio frames to avoid the problem of spectrum leakage. Specifically, the window function can be a Hamming window, a Hanning window, etc. The length of the audio frame can be set according to demand, and this application is not limited here.

[0024] In addition, the speech model can be used to process the original audio frame. For example, the speech model can implement downstream tasks including but not limited to speech recognition and speech translation, which are not limited here. In the process of the speech model processing the original audio frame, the output mask of the mask layer in the speech model can be used to distinguish between speech and noise. For details, please refer to the following related description, which will not be repeated here. As a possible example, in some application scenarios, the speech model can use a feedforward sequential memory network (FSMN). The hidden layer of the feedforward memory network contains a memory module (memory block). The memory module can capture the front and back dependencies of the sequence data. While retaining the modeling ability of the recurrent neural network (RNN) for the front and back relationships of the sequence, it can also speed up the calculation speed of the model and reduce the calculation complexity. The specific principle can refer to the technical details of FSMN, which will not be repeated here. As another possible example, in some application scenarios, the speech model can also use a deep neural network, a recurrent neural network, etc. It should be noted that a mask layer can also be set in these neural networks to obtain the output mask through the mask layer.

[0025] In one implementation scenario, the sound pressure characterization value is a physical quantity used to reflect the sound intensity in an audio frame. In order to obtain the sound pressure characterization value of the original audio frame, the original audio frame can be first converted into the time-frequency domain to obtain the frequency domain data of the original audio frame, and then the energy value of the original audio frame can be obtained based on the energy value of each frequency point in the frequency domain data as the sound pressure characterization value of the original audio frame. It should be noted that the time-frequency domain conversion of the original audio frame refers to the process of converting the original audio frame signal from the time domain to the frequency domain. Specifically, the frequency domain data of the original audio frame can be obtained by performing Fourier transform, fast Fourier transform, etc. on the original audio frame signal.

[0026] In a specific implementation scenario, in a specific implementation scenario, the target device is a remote communication device, and the audio signal collected by the remote communication device is used as the target audio information x(n). After the target audio information x(n) recorded by the remote communication device is obtained, the target audio information x(n) can be framed and windowed into a plurality of target audio frames, and the original audio frames are subjected to fast Fourier transform to obtain the frequency domain data X(l , ω ) ＝ STFT(x(n)), where l is the frame number, ω is the frequency point number, and x(n) is the time domain audio.

[0027] In a specific implementation scenario, as a possible example, after obtaining the energy value of each frequency point in the frequency domain data, the sum or weighted sum of these energy values ​​can be calculated as the total energy value of the original audio frame, and then the total energy value can be mapped to the corresponding range of sound pressure levels to obtain the sound pressure representation value of the original audio frame.

[0028] In one implementation scenario, as another possible implementation method, in order to obtain the sound pressure characterization value of the target audio frame, the original audio frame can also be converted into the time-frequency domain to obtain frequency domain data; based on the energy value of each frequency point in the frequency domain data, the energy value of the original audio frame is obtained; based on the energy value of the original audio frame and the sound pressure bias value of the target device, the sound pressure characterization value of the original audio frame is obtained; wherein the target device is the acquisition device of the original audio frame. Specifically, the target device can be a local device that is communicatively connected to the peer device, and the target device can include but is not limited to a smart phone, a smart speaker, a vehicle-mounted communication device, a robot, a walkie-talkie, etc. The above method, by considering the device-specific sound pressure bias value, can make the sound pressure characterization value closer to the actual usage scenario and improve the accuracy of audio processing.

[0029] In a specific implementation scenario, the energy value of each frequency point in the frequency domain data can be obtained by calculating the square of the complex amplitude corresponding to each frequency point, and the energy value of each frequency point in the frequency domain data can be weighted averaged to obtain the energy value of the original audio frame.

[0030] In a specific implementation scenario, after obtaining the energy value of each frequency point in the frequency domain data, the A-weighted weight curve can be digitized first and converted into a series of discrete series. These values ​​correspond to the weighting factors of each frequency point in the frequency domain data of the original audio frame in the frequency domain. For each frequency point in the frequency domain data, the corresponding weighting factor can be found from the A-weighted weight curve, and the energy value of each frequency point is multiplied by the corresponding weighting factor to obtain the weighted energy value, and then the value obtained by adding the energy values ​​of all weighted frequency points is used as the energy value of the original audio frame. It should be noted that A-weighting is the most commonly used weighting method, which simulates the frequency response of the human ear to sound at a medium loudness level. A-weighting attenuates low-frequency and high-frequency sounds, making the weight of medium-frequency sounds greater. The horizontal axis of the weight curve based on A-weighting represents frequency (Hz) and the vertical axis represents weighting factor. Exemplarily, for each frequency point in the frequency domain data, the weighting factor corresponding to the frequency point can be found on the A-weighted weight curve. In addition, the specific weighting method can also adopt B-weighting, C-weighting, D-weighting, Z-weighting, etc. The specific weighting method can be set according to the actual use scenario. For example, in the application scenario of simulating the subjective perception of the human ear, A weighting can be used, or, in the application scenario where the sensitivity of the low frequency band needs to be increased, B weighting can be used. Of course, the above examples are only a few possible examples in the actual application process, and the method of obtaining the weighting factor is not limited thereto, and examples are not given one by one here. On this basis, the energy value of each frequency point in the frequency domain data can be weighted based on the weighting factor of each frequency point in the frequency domain data to obtain the energy value of the original audio frame. Specifically, the frequency domain data can be displayed using a spectrogram, with the horizontal axis being the time (frame) axis and the vertical axis being the frequency axis. The energy intensity of each frequency point is reflected by the color of the point determined by the coordinates on the horizontal and vertical axes. For example, the larger the energy value of the frequency point, the brighter the color of the frequency point. The above method weights each frequency point of the frequency domain data by a weighting factor, adjusts the contribution of the energy data of frequency points of different frequencies in the energy value of the original audio frame obtained in the end, adapts to different application scenarios, and helps to improve the adaptability of the energy value calculation of the original audio frame. Combined with the visual representation of the spectrogram, the frequency characteristics and energy changes of the audio signal can be intuitively observed, thereby improving the convenience of audio signal processing.

[0031] In a specific implementation scenario, the sound pressure bias value of the target device can be obtained in advance through testing. Specifically, the first audio frame collected by the target device and the second audio frame collected by the sound level meter when playing the test audio frame of the same sound pressure level can be obtained; the energy value of the first audio frame can be obtained, and the sound level value measured by the sound level meter for the second audio frame can be obtained; based on the sound level value and energy value at the same sound pressure level, the sound pressure bias value of the target device at the corresponding sound pressure level can be obtained; based on the sound pressure bias value of the target device at different sound pressure levels, the sound pressure bias value can be fused to obtain the final sound pressure bias value of the target device. In the above method, the target device and the sound level meter respectively detect the test audio frames of different sound pressure levels, so as to combine the sound level values ​​and energy values ​​detected by the two at the same sound pressure level, obtain the sound pressure bias value of the target device at the corresponding sound pressure level, and combine the sound pressure bias values ​​of the target device at different sound pressure levels to obtain the final sound pressure bias value of the target device, which can improve the calculation accuracy of the sound pressure bias value as much as possible.

[0032] In one implementation scenario, the difference between the sound level value and the energy value at the same sound pressure level is the sound pressure bias value of the target device at the corresponding sound pressure level. The sum of the energy value of the original audio frame and the sound pressure bias value of the target device at the corresponding sound pressure level can be used as the final sound pressure bias value of the target device.

[0033] In another implementation scenario, the difference between the energy value and the sound level value at the same sound pressure level can be used as the sound pressure bias value of the target device at the corresponding sound pressure level. The sum of the energy value of the original audio frame and the sound pressure bias value of the target device at the corresponding sound pressure level can be used as the final sound pressure bias value of the target device.

[0034] In a specific implementation scenario, the test audio frame includes at least a white noise signal, and the target device is a remote communication device. The target device and the sound level meter are used to collect the same white noise signal with a known sound pressure level by playing the same white noise signal. When playing the test audio frame, the microphone of the target device and the microphone of the sound level meter are deployed at the same position. The sound level value obtained by measuring a white noise signal at a position by the sound level meter is used as a reference standard, and the test audio collected by obtaining the same white noise signal measured by the remote communication device at the same position can be first framed and windowed to obtain a first audio frame, and then the first audio frame is fast Fourier transformed to obtain the frequency domain data of the first audio frame, and the energy value of each frequency point in the frequency domain data of the first audio frame is weighted by A weighting to obtain the energy value of the first audio frame. The sound level value of the second audio frame corresponding to the first audio frame measured by the sound level meter in the same environment is obtained. It can be understood that the first audio frame and the second audio frame correspond to the same audio frame collected by the sound level meter and the target device for the same white noise signal. After obtaining the sound level value of the second audio frame and the energy value of the first audio frame, the value obtained by subtracting the energy value of the first audio frame from the sound level value of the second audio frame can be used as the sound pressure bias value bias of the target device at the corresponding sound pressure level. The energy value of the original audio frame at the corresponding sound pressure level is compensated sBFSA with the sound pressure bias value bias of the target device at each sound pressure level to obtain the sound pressure characterization value dBSPL=sBFSA+bias of the original audio frame. To further improve the accuracy, multiple sound pressure bias values ​​of the target device at the same sound pressure level can be obtained, and the average value of the multiple sound pressure bias values ​​of the target device at the same sound pressure level can be used as the boost bias of the target device at the corresponding sound pressure level.

[0035] In one implementation scenario, after the output mask of the mask layer in the speech model when processing the original audio frame based on the speech model, the average value of the mask value of the original audio frame at several frequency points in the output mask can be obtained as the mask representation value of the original audio frame. Of course, in actual application, it is not limited to this. For example, different weights can be assigned to different frequency points, so that the mask values ​​of the corresponding frequency points are weighted based on the weights of each frequency point to obtain the mask representation value of the original audio frame.

[0036] In a specific implementation scenario, the speech model outputs a mask vector MASK for each original audio frame. The mask vector is a vector with the same length as the number of frequency points of the frequency domain data. Each element represents the ratio of speech to noise at each frequency point. Each element is a value between 0 and 1. 0 indicates that the frequency point is mainly noise, and 1 indicates that the frequency point is mainly speech. The average value of all elements in the mask vector is calculated as the mask representation value of the original audio frame.

[0037] Step S12: Based on the sound pressure characterization value and the mask characterization value, determine whether to use the original audio frame as a target audio frame to be transmitted to the communication peer.

[0038] In one implementation scenario, the original audio frame may be judged by combining the sound pressure characterization value and the mask characterization value, and the original audio frame that satisfies both the sound pressure characterization value condition and the mask characterization value condition is used as the target audio frame to be transmitted to the communication peer. As a possible example, the sound pressure characterization value condition may include that the sound pressure characterization value of the original audio frame is greater than the sound pressure threshold, or, as another possible example, the mask characterization value condition may include that the mask characterization value of the original audio frame is greater than a preset mask threshold.

[0039] In a specific implementation scenario, in response to the sound pressure characterization value being greater than the sound pressure threshold and the mask characterization value being greater than the mask threshold, the original audio frame may be used as the target audio frame. In addition, in response to the sound pressure characterization value being not greater than the sound pressure threshold and the mask characterization value being not greater than the mask threshold, the original audio frame may not be used as the target audio frame.

[0040] In one implementation scenario, an original audio frame whose mask characterization value is greater than a mask threshold value can be determined as a speech frame, an original audio frame whose mask characterization value is not greater than the mask threshold value can be determined as a silent frame, and an original audio frame whose sound pressure characterization value is greater than the sound pressure threshold value and is a speech frame can be confirmed as a target audio frame. An original audio frame that will be a speech frame but whose sound pressure characterization value is not greater than the sound pressure threshold value is not used as a target audio frame. An original audio frame that is not a speech frame is also not used as a target audio frame. Avoiding the false triggering of the transmission of audio frames in noisy or interfering environments improves communication quality.

[0041] In one implementation scenario, different from the aforementioned situation, in response to the sound pressure characterization value being not greater than the sound pressure threshold or the mask characterization value being not greater than the mask threshold, the original audio frame is not used as the target audio frame. As a possible example, the sound pressure threshold may be a preset threshold, such as an empirical value. The specific value of the sound pressure threshold is not limited here. As another possible example, the sound pressure threshold may also be a number of sound pressure thresholds corresponding to different usage scenarios, and different sound pressure thresholds may be matched according to different usage scenarios. Specifically, a pop-up window may be provided on the client to allow the user to independently select the corresponding usage scenario and match the corresponding sound pressure threshold through a built-in program; or, the environmental noise may be automatically identified through a deep learning neural network and the corresponding sound pressure threshold may be selected. This enables the system to exhibit stronger adaptability and stability when facing different environments and noise levels. At the same time, since key parameters such as the sound pressure level threshold and the mask determination threshold can be adjusted, the system has higher flexibility and accuracy in different usage scenarios.

[0042] In the above scheme, on the one hand, the present application can realize non-sensory voice communication wake-up because it automatically determines whether to transmit the original audio frame to the communication peer without manually pressing a button to confirm whether to conduct voice communication. On the other hand, since the mask characterization value is obtained by the output mask, and the output mask contains a mask value that characterizes the frequency point as noise or voice, the mask characterization value and the sound pressure characterization value are combined to jointly determine whether to transmit the original audio frame to the communication peer, which can reduce the false wake-up of voice communication caused by noise such as background noise and burst noise as much as possible, which is helpful for the accuracy of voice communication wake-up. On the other hand, since it can automatically determine whether to transmit the original audio frame to the communication peer through the above-mentioned joint determination process, the communication device can enter the wake-up state and transmit the audio frame only when communication transmission is required, and can reduce the transmission of invalid information as much as possible, which helps to improve the efficiency of voice communication wake-up. Therefore, the accuracy and efficiency of voice communication wake-up can be improved under the premise of realizing non-sensory voice communication wake-up.

[0043] See also Figure 2 , Figure 2 It is a flow chart of an embodiment of the communication wake-up method of the present application.

[0044] like Figure 2 As shown, the method comprises the following steps:

[0045] Step S21: Acquire each audio frame collected by the first communication device as an original audio frame.

[0046] In one implementation scenario, the first communication device can be any device with an audio acquisition function, such as a smart phone, a smart speaker, an in-vehicle communication device, etc. The first communication device can collect audio frames through a pickup circuit in the first device, and the pickup circuit can be in an on state at all times to continuously collect audio frames.

[0047] In a specific implementation scenario, the first communication device is a remote communication device. The remote communication device collects surrounding audio signals through a microphone integrated in the body. The microphone can be in a normally-on state to achieve continuous collection of audio signals, and the audio signals are framed and windowed to generate several audio frames as original audio frames.

[0048] Step S22: Determine whether the original audio frame is a target audio frame to be transmitted to the communication peer.

[0049] In the embodiments of the present disclosure, the original audio frame can determine whether it is a target audio frame based on the voice information processing method of any of the above embodiments. As a possible example, the first communication device can be preprocessed first to obtain the sound pressure bias value of the first device. Then the energy value of the original audio frame is compensated by using the sound pressure bias value to obtain the sound pressure characterization value of the original audio frame. And the mask characterization value of the original audio frame is obtained, and the original audio frame whose sound pressure characterization value is greater than the sound pressure threshold and the mask characterization value is greater than the mask threshold is used as the target audio frame. The original audio frame is judged based on the comprehensive sound pressure characterization value and the mask characterization value, and the original audio frame that meets both the sound pressure characterization value condition and the mask characterization value condition is used as the target audio frame to be transmitted to the communication counterpart. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here.

[0050] Step S23: Based on whether the original audio frame is a target audio frame, determine whether to send the original audio frame to the second communication device that is communicatively connected to the first communication device.

[0051] In one implementation scenario, in response to confirming that the original audio frame is a target audio frame, the first communication device wakes up a communication circuit in the device and transmits the target audio frame through the awakened communication circuit.

[0052] In a specific implementation scenario, in a remote video conference, multiple participants are located in different geographical locations and conduct real-time voice and video communication through the Internet. Each participant is equipped with a communication device, such as a mobile phone, a computer, etc. The communication device of the participant collects original audio frames through a pickup circuit. After the meeting starts, the pickup circuit of the communication device can be in a normally open state, and the communication circuit of the communication device is in a low-power standby state. The pickup circuit continuously collects audio frames as original audio frames for processing. In response to the original audio frame being determined as a target audio frame, the communication circuit of the communication device is awakened, and the target audio frame is transmitted to the communication device of one or more participants through the awakened communication circuit.

[0053] In a specific implementation scenario, in response to the original audio frame being a target audio frame, the communication circuit in the first communication device can be awakened, and the original audio frame can be sent to the second communication device through the awakened communication circuit. Specifically, during the sending process, additional encoding, encryption or compression processing can be added to ensure the security and transmission efficiency of the audio data. In addition, as a possible example, in response to the original audio frame being a non-target audio frame, the communication circuit in the first communication device may not be awakened. That is to say, in this case, the communication circuit in the first communication device can be kept in a dormant or low-power state, which helps to reduce energy loss. In the above method, on the one hand, only when the original audio frame is determined to be a target audio frame, the communication circuit of the communication device is awakened for data transmission, realizing on-demand awakening, avoiding the continuous operation of the communication circuit when there is no valid data transmission, thereby greatly reducing the energy consumption of the device. On the other hand, for non-target audio frames, the communication circuit is not awakened for transmission, which effectively reduces the occupation of network bandwidth, avoids network congestion, and improves communication quality.

[0054] In a specific implementation scenario, in a remote video conference, a participant's communication device collects an original audio frame, and in response to the original audio frame not being a target audio frame, the communication circuit is not woken up, thereby reducing unnecessary data transmission and improving the communication quality of the conference.

[0055] The above scheme obtains each audio frame collected by the first communication device as an original audio frame, and determines whether the original audio frame is a target audio frame to be transmitted to the communication counterpart. Then, based on whether the original audio frame is the target audio frame, determines whether to send the original audio frame to the second communication device connected to the first communication device. Since the original audio frame is determined to be a target audio frame based on the voice information processing device in the above-mentioned voice information processing device embodiment, it is possible to determine the target audio frame to be transmitted to the communication counterpart under the premise of realizing seamless voice wake-up, thereby improving the accuracy and efficiency of voice communication wake-up.

[0056] See also Figure 3 , Figure 3 It is a framework diagram of an embodiment of a speech information processing device of the present application. In this embodiment, the speech information processing device 30 includes an acquisition module 31 and a determination module 32. Among them, the acquisition module 31 is used to obtain the sound pressure characterization value of the original audio frame, and obtain the mask characterization value of the original audio frame based on the output mask of the mask layer in the speech model when the speech model processes the original audio frame. Among them, the output mask includes the mask values ​​of the original audio frame at several frequency points, and the mask value is a first value characterizing the noise or a second value characterizing the speech. The determination module 32 is used to determine whether to use the original audio frame as the target audio frame to be transmitted to the communication counterpart based on the sound pressure characterization value and the mask characterization value.

[0057] In the above scheme, the voice information processing device 30 can realize non-sensory voice communication wake-up because it automatically determines whether to transmit the original audio frame to the communication peer without manually pressing a button to confirm whether to conduct voice communication. On the other hand, since the mask characterization value is obtained by the output mask, and the output mask contains a mask value that characterizes the frequency point as noise or voice, the mask characterization value and the sound pressure characterization value are combined to jointly determine whether to transmit the original audio frame to the communication peer, which can reduce the false wake-up of voice communication caused by noise such as background noise and burst noise as much as possible, which is helpful for the accuracy of voice communication wake-up. On the other hand, since it can automatically determine whether to transmit the original audio frame to the communication peer through the above joint determination process, the communication device can enter the wake-up state and transmit the audio frame only when communication transmission is required, and can reduce the transmission of invalid information as much as possible, which is helpful for improving the efficiency of voice communication wake-up. Therefore, the accuracy and efficiency of voice communication wake-up can be improved under the premise of realizing non-sensory voice communication wake-up.

[0058] In one implementation scenario, the acquisition module 31 is used to obtain the sound pressure characterization value of the target audio frame, including: performing time-frequency domain conversion on the original audio frame to obtain frequency domain data; obtaining the energy value of the original audio frame based on the energy value of each frequency point in the frequency domain data; obtaining the sound pressure characterization value of the original audio frame based on the energy value of the original audio frame and the sound pressure bias value of the target device; wherein the target device is the acquisition device of the original audio frame.

[0059] In one implementation scenario, before obtaining the sound pressure characterization value of the original audio frame based on the energy value of the original audio frame and the sound pressure bias value of the target device, the acquisition module 31 is also used to obtain the first audio frame collected by the target device and the second audio frame collected by the sound level meter when playing the test audio frame of the same sound pressure level; obtain the energy value of the first audio frame, and obtain the sound level value measured by the sound level meter for the second audio frame; based on the sound level value and energy value at the same sound pressure level, obtain the sound pressure bias value of the target device at the corresponding sound pressure level; based on the sound pressure bias values ​​of the target device at different sound pressure levels, perform fusion to obtain the final sound pressure bias value of the target device. In some application scenarios, the test audio frame includes at least a white noise signal; in some application scenarios, the microphone of the target device and the microphone of the sound level meter are deployed at the same position when playing the test audio frame; in some application scenarios, the difference between the sound level value and the energy value at the same sound pressure level is the sound pressure bias value of the target device at the corresponding sound pressure level.

[0060] In one implementation scenario, the acquisition module 31 obtains the energy value of the original audio frame based on the energy value of each frequency point in the frequency domain data, including: obtaining the weighting factor of each frequency point in the frequency domain data based on the A-weighted weight curve; weighting the energy value of each frequency point in the frequency domain data based on the weighting factor of each frequency point in the frequency domain data to obtain the energy value of the original audio frame.

[0061] In one implementation scenario, the acquisition module 31 obtains the output mask of the mask layer in the speech model when processing the original audio frame based on the speech model to obtain the mask representation value of the original audio frame, including: obtaining the average value of the mask values ​​of the original audio frame at several frequency points in the output mask as the mask representation value of the original audio frame.

[0062] In one implementation scenario, the determination module 32 determines whether to use the original audio frame as the target audio frame to be transmitted to the communication counterpart based on the sound pressure characterization value and the mask characterization value, including at least one of the following: the determination module 32 uses the original audio frame as the target audio frame in response to the sound pressure characterization value being greater than the sound pressure threshold and the mask characterization value being greater than the mask threshold; the determination module 32 does not use the original audio frame as the target audio frame in response to the sound pressure characterization value being not greater than the sound pressure threshold and / or the mask characterization value being not greater than the mask threshold.

[0063] See also Figure 4 , Figure 4 Schematic diagram of the framework of an embodiment of the communication awakening device of the present application. Figure 4 As shown, the communication wake-up device 40 includes: a collection module 41, a judgment module 42 and a sending module 43. The collection module 41 is used to obtain each audio frame collected by the first communication device as an original audio frame; the judgment module 42 is used to determine whether the original audio frame is a target audio frame to be transmitted to the communication peer; the original audio frame is determined to be a target audio frame based on the above-mentioned voice information processing device 30; the sending module: based on whether the original audio frame is a target audio frame, determines whether to send the original audio frame to the second communication device connected to the first communication device.

[0064] In the above scheme, the communication wake-up device 40 obtains each audio frame collected by the first communication device as the original audio frame, and determines whether the original audio frame is the target audio frame to be transmitted to the communication counterpart, and then determines whether to send the original audio frame to the second communication device connected to the first communication device based on whether the original audio frame is the target audio frame. Since the original audio frame is determined to be the target audio frame based on the voice information processing device in the above-mentioned voice information processing device embodiment, it is possible to determine the target audio frame to be transmitted to the communication counterpart under the premise of realizing seamless voice wake-up, thereby improving the accuracy and efficiency of voice communication wake-up.

[0065] In one implementation scenario, the sending module 43 is used to determine whether to send the original audio frame to the second communication device that is communicatively connected to the first communication device based on whether the original audio frame is a target audio frame, including at least one of the following: in response to the original audio frame being the target audio frame, waking up the communication circuit in the first communication device, and sending the original audio frame to the second communication device through the awakened communication circuit. In response to the original audio frame not being a target audio frame, not waking up the circuit in the first communication device.

[0066] Therefore, on the one hand, only when the original audio frame is determined to be a target audio frame, the communication circuit is awakened for data transmission, realizing on-demand awakening, avoiding the continuous operation of the communication circuit when there is no valid data transmission, thereby greatly reducing the energy consumption of the device. On the other hand, for non-target audio frames, the communication circuit is not awakened for transmission, which effectively reduces the network bandwidth usage, avoids network congestion, and improves communication quality.

[0067] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 50 includes a processor 51 and a memory 52, the memory 52 is coupled to the processor 51, and a computing program is stored in the memory 52 to implement any of the above-mentioned voice information processing methods or communication wake-up methods. Specifically, the various components of the electronic device 50 can be coupled together through a bus, or the processor 51 of the electronic device 50 can be connected one by one with other components. The electronic device 50 can be any device with processing capabilities, such as a computer, a tablet computer, a mobile phone, etc.

[0068] The memory 52 is also used to store program data executed by the processor 51 and data during the processing of the processor 51. For example, the amount of data, metadata, timing data, etc. are stored. Among them, the memory 52 includes a non-volatile storage part for storing the above program data. The processor 51 controls the operation of the electronic device 50, and the processor 51 can also be called a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 51 can be implemented by multiple integrated circuit chips. The processor 51 is used to execute instructions to implement any of the above-mentioned voice information processing methods or communication wake-up methods by calling the program data stored in the memory 52.

[0069] In the above scheme, the electronic device 50 obtains the sound pressure characterization value of the original audio frame, and obtains the mask characterization value of the original audio frame based on the output mask of the mask layer in the speech model when processing the original audio frame based on the speech model, and the output mask includes the mask values ​​of the original audio frame at several frequency points, and the mask value is a first value characterizing noise or a second value characterizing speech. Then, based on the sound pressure characterization value and the mask characterization value, it is determined whether to use the original audio frame as the target audio frame to be transmitted to the communication counterpart. Therefore, on the one hand, since it is automatically determined whether to transmit the original audio frame to the communication counterpart without manually pressing a button to confirm whether to conduct voice communication, it can achieve non-sensitive voice communication wake-up, and on the other hand, On the one hand, since the mask characterization value is obtained by the output mask, and the output mask contains a mask value that characterizes the frequency point as noise or speech, the mask characterization value and the sound pressure characterization value are combined to jointly determine whether to transmit the original audio frame to the communication counterpart, which can minimize the false wake-up of voice communication caused by background noise, burst noise and other noises, and help improve the accuracy of voice communication wake-up. On the other hand, since the above-mentioned joint determination process can automatically determine whether to transmit the original audio frame to the communication counterpart, the communication device can enter the wake-up state and transmit the audio frame only when communication transmission is required, and the transmission of invalid information can be reduced as much as possible, which helps to improve the efficiency of voice communication wake-up. Therefore, the accuracy and efficiency of voice communication wake-up can be improved under the premise of realizing non-perceptible voice communication wake-up.

[0070] See also Figure 6 , Figure 6 The computer readable storage medium 60 stores a computer program 61, which is processed and executed to implement any of the above-mentioned voice information processing methods or communication wake-up methods.

[0071] The computer-readable storage medium 60 may specifically be a medium that can store program data, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server storing the computer program 61. The server may send the stored program data to other devices for execution, or may also execute the stored computer program 61 by itself.

[0072] In one implementation scenario, the computer-readable storage medium 60 may also be Figure 5 The memory 52 is shown.

[0073] In the above scheme, the computer-readable storage medium 60 obtains the sound pressure characterization value of the original audio frame, and obtains the mask characterization value of the original audio frame based on the output mask of the mask layer in the speech model when processing the original audio frame based on the speech model, and the output mask includes the mask values ​​of the original audio frame at several frequency points, and the mask value is a first value representing noise or a second value representing speech. Then, based on the sound pressure characterization value and the mask characterization value, it is determined whether to use the original audio frame as the target audio frame to be transmitted to the communication peer. Therefore, on the one hand, since it is automatically determined whether to transmit the original audio frame to the communication peer without manually pressing a button to confirm whether to conduct voice communication, it can achieve non-sensitive voice communication wake-up. On the other hand, since the mask characterization value is obtained by the output mask, and the output mask contains a mask value that characterizes the frequency point as noise or speech, the mask characterization value and the sound pressure characterization value are combined to jointly determine whether to transmit the original audio frame to the communication counterpart, which can minimize the false wake-up of voice communication caused by background noise, burst noise and other noises, and help improve the accuracy of voice communication wake-up. On the other hand, since the above-mentioned joint determination process can automatically determine whether to transmit the original audio frame to the communication counterpart, the communication device can enter the wake-up state and transmit the audio frame only when communication transmission is required, and the transmission of invalid information can be reduced as much as possible, which helps to improve the efficiency of voice communication wake-up. Therefore, the accuracy and efficiency of voice communication wake-up can be improved under the premise of realizing non-perceptible voice communication wake-up.

[0074] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0075] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0076] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0080] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A method for processing speech information, characterized in that: The method comprises: Acquire a sound pressure representation value of an original audio frame, and obtain a mask representation value of the original audio frame based on an output mask of a mask layer in the speech model when the original audio frame is processed by the speech model; wherein the output mask includes mask values ​​of the original audio frame at a plurality of frequency points, and the mask value is a first value representing noise or a second value representing speech; Based on the sound pressure characterization value and the mask characterization value, it is determined whether to use the original audio frame as a target audio frame to be transmitted to the communication peer.

2. The method according to claim 1, characterized in that The obtaining of the sound pressure representation value of the target audio frame includes: Performing time-frequency domain conversion on the original audio frame to obtain frequency domain data; Obtaining an energy value of the original audio frame based on the energy value of each frequency point in the frequency domain data; Based on the energy value of the original audio frame and the sound pressure bias value of the target device, a sound pressure representation value of the original audio frame is obtained; wherein the target device is a device for collecting the original audio frame.

3. The method according to claim 2, characterized in that Before obtaining the sound pressure representation value of the original audio frame based on the energy value of the original audio frame and the sound pressure offset value of the target device, the method further includes: Acquire a first audio frame respectively collected by the target device and a second audio frame respectively collected by the sound level meter when playing a test audio frame of the same sound pressure level; Acquire an energy value of the first audio frame, and acquire a sound level value measured by the sound level meter for the second audio frame; Based on the sound level value and the energy value at the same sound pressure level, obtaining a sound pressure bias value of the target device at the corresponding sound pressure level; The sound pressure offset values ​​of the target device at different sound pressure levels are fused to obtain a final sound pressure offset value of the target device.

4. The method according to claim 3, characterized in that The test audio frame at least includes a white noise signal; and / or, the microphone of the target device and the microphone of the sound level meter are deployed at the same position when playing the test audio frame; And / or, the difference between the sound level value and the energy value at the same sound pressure level is the sound pressure bias value of the target device at the corresponding sound pressure level.

5. The method according to claim 2, characterized in that: The obtaining the energy value of the original audio frame based on the energy value of each frequency point in the frequency domain data includes: Based on the A-weighted weight curve, a weighting factor of each frequency point in the frequency domain data is obtained; The energy value of each frequency point in the frequency domain data is weighted based on the weighting factor of each frequency point in the frequency domain data to obtain the energy value of the original audio frame.

6. The method according to claim 1, characterized in that The output mask of the mask layer in the speech model when processing the original audio frame based on the speech model to obtain the mask representation value of the original audio frame includes: An average value of mask values ​​of the original audio frame at several frequency points in the output mask is obtained as a mask representation value of the original audio frame.

7. The method according to claim 1, characterized in that The determining, based on the sound pressure characterization value and the mask characterization value, whether to use the original audio frame as a target audio frame to be transmitted to the communication peer end includes at least one of the following: In response to the sound pressure representation value being greater than a sound pressure threshold and the mask representation value being greater than a mask threshold, taking the original audio frame as the target audio frame; In response to the sound pressure representation value being not greater than a sound pressure threshold and / or the mask representation value being not greater than a mask threshold, the original audio frame is not used as the target audio frame.

8. A communication wake-up method, characterized in that: include: Acquire each audio frame collected by the first communication device as an original audio frame; Determine whether the original audio frame is a target audio frame to be transmitted to the communication peer; wherein the original audio frame is determined to be the target audio frame based on the voice information processing method according to any one of claims 1 to 7; Based on whether the original audio frame is the target audio frame, it is determined whether to send the original audio frame to a second communication device that is communicatively connected to the first communication device.

9. The method according to claim 8, characterized in that The determining whether to send the original audio frame to a second communication device that is communicatively connected to the first communication device based on whether the original audio frame is a target audio frame includes at least one of the following: In response to the original audio frame being the target audio frame, waking up a communication circuit in the first communication device, and sending the original audio frame to the second communication device through the awakened communication circuit; In response to the original audio frame not being the target audio frame, a communication circuit in the first communication device is not awakened.

10. A voice information processing device, characterized in that: The device comprises: An acquisition module, used to acquire a sound pressure representation value of an original audio frame, and obtain a mask representation value of the original audio frame based on an output mask of a mask layer in the speech model when the original audio frame is processed by the speech model; wherein the output mask includes mask values ​​of the original audio frame at a plurality of frequency points, and the mask value is a first value representing noise or a second value representing speech; A determination module is used to determine whether to use the original audio frame as a target audio frame to be transmitted to a communication peer based on the sound pressure characterization value and the mask characterization value.

11. A communication awakening device, characterized in that: include: A collection module, used to obtain each audio frame collected by the first communication device as an original audio frame; A judgment module, used to determine whether the original audio frame is a target audio frame to be transmitted to the communication peer; wherein the original audio frame is determined to be the target audio frame based on the voice information processing device according to claim 10; The sending module determines whether to send the original audio frame to a second communication device communicatively connected to the first communication device based on whether the original audio frame is the target audio frame.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory is coupled to the processor and a computing program is stored in the memory to implement the voice information processing method according to any one of claims 1 to 7, or the communication wake-up method according to claim 8 or 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is processed and executed to implement the voice information processing method according to any one of claims 1 to 7, or the communication wake-up method according to claim 8 or 9.

Images