Data processing method, electronic device, storage medium and computer program product
By using microphone array and mute detection technology in smart glasses, accurately distinguishing the voice signals of wearers and non-wearers, solving the problem of speech recognition accuracy of the device in simultaneous speech scenarios, improving the effect of the translation function and reducing the device Performance requirements.
Patent Information
- Application Number
- CN202410930876.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-11
AI Technical Summary
Wearable devices such as smart glasses are difficult to effectively distinguish voice signals when the wearer and non-wearer speak at the same time, affecting the accuracy of speech recognition and the effect of translation functions.
Using a microphone array, including a microphone closer to the wearer and a microphone farther away from the wearer, the sound source is determined through mute detection and complex machine learning algorithms to achieve accurate distinction between wearer and non-wearer audio and voice. Identify.
It improves the accuracy of vocal separation, improves the application effect of wearable devices in translation scenarios, reduces the processor performance requirements of the device, and extends the battery life of the device.
Smart Images

Figure CN118942491A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, electronic equipment, storage medium and computer program product. Background Art
[0002] There are more and more application scenarios for wearable devices such as smart glasses. For example, they can support translation functions, recognize the speech of non-wearers and translate it into text content in the target language, and then display the text content so that the wearer can directly know the speech content of the non-wearer by viewing the displayed text content, thereby improving the user's communication convenience. However, in the process of the wearer and the non-wearer speaking at the same time, smart glasses in related technologies are difficult to effectively distinguish the collected voices, which affects the accuracy of voice recognition, resulting in limited translation functions and affecting the user experience. Summary of the invention
[0003] In a first aspect, an embodiment of the present application provides a data processing method for a wearable device, wherein the wearable device includes a microphone array, the microphone array includes a first microphone and a second microphone, the target sound source of the first microphone is a wearer, and the target sound source of the second microphone is a non-wearer; wherein the data processing method includes:
[0004] Performing a first mute detection on the first audio collected by the first microphone to obtain a first mute detection result;
[0005] In response to the first silence detection result indicating that a voice segment exists, performing a second silence detection on the second audio collected by the second microphone to obtain a second silence detection result;
[0006] Determining sound sources of the first audio and the second audio according to the first silence detection result and the second silence detection result;
[0007] In response to the sound source including a target sound source in a current application scenario, speech recognition is performed on audio collected by a microphone corresponding to the target sound source in the current application scenario.
[0008] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.
[0009] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.
[0010] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above methods when executed by a processor.
[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.
[0013] Figure 1A A pair of smart glasses provided by an embodiment of the present application is shown;
[0014] Figure 1B Another type of smart glasses provided by an embodiment of the present application is shown;
[0015] Figure 2 A flowchart showing a data processing method for a wearable device according to an embodiment of the present application;
[0016] Figure 3 A schematic diagram showing an application scenario of the data processing method for a wearable device according to an embodiment of the present application;
[0017] Figure 4 A block diagram showing a data processing apparatus for a wearable device according to an embodiment of the present application;
[0018] Figure 5 A block diagram of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION
[0019] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.
[0020] When wearable devices such as smart glasses are used in translation scenarios, the collected audio usually includes the voices of the wearer and non-wearers. It is necessary to distinguish the audio corresponding to different sound sources in the collected audio and extract the audio of non-wearers separately for speech processing. In view of the above situation, two solutions are proposed in the relevant technology to distinguish the sounds of different sound sources.
[0021] In one solution, a microphone array can be formed by using multiple microphones set at different positions of smart glasses, and the sound signal in a specific direction can be enhanced by adjusting the signal phase of each microphone, so as to enhance and extract the sound signal of the non-wearer. However, when the wearer and the non-wearer speak at the same time, this solution is difficult to effectively distinguish the voices of the wearer and the non-wearer, thus affecting the accuracy of the translation function.
[0022] In another solution, machine learning and artificial intelligence algorithms can be used to distinguish the voices of wearers and non-wearers. For example, the collected audio can be processed through a pre-trained convolutional neural network or recurrent neural network. However, these deep learning models have high requirements for computing power. Due to the processor performance and power consumption requirements of wearable devices such as smart glasses, they need to be operated with the help of other terminal devices such as mobile phones, computers or cloud servers. When the network connection is unstable or there is no network service, smart glasses cannot realize the translation function alone. It can be seen that this solution has certain limitations and cannot meet the user's usage needs in specific scenarios.
[0023] The above two solutions in the related art both have certain defects or limitations in the implementation scenario of the translation function of wearable devices such as smart glasses. In this regard, the embodiment of the present application provides a data processing method for a wearable device, which aims to improve the accuracy of human voice separation and is also conducive to reducing the performance requirements for wearable devices.
[0024] In the data processing method of the wearable device provided in the embodiment of the present application, the wearable device may be provided with a microphone array, and the microphone array may include a first microphone with the wearer's sound source as the target sound source and a second microphone with the non-wearer's sound source as the target sound source. It can be understood that both microphones have the function of collecting surrounding audio, that is, the microphone with the wearer as the target sound source may also collect the sound emitted by the non-wearer, and the same is true for the microphone with the non-wearer as the target sound source. Generally, it is necessary to enhance the collection effect of each microphone on the target sound source and reduce the influence of the non-target sound source on the microphone. In conventional design, this purpose can be achieved by optimizing the position of the microphone. For example, the microphone with the wearer as the target sound source is mostly a microphone arranged near the wearer's mouth, so that it has a better collection effect on the sound emitted by the wearer. The microphone with the non-wearer as the target sound source is mostly a microphone arranged away from the wearer's mouth, which is conducive to reducing the interference caused by the wearer's speech to the microphone.
[0025] Figure 1A A smart glasses provided by an embodiment of the present application is shown, such as Figure 1AAs shown, in some examples, the smart glasses may include a frame and two temples, and the two temples are respectively arranged on opposite sides of the frame. A first microphone (MIC1 in the figure) and a second microphone (MIC2 in the figure) may be arranged on one of the temples, and the first microphone is arranged on a side of the temple adjacent to the frame, and the second microphone is arranged on a side of the temple away from the frame. It can be understood that when the smart glasses are in a wearing state, the distance between the first microphone and the wearer's mouth is smaller than the distance between the second microphone and the wearer's mouth.
[0026] Figure 1B Another smart glasses provided by the embodiment of the present application is shown as follows: Figure 1B As shown, in other examples, the first microphone (MIC1 in the figure) can be set on the side of the temple adjacent to the frame, and the second microphone (MIC2 in the figure) can be set on the frame, specifically, in the middle of the frame. It can be understood that when the smart glasses are in the wearing state, the distance between the first microphone and the wearer's mouth is smaller than the distance between the second microphone and the wearer's mouth.
[0027] The above two examples are only used to illustrate the setting methods of the first microphone and the second microphone. In fact, the setting methods of the first microphone and the second microphone on the smart glasses can be various, as long as the requirement of setting the first microphone closer to the wearer's mouth relative to the second microphone can be met. Those skilled in the art can flexibly set it according to actual conditions.
[0028] It should be noted that the first microphone is arranged closer to the mouth of the wearer than the second microphone, and accordingly, the second microphone is arranged closer to the mouth of the non-wearer than the first microphone. Accordingly, the intensity of the sound signal of the wearer collected by the first microphone is stronger than the intensity of the sound signal of the wearer collected by the second microphone, and conversely, the intensity of the sound signal of the non-wearer collected by the first microphone is weaker than the intensity of the sound signal of the non-wearer collected by the second microphone. In addition, for the first microphone, when the volume of the wearer's sound source and the non-wearer's sound source are the same, the intensity of the sound signal of the wearer collected by it is stronger than the intensity of the sound signal of the non-wearer; for the second microphone, when the volume of the wearer's sound source and the non-wearer's sound source are the same, the intensity of the sound signal of the non-wearer collected by it is stronger than the intensity of the sound signal of the wearer.
[0029] Figure 2 A flow chart showing a data processing method for a wearable device according to an embodiment of the present application is shown as follows: Figure 2 As shown, the data processing method of the wearable device of the embodiment of the present application may include the following steps:
[0030] S101: Performing a first mute detection on a first audio collected by a first microphone to obtain a first mute detection result;
[0031] S102: In response to the first silence detection result indicating that a voice segment exists, performing a second silence detection on the second audio collected by the second microphone to obtain a second silence detection result;
[0032] S103: Determine sound sources of the first audio and the second audio according to the first silence detection result and the second silence detection result;
[0033] S104: In response to the sound source including a target sound source in the current application scenario, performing speech recognition on audio collected by a microphone corresponding to the target sound source in the current application scenario.
[0034] The data processing method of the embodiment of the present application can be applied to various wearable devices with voice processing functions, such as smart glasses, smart watches, smart bracelets, etc., but is not limited thereto. In the following description of the specification of the present application, smart glasses are taken as an example for detailed description.
[0035] In an embodiment of the present application, the first audio collected by the first microphone and the second audio collected by the second microphone may be audio collected based on the same time period. Exemplarily, the first microphone and the second microphone may continuously collect audio, and store the first audio and the second audio corresponding to the same time period collected in the storage module of the wearable device at intervals. The processor of the wearable device reads the first audio and the second audio from the storage module and executes the data processing method provided in the embodiment of the present application. That is, before step S101, the first audio may be obtained from the storage module; before step S102, the second audio may be obtained from the storage module.
[0036] In the embodiments of the present application, in some examples, VAD (Voice Activity Detection) technology can be used to detect whether there is a voice segment in the audio collected by the first microphone or the second microphone, thereby realizing silence detection. Exemplarily, the silence detection process may specifically include the following steps: (1) Preprocessing: Preprocessing the audio signal, for example, including pre-emphasis, framing and windowing processing; (2) Calculating short-time energy: For each audio frame included in the audio signal, calculating its energy value, for example, it can be calculated by summing the squares of all audio sample values; (3) Comparing the energy value of each audio frame with a preset energy threshold, and determining whether there is voice activity in the audio frame based on the comparison result; for example, when the energy value is greater than or equal to the energy threshold, it is determined that there is voice activity in the audio frame, and when the energy value is less than the energy threshold, it is determined that there is no voice activity in the audio frame, that is, the audio frame may be background noise or silence; (4) Smoothing: In order to avoid the result of speech detection being too fragmented, smoothing processing can be introduced, for example, only when multiple consecutive audio frames are detected as having voice activity, these multiple audio frames are determined as voice segments; (5) Post-processing: Further processing the detected voice segments, for example, eliminating voice breakpoints that are too short, merging silent frames that are too short, etc.
[0037] In addition, in other examples of the present application, more complex machine learning algorithms, such as neural networks, can also be used to detect whether there is a voice segment in the audio collected by the first microphone or the second microphone. Exemplarily, the silence detection process may include the following steps: (1) preprocessing of the audio signal: for example, it may include noise removal, echo cancellation, etc. to improve the quality of the audio data; (2) feature extraction of the audio signal: extracting useful features from the preprocessed audio signal, such as short-time energy, short-time average zero crossing rate, spectral features (such as Mel frequency cepstral coefficients, MFCC), etc.; (3) making a decision, such as using some form of model or algorithm (such as threshold detection, machine learning classifier, etc.) to determine whether there is a voice segment in the current frame; (4) smoothing processing: in order to avoid the output voice activity signal being too fragmented, some smoothing processing is required, such as sliding window averaging or using a hidden Markov model (HMM).
[0038] Exemplarily, the silence detection result may include whether there is a voice segment in the audio. Wherein, in the case of a voice segment, the silence detection result may also include a voice segment after removing the non-voice segment from the audio. Wherein, the non-voice segment refers to an audio segment composed of continuous audio frames without voice activity.
[0039] In an embodiment of the present application, the first silence detection refers to the silence detection performed on the first audio, and the second silence detection refers to the silence detection performed on the second audio. The process of the first silence detection and the second silence detection can refer to the aforementioned example and will not be described in detail here. In an embodiment of the present application, silence detection is first performed on the first audio, and when the result of the first silence detection indicates that there is a voice segment in the first audio, silence detection is then performed on the second audio. The adoption of this implementation method is conducive to reducing the number of silence detections performed on the second audio, and thus is conducive to reducing power consumption. Moreover, the first microphone can be a microphone with the wearer's sound source as the target sound source. This is because compared with the silence detection of non-wearers, the power consumption of silence detection on the wearer is lower, which is conducive to further reducing the overall power consumption.
[0040] In the embodiment of the present application, the sound source refers to the target object that emits the sound corresponding to the voice segment. In different application scenarios, the target sound source that needs to be identified is different. Take the common translation scenario and voice assistant scenario as examples. In the translation scenario, the process of the wearer talking to the non-wearer is involved, and the sound source corresponding to the first audio or the second audio may include at least one of the wearer and the non-wearer. Usually, in the translation scenario, the voice segment corresponding to the non-wearer must eventually be voice processed, so the target sound source in the translation scenario is the non-wearer sound source. In the voice assistant scenario, the voice commands issued by the wearer need to be identified, so the target sound source in the voice assistant scenario is the wearer sound source.
[0041] Taking the translation scenario as an example, when the sound source corresponding to the voice segment existing in the first audio or the second audio includes the target sound source, that is, when there is a voice segment corresponding to the sound emitted by the non-wearer in the first audio or the second audio, for example, the wearer and the non-wearer speak at the same time or only the non-wearer speaks, step S104 is executed, that is, voice recognition is performed on the second audio collected by the second microphone. When there is no voice segment in the first audio and the second audio or the sound source corresponding to the voice segment does not include the target sound source, for example, neither the wearer nor the non-wearer speaks or only the wearer speaks, step S104 is not executed, that is, voice recognition is not performed on the second audio collected by the second microphone.
[0042] Exemplarily, in step S101, for the first silence detection performed on the first audio, the first audio may be input into a pre-trained silence detection model to obtain a first confidence of the first audio. For example, the silence detection model may be a model based on the aforementioned VAD technology, and the first confidence may be an energy value obtained after the silence detection model preprocesses the first audio and calculates the short-time energy, or a probability value further calculated based on the energy meter, and the probability value may represent the probability of the presence of voice activity in the first audio. After obtaining the first confidence, a smaller confidence threshold may be used to compare with the first confidence of the first audio, and when the first confidence is greater than or equal to the confidence threshold, it is determined that there is a voice segment in the first silence detection result.
[0043] It is understandable that when there is a voice segment in the first silence detection result, the voice segment may be emitted by the target sound source or the non-target sound source, and this is not distinguished in step S101. As mentioned above, since the first audio collected by the first microphone may include sounds emitted by the target sound source and sounds emitted by non-target sound sources, the energy values of the sounds emitted by the target sound source and the sounds emitted by the non-target sound source are different. In order to avoid missed detection, a smaller confidence threshold is required to be compared with the first confidence. Taking the translation scenario as an example, the confidence threshold used in S101 can be the first threshold, and the first threshold can be set to be less than the confidence output by the silence detection model when there is only sound emitted by non-target sound sources in the first audio, so as to further reduce the missed detection rate.
[0044] Taking the translation scenario as an example, if the first silence detection result is that there is a voice segment, there may be three situations in the time period corresponding to the first audio: the first situation is that only the wearer speaks, that is, the sound source of the first audio only includes the non-target sound source in the current application scenario; the second situation is that only the non-wearer speaks, that is, the sound source of the first audio only includes the target sound source in the current application scenario; the third situation is that both the wearer and the non-wearer speak, that is, the sound source of the first audio includes the target sound source and the non-target sound source in the current application scenario. Among them, since it is impossible to determine whether the sound source corresponding to the first audio includes the target sound source in the current application scenario when there is a voice segment in the first silence detection result, it is necessary to perform a second silence detection on the second audio, and further determine whether the sound sources of the first audio and the second audio include the target sound source in the current application scenario according to the first silence detection result and the second silence detection result. It can be understood that if the first silence detection result is that there is no voice segment in the first audio, it means that no one speaks in the time period corresponding to the first audio, that is, the first microphone collects silence or background noise. In this case, there is no need to execute step S102, that is, there is no need to perform the second silence detection on the second audio.
[0045] In step S102, for the second silence detection performed on the second audio, the second audio may be input into a pre-trained silence detection model to obtain a second confidence of the second audio. For example, the silence detection model may be a model based on the aforementioned VAD technology, and the second confidence may be an energy value obtained after the silence detection model pre-processes the second audio and calculates the short-time energy, or a probability value further calculated based on the energy meter, and the probability value may represent the probability of the presence of voice activity in the second audio. After obtaining the second confidence, a larger confidence threshold may be used to compare with the second confidence of the second audio. When the second confidence is greater than or equal to the confidence threshold, it is determined that the second detection result contains a voice segment that is not the wearer's sound source.
[0046] It is understandable that the purpose of performing the second silence detection on the second audio is to determine whether the second audio contains the sound of the target sound source corresponding to the second microphone. Based on this, taking the translation scenario as an example, the confidence threshold used in step S102 can be a third threshold, which can be equal to or slightly less than the confidence output by the silence detection model when only the sound emitted by the non-wearer is in the second audio, but should be greater than the confidence output when only the sound emitted by the wearer is in the second audio, so that when the second confidence is greater than the confidence threshold, it can be determined that there is a voice segment of the non-wearer sound source in the second detection result corresponding to the second audio.
[0047] In step S103, if the first silence detection result is that there is a voice segment in the first audio and the second silence detection result is that there is no voice segment in the second audio, it can be determined that the sound sources of the first audio and the second audio only include the wearer sound source and do not include non-wearer sound sources.
[0048] If the first silence detection result is that there is a voice segment in the first audio and the second silence detection result is that there is a voice segment in the second audio, it can be determined that the sound sources of the first audio and the second audio may include only non-wearer sound sources, or may include both wearer sound sources and non-wearer sound sources.
[0049] In step S104, taking the current application scenario as a translation scenario as an example, the target sound source is a non-wearer sound source. If step S103 determines that the sound sources of the first audio and the second audio include at least a non-wearer sound source, then there is a voice segment corresponding to the target sound source in the first audio and the second audio. Based on this, speech recognition can be performed on the second audio collected by the second microphone corresponding to the non-wearer sound source, i.e., the target sound source, to obtain a speech recognition result corresponding to the voice segment emitted by the non-wearer.
[0050] In an embodiment of the present application, the speech recognition result can also be translated into a target text according to the target language and displayed on a display interface of a wearable device such as smart glasses, so that the wearer can know the speech content of the interlocutor (such as the non-wearer).
[0051] According to the data processing method of the embodiment of the present application, the first mute detection is performed on the first audio collected by the first microphone to obtain the first mute detection result, and when the first mute detection result is that there is a voice segment, the second mute detection is performed on the second audio collected by the second microphone to obtain the second mute detection result, and the sound source of the first audio and the second audio is determined according to the first mute detection result and the second mute detection result. Taking the current application scenario as a translation scenario as an example, when the sound source of the second audio includes the target sound source, that is, the non-wearer sound source, the second audio is subjected to speech recognition to obtain the speech recognition result of the sound emitted by the non-wearer sound source. In summary, the data processing method of the embodiment of the present application can realize the recognition of the sound source, specifically, it can recognize the situation where the wearer and the non-wearer are speaking at the same time, and only in this case does the voiceprint separation process perform. Based on this, the data processing method of the embodiment of the present application improves the accuracy of human voice separation, thereby improving the application effect of the wearable device in the translation scenario, and is also conducive to reducing the processor performance of the wearable device, thereby improving the battery life of the wearable device.
[0052] In one implementation, before performing the first silence detection on the first audio and performing the second silence detection on the second audio, echo cancellation processing may be performed on the first audio and the second audio.
[0053] Exemplarily, the audio signal played by the speaker of the smart glasses can be used to perform echo cancellation processing on the first audio or the second audio. Specifically, the delay of the echo signal can be calculated based on the audio signal played by the speaker and the first audio or the second audio. According to the delay of the echo signal, the echo signal is time-aligned with the first audio and the second audio respectively, and then the cancellation signal with the same waveform and opposite phase as the echo signal output by the pre-set filter is used to accumulate the cancellation signal with the first audio and the second audio respectively to cancel out the echo signals in the first audio and the second audio. The filter can be obtained based on NLMS (Normalized Least Mean Square, an adaptive filtering algorithm).
[0054] In other examples of the present application, a neural network-based echo cancellation algorithm may also be used for echo cancellation processing, and those skilled in the art may flexibly select the algorithm based on actual conditions.
[0055] Through the above implementation, the interference of the sound signal played by the speaker collected by the first microphone and the second microphone on the subsequent silence detection and speech recognition can be eliminated, thereby improving the accuracy of the subsequent silence detection and speech recognition.
[0056] In one embodiment, the first silence detection result includes a first confidence level of the first audio. Wherein, when the first confidence level is greater than or equal to a first threshold, the first silence detection result is that there is a voice segment, and step S103 may include:
[0057] In response to the first confidence being greater than or equal to a second threshold, it is determined that the sound source includes a wearer sound source, and the second threshold is greater than or equal to the first threshold.
[0058] In the embodiment of the present application, the first confidence is used to characterize the probability that the first audio includes a speech segment. The greater the first confidence, the greater the probability that the first audio includes a speech segment, and vice versa. Exemplarily, the first confidence can be obtained by inputting the first audio into a pre-trained silence detection model.
[0059] Exemplarily, the first threshold value may be smaller than the second threshold value. For setting the first threshold value, the voice segment corresponding to the wearer's sound source in the first audio collected by the first microphone may be input into the silence detection model to obtain the first reference confidence, and the voice segment corresponding to the non-wearer's sound source in the first audio collected by the first microphone may be input into the silence detection model to obtain the second reference confidence. The first reference confidence is greater than the second reference confidence. The first threshold value may be set to be smaller than the second reference confidence, and the second threshold value may be set to be greater than the second reference confidence and less than the first reference confidence.
[0060] In one implementation, the second silence detection result includes a second confidence level of the second audio. Step S103 may further include:
[0061] In response to the second confidence being greater than or equal to the third threshold, it is determined that the sound source includes a non-wearer sound source.
[0062] In the embodiment of the present application, the second confidence is used to characterize the probability that the second audio includes a speech segment. The greater the second confidence, the greater the probability that the second audio includes a speech segment, and vice versa. Exemplarily, the second confidence can be obtained by inputting the second audio into a pre-trained silence detection model.
[0063] Exemplarily, for setting the third threshold, a voice segment corresponding to the wearer's sound source in the second audio collected by the second microphone can be input into the silence detection model to obtain a third reference confidence, and a voice segment corresponding to a non-wearer's sound source in the second audio collected by the second microphone can be input into the silence detection model to obtain a fourth reference confidence. The third reference confidence is less than the fourth reference confidence. The third threshold can be set to be greater than the third reference confidence and less than the fourth reference confidence.
[0064] Therefore, when the second confidence level is greater than or equal to the third threshold, it can be determined that the sound source of the second audio includes a non-wearer sound source.
[0065] It should be noted that, when the first confidence is greater than or equal to the second threshold and the second confidence is greater than or equal to the third threshold, it can be determined that the sound sources of the first audio and the second audio include both the wearer sound source and the non-wearer sound source. When the first confidence is less than the second threshold and the second confidence is greater than or equal to the third threshold, it can be determined that the sound sources of the first audio and the second audio include only the non-wearer sound source and do not include the wearer sound source. When the first confidence is greater than or equal to the second threshold and the second confidence is less than the third threshold, it can be determined that the sound sources of the first audio and the second audio include only the wearer sound source and do not include the non-wearer sound source.
[0066] In one implementation, step S104 may specifically include the following steps:
[0067] S1401: In response to the target sound source in the current application scenario being a non-wearer sound source, and the sound source includes a wearer sound source and a non-wearer sound source, performing directional speech enhancement on the second audio to obtain an enhanced second audio;
[0068] S1402: Perform speech recognition on the enhanced second audio.
[0069] For example, when the current application scenario is a translation scenario, the target sound source in the current application scenario is a non-wearer sound source. When the first confidence of the first audio is greater than or equal to the second threshold, and the second confidence of the second audio is greater than or equal to the third threshold, it is determined that the sound source includes the wearer sound source and the non-wearer sound source.
[0070] Exemplarily, in step S1401, beamforming technology can be used to perform directional speech enhancement on the microphone array of the wearable device. Specifically, a specific geometric shape formed by the arrangement of the first microphone and the second microphone is used to capture sound signals from different directions. For example, since the first microphone is arranged closer to the mouth of the wearer than the second microphone, and the second microphone is arranged closer to the mouth of the non-wearer than the first microphone, the first microphone can be used to collect sound signals from the direction of the wearer's sound source, and the second microphone can be used to collect sound signals from the direction of the non-wearer's sound source. The second microphone is used as the target microphone, and the time delay of the first audio collected by the first microphone relative to the second audio collected by the second microphone is calculated. According to the time delay, the first audio and the second audio are time-aligned and phase-adjusted so that the sound signals in the first audio and the second audio in the direction of the non-wearer's sound source can be coherently superimposed. Using a preset beamformer, the superimposed sound signal is enhanced, while suppressing sound signals from other directions (such as the direction of the wearer's sound source). The beamformer can be obtained based on a generalized sidelobe canceller (GSC) algorithm and a post-filtering architecture, thereby obtaining an enhanced second audio, and the enhanced second audio includes a sound segment corresponding to a sound source other than the wearer.
[0071] Exemplarily, in step S1402, feature extraction may be performed on the enhanced second audio to obtain corresponding audio features, and the audio features may include Mel frequency cepstral coefficients (MFCCs) or linear predictive coding (LPC), etc. The audio features are analyzed using a pre-trained acoustic model to obtain a corresponding speech recognition result. The acoustic model may be established based on a hidden Markov model (HMM), a deep neural network (DNN), or a combination thereof.
[0072] In one implementation, step S1402 may specifically include the following steps:
[0073] Performing voiceprint separation processing on the enhanced second audio to obtain a sound segment emitted by a person other than the wearer;
[0074] The sound segment emitted by the non-wearer is input into the speech recognition model to obtain the speech recognition result of the second audio.
[0075] In this embodiment, a pre-trained voiceprint separation module can be used to perform voiceprint separation processing on the enhanced second audio. The voiceprint separation module can be obtained based on a neural network model, for example, it can be based on a convolutional neural network (CNN), a recurrent neural network (RNN), a hybrid model of a convolutional neural network and a recurrent neural network, a Transformer model, or an end-to-end model.
[0076] For example, the enhanced second audio may be subjected to feature extraction to obtain corresponding audio features, which may specifically include time domain features or FBANK features. Then, the audio features are input into the voiceprint separation module to obtain separated sound segments emitted by non-wearers. Finally, the sound segments emitted by non-wearers are input into the speech recognition model to obtain the speech recognition result of the second audio.
[0077] It should be noted that in the embodiments of the present application, the voiceprint separation process can be performed on the wearable device or on the terminal device. For example, the wearable device can send the enhanced second audio to the terminal device, and perform voiceprint separation on the enhanced second audio through the voiceprint separation module pre-set on the terminal device. Among them, Bluetooth transmission can be used between the wearable device and the terminal device, or it can be transmitted through a wireless communication network, and the embodiments of the present application do not specifically limit this. Similarly, voice recognition processing using a speech recognition model can be performed on a wearable device or on a terminal device.
[0078] In one implementation, step S1402 may specifically include the following steps:
[0079] intercepting at least one voice frame with voice activity from the enhanced second audio according to the second silence detection result;
[0080] The speech frame with speech activity is input into the speech recognition model to obtain the speech recognition result of the second audio.
[0081] In some examples, for the enhanced second audio, speech frames with voice activity can be intercepted based on the time domain features of the audio frames included in the second audio. Specifically, the time domain root mean square (RMS) of the audio frames included in the second audio is calculated, and the calculation formula is: {(x1^2+x2^2+x3^2+....+xn^2) / n}^0.5, where xn is used to characterize the time domain feature value of the nth audio frame. In order to increase robustness, the duration of the sliding window used to calculate the time domain root mean square can be 1 second, that is, the average value of the time domain feature values of all audio frames within 1 second. The audio frames are filtered according to the calculated time domain root mean square, and audio frames greater than or equal to the third threshold are determined as speech frames with voice activity.
[0082] In other examples, for the enhanced second audio, the speech frames with voice activity can also be intercepted based on the frequency domain energy calculation of the audio frames included in the second audio. Specifically, the frequency domain energy calculation uses a short-time FFT to calculate the frequency domain signal of an audio frame, obtains the complex value x at N (for example, 256) frequency domain points, and then calculates the energy of the frequency domain signal. The sliding average is also used for the frequency domain energy calculation, and the sliding window length can be 2 seconds, that is, the average of the frequency domain energy values of all audio frames within 2 seconds. The audio frames are then filtered according to the calculated frequency domain energy values, and audio frames greater than or equal to the third threshold are determined as speech frames with voice activity.
[0083] Through the above implementation, the amount of data processing in the speech recognition process can be reduced, thereby improving the speech recognition efficiency.
[0084] Optionally, inputting a speech frame with speech activity into a speech recognition model to obtain a speech recognition result of the second audio includes:
[0085] Sending a speech frame with speech activity to a terminal device, where the terminal device is used to run a speech recognition model to obtain a speech recognition result of a second audio;
[0086] The speech recognition result of the second audio fed back by the receiving terminal device.
[0087] By deploying the speech recognition model on the terminal device and sending the speech frames with speech activity to the terminal device so as to use the terminal device to perform speech recognition processing, the performance requirements of the wearable device can be reduced and the battery life of the wearable device can be improved.
[0088] Refer to the following Figure 3 The data processing method of the wearable device according to the embodiment of the present application is described by taking a specific application scenario as an example.
[0089] like Figure 3 As shown, the data processing method can be implemented by a wearable device in conjunction with a mobile terminal, wherein the wearable device can be smart glasses and the mobile terminal can be a mobile phone. Figure 3 It is shown that the data processing method in the translation scenario includes steps S1 to S12, wherein steps S1 to S7 and S12 can be performed by smart glasses, and steps S8 to S11 can be performed by a mobile phone, as follows:
[0090] S1: Recording. The first microphone and the second microphone on the smart glasses continuously record, and transmit the collected audio data to the wake-up module for processing every period of time (one frame). The first microphone is arranged closer to the wearer's mouth than the second microphone.
[0091] S2: echo cancellation processing. During the voice interaction process, the signal formed after the sound played by the speaker is collected by the microphone is called an echo. The purpose of the microphone is to collect human speech. The echo needs to be eliminated, otherwise it will interfere with the microphone's collection of human voice and speech recognition. Specifically, the audio signal played by the speaker can be used to perform echo cancellation processing on the first audio collected by the first microphone and the second audio collected by the second microphone.
[0092] S3: Single-channel silence detection. During the voice interaction process, the user is silent for more than half of the time, and more than half of the audio signals collected by the recording are silent. Performing voice recognition processing on these audios with silence will lead to a waste of computing resources. If they are transmitted to the cloud for recognition, it will also waste network bandwidth. Therefore, it is necessary to detect the silent part, delete the silence, and then hand it over to the subsequent processing module. Specifically, the silence detection model can be used to perform a first silence detection on the first audio. When the first confidence level of the first audio is greater than or equal to the first threshold, the first silence detection result is that there is a voice segment in the first audio.
[0093] S4: Two-way silence detection: If the first silence detection result is that there is a voice segment in the first audio, a second silence detection is performed on the second audio using the silence detection model to obtain a second confidence level of the second audio.
[0094] S5: Double talk judgment. When the first confidence of the first audio is greater than or equal to the second threshold and the second confidence of the second audio is greater than or equal to the third threshold, it is determined that the sound sources of the first audio and the second audio include both the wearer sound source and the non-wearer sound source. When the first confidence of the first audio is less than the second threshold and the second confidence of the second audio is greater than or equal to the third threshold, it is determined that the sound sources of the first audio and the second audio include only the non-wearer sound source. When the first confidence of the first audio is greater than or equal to the second threshold and the second confidence of the second audio is less than the third threshold, it is determined that the sound sources of the first audio and the second audio include only the wearer sound source.
[0095] If the result of the double talk judgment is yes, that is, the sound sources of the first audio and the second audio include both the wearer's sound source and the non-wearer's sound source, then the process proceeds to step S6. If the result of the double talk judgment is no, that is, the sound sources of the first audio and the second audio include only the non-wearer's sound source, then the process proceeds to step S7.
[0096] In addition, the smart glasses can also transmit the result of the double-talk judgment to the mobile phone, so that the mobile phone performs subsequent processing on the second audio transmitted to the mobile phone according to the result of the double-talk judgment.
[0097] S6: Beamforming enhancement: If the result of the double talk determination is yes, the beamforming technology is used to perform directional speech enhancement on the second audio.
[0098] S7: Transmission. If the result of the dual-talk judgment is yes, the second audio after directional speech enhancement obtained in step S6 is transmitted to the mobile phone via Bluetooth, so that the mobile phone can execute step S8. If the result of the dual-talk judgment is no, the second audio is transmitted to the mobile phone via Bluetooth, so that the mobile phone can execute step S9.
[0099] S8: Voiceprint separation: If the result of the double talk judgment is yes, the second audio is subjected to voiceprint separation processing by a voiceprint separation model pre-set on the mobile phone to obtain a sound segment corresponding to a sound source other than the wearer.
[0100] S9: Speech recognition. If the result of the double talk judgment is yes, the speech recognition model is used to perform speech recognition processing on the sound segment corresponding to the non-wearer obtained based on step S8 to obtain a speech recognition result. If the result of the double talk judgment is no, the second audio obtained based on step S7 is directly subjected to speech recognition processing to obtain a speech recognition result. Among them, step S9 can be executed by the mobile phone, or sent by the mobile phone to the cloud server for execution, and the speech recognition result is obtained from the cloud server.
[0101] S10: Translation. The speech recognition result is translated into a preset language to obtain a target text in the preset language. Step S10 can be executed by the mobile phone, or sent by the mobile phone to a cloud server for execution, and the target text is obtained from the cloud server.
[0102] S11: Transmitting translation results. The mobile phone transmits the target text to the smart glasses via Bluetooth.
[0103] S12: Display. The smart glasses display the target text on the display interface.
[0104] Corresponding to the method provided in the embodiment of the present application, the embodiment of the present application also provides a data processing device for a wearable device. The wearable device includes a microphone array, the microphone array includes a first microphone and a second microphone, the target sound source of the first microphone is the wearer, and the target sound source of the second microphone is non-wearer. Figure 4 The figure is a structural block diagram of a data processing device according to an embodiment of the present application, and the data processing device may include:
[0105] A first detection module 401 is used to perform a first mute detection on the first audio collected by the first microphone to obtain a first mute detection result;
[0106] A second detection module 402 is configured to perform a second silence detection on the second audio collected by the second microphone in response to the first silence detection result indicating that a voice segment exists, to obtain a second silence detection result;
[0107] A sound source determination module 403, configured to determine the sound sources of the first audio and the second audio according to the first silence detection result and the second silence detection result;
[0108] The speech recognition module 404 is configured to perform speech recognition on the audio collected by the microphone corresponding to the target sound source in the current application scenario in response to the sound source including the target sound source in the current application scenario.
[0109] In one implementation, the first silence detection result includes a first confidence level of the first audio; wherein, when the first confidence level is greater than or equal to a first threshold, the first silence detection result is that there is a voice segment; and the sound source determination module is further configured to:
[0110] In response to the first confidence being greater than or equal to a second threshold, it is determined that the sound source includes a wearer sound source, and the second threshold is greater than the first threshold.
[0111] In one implementation, the second silence detection result includes a second confidence level of the second audio; and the sound source determination module is further configured to:
[0112] In response to the second confidence being greater than or equal to a third threshold, it is determined that the sound source includes a non-wearer sound source, and the third threshold is greater than the first threshold and less than the second threshold.
[0113] In one embodiment, the speech recognition module includes:
[0114] an enhancement submodule, configured to perform directional speech enhancement on the second audio in response to the target sound source in the current application scenario being a non-wearer sound source, and the sound source includes a wearer sound source and a non-wearer sound source, to obtain an enhanced second audio;
[0115] The recognition submodule is used to perform speech recognition on the enhanced second audio.
[0116] In one embodiment, the identification submodule includes:
[0117] A voiceprint separation unit, used to perform voiceprint separation processing on the enhanced second audio to obtain a sound segment emitted by a person other than the wearer;
[0118] The speech recognition unit is used to input the sound segment emitted by the non-wearer into the speech recognition model to obtain the speech recognition result of the second audio.
[0119] In one embodiment, the identification submodule includes:
[0120] An interception unit, configured to intercept at least one voice frame having voice activity from the enhanced second audio according to the second silence detection result;
[0121] The speech recognition unit is used to input the speech frame with speech activity into the speech recognition model to obtain the speech recognition result of the second audio.
[0122] In one embodiment, the speech recognition unit is further configured to:
[0123] Sending a speech frame with speech activity to a terminal device, where the terminal device is used to run a speech recognition model to obtain a speech recognition result of a second audio;
[0124] The speech recognition result of the second audio fed back by the receiving terminal device.
[0125] Figure 5 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 5 As shown, the electronic device includes: a memory 510 and a processor 520. The memory 510 stores a computer program that can be run on the processor 520. When the processor 520 executes the computer program, the method in the above embodiment is implemented. The number of the memory 510 and the processor 520 can be one or more.
[0126] The processor 520 may be a device on a chip (SOC), and the processor 520 may include a central processing unit (CPU), and may further include other types of processors.
[0127] The processor involved in the processor 520 may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units, NPU) and an image signal processor (Image Signal Processing, ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.
[0128] The processor 520 may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors.
[0129] The memory 510 may include a code storage area and a data storage area. The code storage area may store an operating system. The data storage area may store data created during the use of the electronic device, etc. In addition, the memory 510 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, a flash memory component, a universal flash storage (Universal Flash Storage, UFS), etc.
[0130] The memory 510 can be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or it can also be any computer-readable medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.
[0131] The electronic device also includes:
[0132] The communication interface 530 is used to communicate with external devices and perform data exchange transmission.
[0133] If the memory 510, the processor 520 and the communication interface 530 are implemented independently, the memory 510, the processor 520 and the communication interface 530 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0134] In some embodiments, the processor 520 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. Among them, the USB interface is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge an electronic device, and can also be used to transmit data between an electronic device and a peripheral device.
[0135] The electronic device may also include an external memory interface, which is used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 520 through the external memory interface to implement a data storage function. For example, files such as music and videos are stored in the external memory card.
[0136] Optionally, in a specific implementation, if the memory 510, the processor 520 and the communication interface 530 are integrated on a chip, the memory 510, the processor 520 and the communication interface 530 can communicate with each other through an internal interface.
[0137] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.
[0138] An embodiment of the present application provides a computer program product, including a computer program, which implements any of the above methods when executed by a processor.
[0139] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.
[0140] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.
[0141] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.
[0142] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).
[0143] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.
[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.
[0145] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0146] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.
[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.
[0148] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.
[0149] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.
[0150] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A data processing method for a wearable device, wherein the wearable device comprises a microphone array, wherein the microphone array comprises a first microphone and a second microphone, wherein a target sound source of the first microphone is a wearer, and a target sound source of the second microphone is a non-wearer; in, The data processing method comprises: Performing a first mute detection on the first audio collected by the first microphone to obtain a first mute detection result; In response to the first silence detection result indicating that a voice segment exists, performing a second silence detection on the second audio collected by the second microphone to obtain a second silence detection result; Determining sound sources of the first audio and the second audio according to the first silence detection result and the second silence detection result; In response to the sound source including a target sound source in a current application scenario, speech recognition is performed on audio collected by a microphone corresponding to the target sound source in the current application scenario.
2. The data processing method for a wearable device according to claim 1, wherein: The first silence detection result includes a first confidence level of the first audio; Wherein, when the first confidence level is greater than or equal to a first threshold, the first silence detection result is that a voice segment exists; Determining sound sources of the first audio and the second audio according to the first silence detection result and the second silence detection result includes: In response to the first confidence being greater than or equal to a second threshold, it is determined that the sound source includes the wearer sound source, and the second threshold is greater than the first threshold.
3. The data processing method for a wearable device according to claim 2, wherein: The second silence detection result includes a second confidence level of the second audio, and determining the sound sources of the first audio and the second audio according to the first silence detection result and the second silence detection result includes: In response to the second confidence being greater than or equal to a third threshold, it is determined that the sound source includes the non-wearer sound source.
4. The data processing method for a wearable device according to claim 1, wherein: In response to the sound source including a target sound source in a current application scenario, performing speech recognition on audio collected by a microphone corresponding to the target sound source in the current application scenario includes: In response to the target sound source in the current application scenario being a non-wearer sound source, and the sound source including a wearer sound source and a non-wearer sound source, performing directional speech enhancement on the second audio to obtain an enhanced second audio; Perform speech recognition on the enhanced second audio.
5. The data processing method for a wearable device according to claim 4, wherein: Performing speech recognition on the enhanced second audio, including: Performing voiceprint separation processing on the enhanced second audio to obtain a sound segment emitted by the non-wearer; The sound segment emitted by the non-wearer is input into a speech recognition model to obtain a speech recognition result of the second audio.
6. The data processing method for a wearable device according to claim 4, wherein: Performing speech recognition on the enhanced second audio, including: intercepting at least one voice frame with voice activity from the enhanced second audio according to the second silence detection result; The speech frame with speech activity is input into a speech recognition model to obtain a speech recognition result of the second audio.
7. The data processing method for a wearable device according to claim 6, wherein: Inputting the speech frame with speech activity into a speech recognition model to obtain a speech recognition result of the second audio includes: Sending the speech frame with speech activity to a terminal device, wherein the terminal device is used to run the speech recognition model to obtain a speech recognition result of the second audio; Receive the speech recognition result of the second audio fed back by the terminal device.
8. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
9. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Wearable sound source positioning tracking system and method
CN105223551A
Audio enhancement method and device, storage medium and wearable equipment
CN111935573A
Voice signal acquisition method and device, electronic equipment and storage medium
CN112992169A
Device and method for speech quality objective evaluation
JP2005077885A
Music playing apparatus, music playing method, music playing program, and integrated circuit
US20100208901A1
Cited By
Voice activity detection method, electronic equipment and storage medium
CN120564767A
Voice activity detection method, electronic device, and storage medium
CN120564767B
Sound acquisition and processing system based on cooperation of multiple microphone arrays
CN120954451A
Intelligent glasses voice processing method and system based on special-shaped microphone array
CN121687096A
Head-mounted real-time speech translation control method and system based on intelligent glasses
CN122635381A
