Speech signal processing method and related device

Abstract

Disclosed are a speech signal processing method and a related device, so as to ensure that wearable devices can achieve very good pickup signal-to-noise ratios in different noise scenarios. The method comprises: acquiring a first path of speech signals collected by a first pickup, wherein the first path of speech signals comprises an environmental noise signal, and the environmental noise signal indicates environmental noise of an environment where a wearable device is located; during the process of executing a speech service by using a second path of speech signals collected by a second pickup of the wearable device, acquiring a first duration on the basis of the first path of speech signals, wherein the first duration is a duration in which the noise energy of the current environmental noise signal has been continuously greater than a first energy threshold value; and on the basis of the first duration, determining whether to switch to execute the speech service by using a third path of speech signals collected by a third pickup of the wearable device, wherein when the noise energy of the environmental noise is greater than the first energy threshold value, the signal-to-noise ratio of the third path of speech signals is greater than the signal-to-noise ratio of the second path of speech signals.

Description

Voice signal processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 22, 2023, with application number 202311242905.5 and application name “Speech Signal Processing Method and Related Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of acoustic technology, and in particular to a speech signal processing method and related equipment. Background Art

[0003] During a call, the microphone on an electronic device is generally required to capture clear voice signals to ensure call quality. However, the real-world environment is often complex, especially in noisy areas such as roadsides, workshops, and factory areas. The extremely high ambient noise levels pose a significant challenge to clear calls with headphones.

[0004] Bone conduction microphones can effectively pick up voice signals in high-noise environments, ensuring call quality. The working principle of bone conduction microphones is that when a person speaks, the skin and jaw vibrate. The bone conduction microphone picks up the vibration signals of the skin and jaw and converts them into speech signals.

[0005] However, the voice signals collected by bone conduction microphones are severely distorted, resulting in poor vocal clarity, which limits their application scenarios. Therefore, there is an urgent need for a technical solution that can take into account various noise scenarios and ensure clear sound pickup and clear calls in various complex noise scenarios.

[0006] Summary of the Invention

[0007] The present application provides a speech signal processing method and related equipment to solve the problem in related technologies that it is difficult for headphones to obtain stable speech signals with a high signal-to-noise ratio in complex noise scenarios.

[0008] The first aspect provides a method for processing a voice signal. The method includes: obtaining a first voice signal collected by a first microphone, the first voice signal including an ambient noise signal, and the ambient noise signal indicating the ambient noise of the environment in which the wearable device is located. The first microphone is a microphone for collecting ambient noise, and the ambient noise in the first voice signal can accurately represent the ambient noise conditions of the environment in which the wearable device is located. During the period when a second voice signal collected by a second microphone of the wearable device is used to perform a voice service, a first duration is obtained based on the first voice signal, and the first duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than a first energy threshold. Based on the first duration, it is determined whether to switch to using a third voice signal to perform the voice service, the third voice signal is collected by a third microphone of the wearable device, and when the noise energy of the ambient noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

[0009] By providing at least two different microphones on the wearable device, the third microphone has a higher signal-to-noise ratio in high-noise environments. Based on the first voice signal, the first duration during which the ambient noise energy in the current environment is continuously greater than the first energy threshold is obtained. This allows the user to determine whether the current environment is high-noise or low-noise, and whether the ambient noise state is stable in a high-noise state, and then adaptively switch the voice signal used to perform voice services based on the first duration, ensuring that voice signals in complex environments have a high signal-to-noise ratio, clear and reliable voice, and stable voice services.

[0010] In one possible implementation, determining whether to switch to using a third-channel voice signal to perform voice services is performed based on the first duration, including: when the first duration is greater than a first duration threshold, determining to switch to using the third-channel voice signal to perform voice services; when the first duration is less than or equal to the first duration threshold, determining to still use the second-channel voice signal to perform voice services.

[0011] If the first duration is greater than the first duration threshold, the environment is currently experiencing relatively stable high noise levels. The voice signal used for voice services can be switched to a third voice signal with a higher signal-to-noise ratio in this environment, thereby improving voice clarity and ensuring stable voice services. If the first duration is less than or equal to the first duration threshold, the ambient noise is currently unstable. To avoid frequent voice signal switching, the current voice signal can be maintained without switching.

[0012] In one possible implementation, determining whether to switch to using a third voice signal to perform voice services based on the first duration includes: determining whether to switch to using a third voice signal to perform voice services based on the first duration and a second duration, where the second duration is the duration during which the second voice signal has been continuously used to perform voice services.

[0013] Including the duration of the current continuous use of the second voice signal to perform the voice service into the switching condition can avoid frequent switching of voice signals and improve the stability of the voice service.

[0014] In one possible implementation, determining whether to switch to using a third-channel voice signal to perform voice services based on the first duration and the second duration includes: determining to switch to using the third-channel voice signal to perform voice services when the first duration is greater than a first duration threshold and the second duration is greater than a second duration threshold; and determining to continue using the second-channel voice signal to perform voice services when the first duration is less than or equal to the first duration threshold and / or the second duration is less than or equal to the second duration threshold. Optionally, the second duration threshold is greater than the first duration threshold.

[0015] Switching the voice signal when the first duration is greater than the first duration threshold and the second duration is greater than the second duration threshold is simultaneously satisfied; switching the voice signal after the environmental noise is stable and the voice service is performed for a period of time using the second duration threshold can avoid frequent switching of the voice signal, ensure the stability of the voice service, and switch to a voice signal with a higher signal-to-noise ratio, which can improve the quality of the voice service.

[0016] In one possible implementation, while using a third voice signal to perform voice services, a third duration is obtained based on the first voice signal, where the third duration is the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to a second energy threshold, and the second energy threshold is less than or equal to the first energy threshold; based on the third duration, it is determined whether to switch to using the second voice signal to perform voice services, and when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

[0017] In one possible implementation, determining whether to switch to using the second voice signal to perform voice services is based on the third duration, including: when the third duration is greater than a third duration threshold, determining to switch to using the second voice signal to perform voice services; when the third duration is less than or equal to the third duration threshold, determining to still use the third voice signal to perform voice services.

[0018] In one possible implementation, determining whether to switch to using the second voice signal to perform voice services is based on the third duration, including: determining whether to switch to using the third voice signal to perform voice services based on the third duration and a fourth duration, the fourth duration being the duration for which the third voice signal has been continuously used to perform voice services.

[0019] In one possible implementation, determining whether to switch to using the third voice signal to perform voice services is based on the third duration and the fourth duration, including: when the third duration is greater than the third duration threshold, and the fourth duration is greater than the fourth duration threshold, determining to switch to using the second voice signal to perform voice services; when the third duration is less than or equal to the third duration threshold, and / or the fourth duration is less than or equal to the fourth duration threshold, determining to still use the third voice signal to perform voice services.

[0020] In one possible implementation, the method further includes: obtaining a forced switching instruction, the forced switching instruction indicating that a target voice signal is used to perform the voice service, the target voice signal being one of the second voice signal and the third voice signal; when the voice signal currently used to perform the voice service is different from the target voice signal, switching to using the target voice signal to perform the voice service.

[0021] Allows users to forcibly switch to voice signals used for voice services when needed, making switching more flexible and more adaptable to user needs.

[0022] In one possible implementation, after switching to using the target voice signal to perform voice services, the method further includes: when the target voice signal is the second-channel voice signal, obtaining a fifth duration based on the first-channel voice signal, and determining whether to switch to using the third-channel voice signal to perform voice services based on the fifth duration, the fifth duration being the duration during which the noise energy of the current ambient noise signal has been continuously greater than the first energy threshold; when the target voice signal is the third-channel voice signal, obtaining a sixth duration based on the first-channel voice signal, and determining whether to switch to using the second-channel voice signal to perform voice services based on the sixth duration, the sixth duration being the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to the second energy threshold. The combination of forced switching and adaptive switching can both meet user needs and ensure the stability of voice services.

[0023] In one possible implementation, the method further includes: obtaining initial noise energy of the ambient noise based on the first voice signal at the start of a voice service; determining to use the third voice signal for the voice service when the initial noise energy is greater than a fifth energy threshold; and determining to use the second voice signal for the voice service when the initial noise energy is less than or equal to the fifth energy threshold. Selecting the voice signal to use for the voice service based on the initial noise energy at the start of a voice service ensures a high signal-to-noise ratio for the voice signal at the start of the voice service, thereby guaranteeing voice service quality.

[0024] In one possible implementation, the method further includes extracting non-speech segments from the first voice signal to obtain an ambient noise signal; and obtaining noise energy from the ambient noise signal. The first voice signal may also contain a user voice signal. Extracting the non-speech segments from the first voice signal provides a relatively pure ambient noise signal. Thus, the ambient noise signal accurately represents the current ambient noise state, making voice switching decisions based on the ambient noise state more timely and accurate.

[0025] In a possible implementation, the first microphone is provided as a microphone on the wearable device; or the first microphone is provided as a microphone on a smart device connected to the wearable device.

[0026] In one possible implementation, performing a voice service using a second voice signal collected by a second microphone of a wearable device includes: performing noise reduction processing on the second voice signal using the first voice signal to obtain a noise-reduced voice signal; and performing the voice service using the noise-reduced voice signal. The first voice signal can more accurately represent the current ambient noise state, and the first and second voice signals are collected in the same environment. Therefore, performing noise reduction processing on the second voice signal using the first voice signal can more accurately filter the ambient noise in the second voice signal, thereby obtaining a noise-reduced voice signal with a higher signal-to-noise ratio, thereby improving the user's voice clarity and ensuring the quality of the voice service.

[0027] In one possible implementation, the second microphone is an air conduction microphone, and the third microphone is a bone conduction microphone; or the second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

[0028] A second aspect provides a speech signal processing device, which includes a processing module configured to execute the speech signal processing method according to the first aspect or any possible implementation of the first aspect.

[0029] Specifically, the processing module is configured to obtain a first voice signal collected by a first microphone, the first voice signal including an ambient noise signal, and the ambient noise signal indicating the ambient noise of the wearable device's environment. The first microphone is configured to collect ambient noise, and the ambient noise in the first voice signal can accurately represent the ambient noise of the wearable device's environment. While a second voice signal collected by a second microphone of the wearable device is being used to perform a voice service, the processing module is configured to obtain a first duration based on the first voice signal, the first duration being the duration during which the noise energy of the current ambient noise signal has been continuously greater than a first energy threshold. The processing module is configured to determine, based on the first duration, whether to switch to using a third voice signal to perform the voice service, the third voice signal being collected by a third microphone of the wearable device, and the signal-to-noise ratio of the third voice signal being greater than the signal-to-noise ratio of the second voice signal when the noise energy of the ambient noise is greater than the first energy threshold.

[0030] In one possible implementation, the processing module is used to: when the first duration is greater than the first duration threshold, determine to switch to using the third voice signal to perform the voice service; when the first duration is less than or equal to the first duration threshold, determine to still use the second voice signal to perform the voice service.

[0031] In a possible implementation, the processing module is configured to determine whether to switch to using the third voice signal to perform the voice service based on the first duration and the second duration, where the second duration is the duration for which the second voice signal has been continuously used to perform the voice service.

[0032] In one possible implementation, the processing module is configured to: determine to switch to using the third voice signal to perform the voice service when the first duration is greater than a first duration threshold and the second duration is greater than a second duration threshold; and determine to continue to use the second voice signal to perform the voice service when the first duration is less than or equal to the first duration threshold and / or the second duration is less than or equal to the second duration threshold. Optionally, the second duration threshold is greater than the first duration threshold.

[0033] In one possible implementation, the processing module is used to: during the period of using the third voice signal to perform voice services, obtain a third duration based on the first voice signal, where the third duration is the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to the second energy threshold, and the second energy threshold is less than or equal to the first energy threshold; determine whether to switch to using the second voice signal to perform voice services based on the third duration, and when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

[0034] In one possible implementation, the processing module is used to: when the third duration is greater than a third duration threshold, determine to switch to using the second voice signal to perform voice services; when the third duration is less than or equal to the third duration threshold, determine to still use the third voice signal to perform voice services.

[0035] In a possible implementation, the processing module is configured to determine whether to switch to using the third voice signal to perform the voice service based on the third duration and the fourth duration, where the fourth duration is the duration for which the third voice signal has been continuously used to perform the voice service.

[0036] In one possible implementation, the processing module is used to: when the third duration is greater than the third duration threshold, and the fourth duration is greater than the fourth duration threshold, determine to switch to using the second voice signal to perform voice services; when the third duration is less than or equal to the third duration threshold, and / or the fourth duration is less than or equal to the fourth duration threshold, determine to still use the third voice signal to perform voice services.

[0037] In one possible implementation, the processing module is used to: obtain a forced switching instruction, the forced switching instruction indicates that a target voice signal is used to perform a voice service, the target voice signal is one of a second voice signal and a third voice signal; when the voice signal currently used to perform the voice service is different from the target voice signal, switch to using the target voice signal to perform the voice service.

[0038] In one possible implementation, the processing module is configured to: when the target voice signal is the second-channel voice signal, obtain a fifth duration based on the first-channel voice signal, and determine whether to switch to using the third-channel voice signal to perform the voice service based on the fifth duration, where the fifth duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than the first energy threshold; when the target voice signal is the third-channel voice signal, obtain a sixth duration based on the first-channel voice signal, and determine whether to switch to using the second-channel voice signal to perform the voice service based on the sixth duration, where the noise energy of the current ambient noise signal has been continuously less than or equal to the second energy threshold. The combination of forced switching and adaptive switching can both meet user needs and ensure the stability of voice services.

[0039] In one possible implementation, the processing module is configured to: at the start of a voice service, obtain the initial noise energy of the ambient noise based on the first voice signal; when the initial noise energy is greater than a fifth energy threshold, determine to use the third voice signal for the voice service; and when the initial noise energy is less than or equal to the fifth energy threshold, determine to use the second voice signal for the voice service. Selecting the voice signal to use for the voice service based on the initial noise energy at the start of the voice service ensures a high signal-to-noise ratio (SNR) at the start of the voice service, thereby guaranteeing voice service quality.

[0040] In one possible implementation, the processing module is configured to extract non-speech segments from the first voice signal to obtain an ambient noise signal, and obtain noise energy from the ambient noise signal. The first voice signal may also contain a user voice signal. Extracting the non-speech segments from the first voice signal provides a relatively pure ambient noise signal. Thus, the ambient noise signal accurately represents the current ambient noise state, making voice switching decisions based on the ambient noise state more timely and accurate.

[0041] In a possible implementation, the first microphone is provided as a microphone on the wearable device; or the first microphone is provided as a microphone on a smart device connected to the wearable device.

[0042] In one possible implementation, the processing module is configured to: use the first voice signal to perform noise reduction processing on the second voice signal to obtain a noise-reduced voice signal; and use the noise-reduced voice signal to perform voice services. The first voice signal can more accurately represent the current ambient noise state, and the first and second voice signals are collected in the same environment. Therefore, using the first voice signal to perform noise reduction processing on the second voice signal can more accurately filter out the ambient noise in the second voice signal, thereby obtaining a noise-reduced voice signal with a higher signal-to-noise ratio, thereby improving user voice clarity and ensuring voice service quality.

[0043] In one possible implementation, the second microphone is an air conduction microphone, and the third microphone is a bone conduction microphone; or the second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

[0044] The third aspect provides an electronic device, characterized in that the electronic device includes a processor and a memory, the processor is coupled to the memory, and the processor is configured to execute the speech signal processing method in the first aspect or any one of the embodiments of the first aspect based on instructions stored in the memory.

[0045] In a possible implementation, the electronic device is a wearable device, and the electronic device further includes a second microphone and a third microphone, the second microphone is used to collect the second voice signal, and the third microphone is used to collect the third voice signal.

[0046] In a possible implementation, the electronic device further includes a first microphone, and the first microphone is used to collect the first voice signal.

[0047] The fourth aspect provides a computer-readable storage medium, characterized in that it includes instructions, which, when the computer-readable storage medium is run on a computer, enables the computer to execute the speech signal processing method in the first aspect or any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG1a is a schematic diagram of a real-time call between an air conduction microphone and a mobile phone provided by the present application;

[0049] FIG1b is a schematic diagram of a real-time call between a bone conduction microphone and a mobile phone provided by the present application;

[0050] FIG2 is a schematic diagram of a system architecture involved in a speech processing method provided in this application;

[0051] FIG3 is a schematic diagram of a pickup deployment method of a wearable device provided by the present application;

[0052] FIG4 is a schematic structural diagram of an electronic device provided by the present application;

[0053] FIG5a is a schematic diagram of the logical structure of an electronic device provided by the present application;

[0054] Figure 5b is a schematic diagram of the interaction between the modules in Figure 5a;

[0055] FIG6 is a flow chart of a speech processing method provided by the present application;

[0056] FIG7 is a simulation diagram of voice switching based on the solution provided by the present application;

[0057] FIG8 is a schematic structural diagram of a speech processing device provided in this application. DETAILED DESCRIPTION

[0058] The present application provides a voice signal processing method and related equipment to ensure the voice quality of voice services in complex scenarios.

[0059] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art will appreciate that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0060] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0061] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0062] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0063] In voice scenarios such as calls, voice interaction, recording, voice entertainment, and video conferencing, users sometimes wear wearable devices such as headphones. The wearable devices collect user voice signals, allowing the wearable devices or electronic devices connected to the wearable devices to implement related voice services based on the user voice signals. Using wearable devices to participate in voice services not only frees the user's hands, allowing the user to engage in other activities at the same time, but also allows the wearable devices to be closer to the user's voice source, picking up voice signals with a higher signal-to-noise ratio, improving the quality of voice services. Furthermore, this reduces the risk of privacy leaks and the impact on others, bringing convenience to users' work, work, and study.

[0064] Taking a call scenario as an example, in most cases, air-conducted voice signals are used for the transmission of voice signals. The transmission process can be shown in Figure 1a, where the user on side A and the user on side B make a voice call based on a wearable device (here taking headphones as an example) and an electronic device connected to the wearable device (here taking a mobile phone as an example, it can also be a computer or tablet, etc.). Taking side A speaking and side B listening as an example, the above transmission process is: ① The user on side A speaks and sends a voice signal S; ② The air-conduction microphone on the earphone on side A collects the voice signal S; ③ The earphone on side A sends the voice signal S to the mobile phone on side A; ④ The mobile phone on side A encodes and compresses the voice signal S and transmits it to the mobile phone on side B based on a wireless network; ⑤ The mobile phone on side B decodes the received signal and restores it to obtain the voice signal S; ⑥ The mobile phone on side B plays the voice signal S based on the speaker of the mobile phone on side B (or the wearable device connected to the mobile phone on side B).

[0065] When the above-mentioned air-conducted voice signal is transmitted, the voice signal S will collect environmental noise in addition to the voice of the user on side A. If the environmental noise is loud, it will affect the listening experience of the user on side B. Therefore, in some high-noise scenarios, such as mining and rescue operations, bone conduction voice signals are generally used for information transmission. This is because bone conduction microphones collect less ambient noise and have a high audio signal-to-noise ratio, which can greatly ensure call efficiency in these scenarios. The transmission process is shown in Figure 1b, where users on side A and side B are making a voice call based on their mobile phones. Taking side A speaking and side B (assuming side B does not change) listening as an example, the above transmission process is as follows: ① User on side A speaks, emitting a voice signal S1; ② The bone conduction microphone on the head-mounted device on side A (such as headphones) is placed in close contact with the face of user on side A and collects the audio signal S2 (also called bone conduction voice signal S2) when the user on side A speaks based on skin vibration. This bone conduction voice signal S2 is sent to the mobile phone on side A (e.g., via a Bluetooth connection between the head-mounted device and the mobile phone); ③ Mobile phone on side A encodes and compresses the bone conduction voice signal S2 and transmits it to mobile phone on side B via a wireless network; ④ Mobile phone on side B decodes the received signal and restores the bone conduction voice signal S2; ⑤ Mobile phone on side B plays the bone conduction voice signal S2 through the air conduction speaker on side B.

[0066] The generation models for both air-conduction and bone-conduction speech signals use a convolutional excitation source with an excitation channel (i.e., excitation source * excitation channel = speech signal). However, their specific generation principles differ: The excitation source for both is the vocal cords, but the excitation channel for air-conduction speech is the pharynx and oral cavity, while the excitation channel for bone-conduction speech is the muscles and bones. This difference in excitation channels leads to significant differences in the final listening experience. Because the excitation channel is airborne, air-conduction speech signals exhibit less distortion and greater brightness. However, air-conduction microphones are sensitive to ambient noise, which results in a significant inclusion of ambient noise. In contrast, because the excitation channel is solid and soft, bone-conduction speech signals exhibit a higher signal-to-noise ratio but exhibit greater distortion and a darker brightness than air-conduction speech signals, making them less comfortable to listen to. Furthermore, the sound heard by the other party (e.g., party B) on party A sounds unnatural, and in severe cases, it may be difficult to distinguish the speaker, thus affecting call efficiency.

[0067] Therefore, there is an urgent need for a technical solution that can take into account various noise scenarios and ensure clear and stable voice signals in various complex noise scenarios.

[0068] In order to solve the above technical problems, this application provides the following technical solutions.

[0069] As shown in Figure 2, Figure 2 is a schematic diagram of the system architecture involved in a speech processing method provided in this application. Referring to Figure 2, the system may include a wearable device 201 and a smart device 202. The wearable device 201 and the smart device 202 are connected to communicate via a wired or wireless method. Among them, the wireless method can specifically be Bluetooth, wireless local area network (WLAN), ZigBee or near field communication (NFC), etc., or it can also be other new generation wireless short-range communication technologies, such as NearLink, etc. In an embodiment, the wearable device 201 can be a headset, smart glasses, an augmented reality (AR) head display device, a virtual reality (VR) head display device or a neck massager and other devices. The smart device 202 can be a mobile phone, a tablet computer, a computer, a car computer and other devices.

[0070] In an embodiment of the present application, the wearable device 201 is configured to collect voice signals and transmit the collected voice signals to the smart device 202. The smart device 202 is configured to receive the voice signals transmitted by the wearable device 201 and perform corresponding voice tasks based on the received voice signals. In this embodiment, the voice service may be a call service, a voice interaction service, a recording service, a voice entertainment service (such as karaoke), a video conferencing service, or other services involving voice signal processing.

[0071] The system includes a first sound pickup (or sound pickup sensor, microphone, etc.) for collecting the ambient sound signal of the environment in which the wearable device 201 is located. In one possible implementation, the first sound pickup can be a sound pickup arranged on the wearable device 201. In another possible implementation, the first sound pickup can also be a sound pickup arranged on the smart device 202. Since the wearable device 201 and the smart device 202 are close to each other and are in the same environment, the ambient sound signal collected by the sound pickup on the smart device 202 can be considered as the ambient sound signal of the environment in which the wearable device 201 is located. In addition, the smart device 202 has a certain distance from the user sound source, and the user voice signal in the collected ambient sound signal is less, which can more accurately represent the noise situation in the environment. Optionally, in a possible implementation, the system also includes other devices in the same environment and connected to the smart device, such as wearable devices (such as smart watches), smart smart devices (tablets, laptops) or Internet of Things devices (smart home appliances, smart toys, smart robots or industrial Internet of Things devices), etc. The first sound pickup can be a sound pickup on other devices.

[0072] When the first sound pickup is a sound pickup on the wearable device 201, the first sound pickup can be a bone conduction microphone or an air conduction microphone. Optionally, the first sound pickup can be deployed toward the external environment, away from the human voice source, to collect more accurate environmental noise. When the first sound pickup is a sound pickup on the smart device 202, the first sound pickup can be an air conduction microphone. Of course, when the first sound pickup is a sound pickup on the smart device 202, the first sound pickup can also be a bone conduction microphone.

[0073] The wearable device 201 includes at least two microphones, for example, a second microphone and a third microphone. The second microphone and the third microphone are different microphones. In a low-noise environment, the signal-to-noise ratio of the voice signal collected by the second microphone is higher than that of the voice signal collected by the third microphone, that is, the human voice is clearer in the voice signal collected by the second microphone in a low-noise environment. In a high-noise environment, the signal-to-noise ratio of the voice signal collected by the third microphone is higher than that of the voice signal collected by the second microphone, that is, the human voice is clearer in the voice signal collected by the third microphone in a high-noise environment. In one possible implementation, the second microphone is an air conduction microphone, and the third microphone is a bone conduction microphone. In another possible implementation, the second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

[0074] The voice signal collected in real time by the first microphone is called the first voice signal, which includes the ambient noise signal. The voice signal collected in real time by the second microphone is called the second voice signal. The voice signal collected in real time by the third microphone is called the third voice signal. The second and third microphones serve as the primary microphones, and the second and third voice signals are used to perform voice services. In this embodiment, the first microphone serves as the secondary microphone. The first voice signal collected by the first microphone is primarily used to analyze the ambient noise level. This ambient noise level is used to determine which of the second and third voice signals should be used for voice services, thus achieving adaptive voice signal switching. When performing voice services, at most one of the second and third voice signals is used simultaneously for processing. When a preset voice switching condition is met, the voice signal currently used for voice services can be switched from one of the two signals to the other.

[0075] When the first microphone is a microphone on the wearable device 201, in one possible implementation, the first microphone and the second microphone can be the same microphone, and the first voice signal and the second voice signal are the same voice signal, thereby reducing the number of microphones on the wearable device 201 and reducing the cost of the wearable device 201.

[0076] When the first microphone is a microphone on the wearable device 201, in another possible implementation, the first microphone, the second microphone and the third microphone on the wearable device 201 are three different microphones. By setting a first microphone dedicated to collecting ambient noise, the energy accuracy of the estimated ambient noise in the first voice signal can be improved, thereby ensuring that the voice switching timing can be grasped in a timely and accurate manner, thereby improving the voice switching efficiency. Optionally, a microphone with a lower sensitivity than the second microphone can be selected as the first microphone to reduce the influence of the user's voice signal in the first voice signal, increase the proportion of ambient noise, and enable the first voice signal to more accurately characterize the ambient noise situation. As shown in Figure 3, Figure 3 is a schematic diagram of a wearable device provided by the present application. Figure 3 takes the wearable device as headphones as an example. It can be understood that the shape of the headphones here is only for illustration, and the headphones can also be in other forms, such as in-ear or neck-hanging types. The second microphone and the third microphone can be set in a position close to the user and close to the user's vocal part, so that the second microphone and the third microphone can collect the user's voice signal more clearly, and improve the signal-to-noise ratio of the second voice signal and the third voice signal. The position and orientation of the first microphone on the wearable device 201 can be reasonably designed. For example, the first microphone is set on the outside of the wearable device 21, and the first microphone is oriented away from the user. When the user wears the wearable device, the first microphone is far away from the user's sound source, and can better collect the environmental noise signal while collecting as little of the user's own voice signal as possible. Figure 3 only shows the possible settings of the first microphone, the second microphone, and the third microphone of the wearable device 201, which should not be understood as a limitation to this application. The actual implementation can be based on the product form of the wearable device 201.

[0077] In one possible implementation, the wearable device 201 can determine in real time whether the voice switching condition is currently met based on the first voice signal, and when the voice switching condition is currently met, determine to switch to using the other voice signal of the second voice signal and the third voice signal to perform the voice service. In this case, after determining to switch the voice signal for performing the voice service, the wearable device 201 can only send the switched voice signal for performing the voice service to the smart device 202, without sending the voice signal for performing the voice service before the switch, and the smart device 202 uses the switched voice signal to perform the voice service. At this time, the second microphone and the third microphone can both be in the on state, that is, the second microphone and the third microphone are still collecting voice signals in real time, but the wearable device 202 only sends the switched voice signal to the smart device 202. Alternatively, after determining to switch the voice signal for performing the voice service, the wearable device 202 turns off the microphone corresponding to the voice signal used to perform the voice service before the switch, and turns on the microphone corresponding to the voice signal used to perform the voice service after the switch, thereby turning on the microphone currently required between the second microphone and the third microphone, and temporarily turning off the other microphone that is not currently required, which can reduce the power consumption of the wearable device 202. Of course, the wearable device 201 can also send the first voice signal and the second voice signal to the smart device 202 at the same time. After determining to switch the voice signal for performing the voice service, the wearable device 201 can send a switching instruction to the smart device 202, and the smart device 202 switches the voice signal for performing the voice service according to the switching instruction, and uses the switched voice signal to perform the voice service.

[0078] In another possible implementation, the smart device 202 can determine in real time whether the voice switching condition is currently met based on the first voice signal. When the first microphone is a microphone on the wearable device 201, the wearable device sends the first voice signal to the smart device 202 in real time. When the first microphone is a microphone on the smart device 202, the smart device 202 collects the first voice signal in real time through the first microphone. When the smart device 202 determines whether the voice switching condition is currently met, the wearable device 201 can simultaneously send the first voice signal and the second voice signal to the smart device 202. When the smart device 202 determines that the voice switching condition is currently met, it is determined to switch to using the other voice signal of the second voice signal and the third voice signal to perform the voice service, and use the switched voice signal to perform the voice service. Switching the voice signal for performing the voice service at the back end (smart device 202) can reduce the switching delay and reduce the voice jamming and fragmentation introduced by the switching. Alternatively, when the smart device 202 determines whether the voice switching condition is currently met, when the smart device 202 determines that the voice switching condition is currently met, it sends a switching instruction to the wearable device 201, and the wearable device 201 only sends the switched voice signal to the smart device 202.

[0079] The specific implementation of determining whether the voice switching condition is currently met in real time based on the first voice signal will be described in the following method embodiment.

[0080] By setting at least two different microphones on the wearable device 201, when the ambient noise changes, the appropriate voice signal can be adaptively adjusted to perform voice services, ensuring that no matter in a high-noise environment or a low-noise environment, or when the ambient noise situation changes drastically, a higher signal-to-noise ratio and clearer voice signal can be used to perform voice services, thereby improving the quality of voice services.

[0081] It should be noted that the system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0082] Please refer to Figure 4, which is a schematic diagram of the structure of an electronic device provided in this application. Optionally, the electronic device is the wearable device 201 or smart device 202 shown in Figure 2. The electronic device includes one or more processors 401, a communication bus 402, a memory 403, and one or more communication interfaces 404.

[0083] Processor 401 is a general-purpose central processing unit (CPU), a network processing processor (NP), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. Optionally, the PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0084] Communication bus 402 is used to transmit information between the above components. Optionally, communication bus 402 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus.

[0085] Optionally, the memory 403 is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 403 exists independently and is connected to the processor 401 via the communication bus 402, or the memory 403 is integrated with the processor 401.

[0086] Communication interface 404 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 404 includes a wired communication interface and, optionally, a wireless communication interface. Examples of wired communication interfaces include an Ethernet interface. Optionally, the Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface can be a wireless local area network (WLAN) interface, a Bluetooth interface, a ZigBee interface, an NFC interface, a StarFlash interface, a cellular network communication interface, or a combination thereof.

[0087] Optionally, in some embodiments, the electronic device includes multiple processors, such as processor 401 and processor 405 shown in Figure 4. Each of these processors is a single-core processor or a multi-core processor. Optionally, the processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0088] When the electronic device is the wearable device 201 shown in Figure 2, it also includes a second microphone (not shown) and a third microphone (not shown). The second microphone is used to collect the second voice signal, and the third microphone is used to collect the third voice signal. The communication interface 404 is used to send the second voice signal and / or the third voice signal to the smart device. Optionally, the electronic device may further include a first microphone, which is used to collect the first voice signal. Thus, the electronic device can determine whether the voice switching condition is currently met based on the first voice signal to determine whether the voice signal used to perform the voice service is switched from the second voice signal to the third voice signal, or from the third voice signal to the second voice signal. Optionally, the communication interface 404 is also used to send the first voice signal to the smart device. Thus, the smart device can determine whether the voice switching condition is currently met based on the first voice signal to determine whether the voice signal used to perform the voice service is switched from the second voice signal to the third voice signal, or from the third voice signal to the second voice signal.

[0089] When the electronic device is the smart device 202 shown in Figure 2, the electronic device may further include an output device (not shown) and an input device (not shown). The output device communicates with the processor and can display information in a variety of ways. For example, the output device is a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor and can receive user input in a variety of ways. For example, the input device 407 includes one or more of a mouse, a keyboard, a touch screen device, or a sensor device. The communication interface 404 is used to receive a second voice signal and / or a third voice signal sent by the wearable device. Optionally, the communication interface 404 is used to receive a first voice signal sent by the wearable device. Optionally, the electronic device may further include a first microphone, which is used to collect the first voice signal. Thus, the smart device can determine whether the voice switching condition is currently met based on the first voice signal to determine whether the voice signal used to perform the voice service will be switched from the second voice signal to the third voice signal, or from the third voice signal to the second voice signal.

[0090] In some embodiments, the memory 403 is used to store program code 406 for executing the solution of the present application, and the processor 401 is capable of executing the program code 406 stored in the memory 403. The program code 406 includes one or more software modules, and the electronic device can implement the speech processing method provided in the embodiment of FIG. 6 below through the processor 401 and the program code 406 in the memory 403.

[0091] As shown in Figures 5a and 5b, Figure 5a is a schematic diagram of the logical structure of an electronic device provided by this application; Figure 5b is a schematic diagram of the interaction between the modules in Figure 5a. In this embodiment, the electronic device includes a first voice signal acquisition module, a second voice signal acquisition module, a third voice signal acquisition module, an ambient noise energy calculation module, a handover determination module, a handover control module, and a voice service processing module.

[0092] The first voice signal acquisition module is used to acquire the first voice signal collected in real time by the first microphone. The second voice acquisition module is used to acquire the second voice signal collected in real time by the second microphone. The third voice acquisition module is used to acquire the third voice signal collected in real time by the third microphone.

[0093] The environmental noise energy calculation module is used to calculate the noise energy of the environmental noise signal in the first-channel voice signal in real time according to the first-channel voice signal.

[0094] During the period when the second voice signal is currently used to perform the voice service, the switching judgment module is used to compare the noise energy with the first noise threshold after each noise energy is obtained, and to count the first duration during which the noise energy is continuously greater than the first noise threshold. In one possible implementation, the switching judgment module compares the first duration with the first duration threshold, and when the first duration is greater than the first duration threshold, determines to switch to using the third voice signal to perform the voice service, and generates a switching instruction. In another possible implementation, the switching judgment module compares the first duration with the first duration threshold, and also compares the second duration with the second duration threshold, and when the first duration is greater than the first duration threshold and the second duration is greater than the second duration threshold, determines to switch to using the third voice signal to perform the voice service. The second duration is the duration during which the second voice signal has been continuously used to perform the voice service, and generates a switching instruction.

[0095] During the period when the third voice signal is currently being used to perform the voice service, the switching judgment module is configured to compare the noise energy with the second noise threshold each time noise energy is obtained, and to count the third duration during which the noise energy is continuously greater than the second noise threshold. In one possible implementation, the switching judgment module compares the third duration with the third duration threshold, and when the third duration is greater than the third duration threshold, determines to switch to using the second voice signal to perform the voice service, and generates a switching instruction. In another possible implementation, the switching judgment module compares the third duration with the third duration threshold, and also compares the fourth duration with the fourth duration threshold, and when the third duration is greater than the third duration threshold and the fourth duration is greater than the fourth duration threshold, determines to switch to using the second voice signal to perform the voice service. The fourth duration is the duration during which the third voice signal has been continuously used to perform the voice service, and generates a switching instruction.

[0096] The switching control module switches the voice signal used to perform the voice service from the second voice signal to the third voice signal, or switches the voice signal used to perform the voice service from the third voice signal to the second voice signal according to the switching instruction.

[0097] The voice service processing module is used to perform voice services based on the switched voice signals. The voice service processing module can perform noise reduction, echo reduction, and encoding on the voice signals, and then perform corresponding voice services based on the processed voice signals.

[0098] As shown in Figure 6, Figure 6 is a flow chart of a voice processing method provided by this application. This embodiment is performed by the electronic device shown in Figure 4, which can be the wearable device shown in Figure 2 or the smart device shown in Figure 2. This embodiment includes the following steps:

[0099] S601: Acquire a first voice signal collected by a first microphone, where the first voice signal includes an ambient noise signal, and the ambient noise signal indicates ambient noise in an environment where the wearable device is located.

[0100] The first microphone can be a microphone on a wearable device or a microphone on a smart device connected to the wearable device. The first microphone is used to collect ambient noise from the wearable device's environment. Therefore, the first voice signal collected by the first microphone includes the ambient noise signal. The first voice signal is collected in real time by the first microphone, and the electronic device also performs noise analysis on the most recently collected voice signal by the first microphone in real time.

[0101] In a possible implementation, the first voice signal may be used as the environmental noise signal.

[0102] In another possible implementation, non-speech segments from the first voice signal can be extracted as the ambient sound signal. For example, voice activity detection (VAD) can be used to extract the non-speech segments from the first voice signal. Thus, the ambient sound signal can more accurately represent the noise level in the environment, ensuring the accuracy and efficiency of voice switching decisions.

[0103] S602: While performing a voice service using a second voice signal collected by a second microphone of the wearable device, obtain a first duration based on the first voice signal, where the first duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than a first energy threshold.

[0104] While the second voice signal is being used to perform the voice service, a first duration is obtained based on the first voice signal, wherein the first duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than the first energy threshold.

[0105] The electronic device calculates the noise energy of the latest ambient noise signal and determines whether the noise energy is greater than a first noise threshold. When the noise energy is greater than the first noise threshold, the electronic device records a first duration in the time domain during which the noise energy of the ambient noise signal is continuously greater than the first energy threshold up to the present time.

[0106] Specifically, the electronic device frames the ambient noise signal in the time domain, and then calculates the noise energy of each frame of the ambient noise signal. The noise energy can be the square value of the amplitude of the ambient noise signal of the frame. In another implementation, the noise energy can also refer to a parameter that characterizes the intensity of the ambient noise, such as the loudness, power, sound intensity or sound pressure of the noise. The noise energy of each frame is compared with the first energy threshold. If the noise energy of the frame is greater than the first energy threshold, the value of the first counter is increased by 1. If the noise energy of the frame is less than or equal to the first energy threshold, the value of the first counter is cleared to 0. It should be noted that the first counter needs to count according to the timing of the frame. Exemplarily, a group of continuous frames in the time domain, (frame 1, frame 2, frame 3, frame 4), frame 1 is the earliest frame in the time domain, frame 4 is the latest frame in the time domain, the noise energy of frame 1 is greater than the first energy threshold, then the value of the first counter is increased by 1, the noise energy of frame 2 is greater than the first energy threshold, then the value of the first counter is increased by 1 again, the noise energy of frame 3 is less than the first energy threshold, then the value of the first counter is cleared to 0, at this time the first duration is 0, the noise energy of frame 4 is greater than the first energy threshold, then the value of the first counter is increased by 1, at this time the value of the first counter is 1.

[0107] Of course, the duration during which the noise energy of the ambient noise signal is continuously greater than the first energy threshold may also refer to the cumulative duration during which the noise energy of the ambient noise signal is greater than the first energy threshold in the time domain within the sliding time window.

[0108] S603: Determine whether to switch to using a third voice signal to perform voice services based on the first duration, where the third voice signal is collected by a third microphone of the wearable device, and when the noise energy of the ambient noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

[0109] The first duration is a key criterion for determining whether to switch the voice signal used for voice services to the third voice signal. If the first duration is greater than the first duration threshold, it indicates that the current ambient noise level is relatively stable and high. This prevents frequent voice signal switching in complex ambient noise environments from affecting voice coherence.

[0110] In one possible implementation, the first duration being greater than a first duration threshold can be used as the sole condition for currently satisfying the voice switching condition. Specifically, the electronic device determines whether the first duration is greater than the first duration threshold. If the first duration is greater than the first duration threshold, the electronic device determines to switch to using the third voice signal to perform the voice service. If the first duration is less than or equal to the first duration threshold, the electronic device determines to continue using the second voice signal to perform the voice service. In one possible implementation, the current value m of the first counter (m is a natural number indicating that the noise energy of m consecutive frames in the time domain is greater than the first energy threshold. Since the frame length is fixed, the value m can represent the first duration) can be compared with a first preset value n (n is a positive integer indicating the number of consecutive frames greater than the first energy threshold. Since the frame length is fixed, n can also represent the first duration threshold). When m is greater than n, the first duration can be considered greater than the first duration threshold. When m is less than or equal to n, the first duration can be considered less than or equal to the first duration threshold.

[0111] In another possible implementation, a first duration greater than a first duration threshold and a second duration greater than a second duration threshold can be used as a voice switching condition. The second duration is the duration for which the second voice signal has been used to perform voice services. Optionally, the second duration threshold is greater than the first duration threshold. The second duration threshold can be considered a switching protection period. That is, after switching the voice signal, the switched voice signal must remain in use for a period greater than the second duration threshold before the voice signal is allowed to switch again. This prevents frequent voice signal switching that could affect voice service quality. Specifically, the electronic device compares the first duration with the first duration threshold, and the second duration with the second duration threshold. If the first duration is greater than the first duration threshold and the second duration is greater than the second duration threshold, the voice switching condition is considered met and the voice service is switched to the third voice signal. If the first duration is less than or equal to the first duration threshold and / or the second duration is less than or equal to the second duration threshold, the voice switching condition is considered not met and the voice service is continued to be performed using the second voice signal.

[0112] The first duration threshold is used to ensure that the voice signal is not switched simply because the noise energy in a particular frame exceeds the first energy threshold, thereby ensuring stable and continuous voice service. The second duration threshold is used to ensure that when switching to the second voice signal for voice service, a switch to the third voice signal is not performed immediately after a short period of time, thereby ensuring stable and continuous voice service.

[0113] In this embodiment, when the noise energy of the ambient noise is greater than the first energy threshold, the signal-to-noise ratio of the third-channel voice signal is greater than the signal-to-noise ratio of the second-channel voice signal. That is, in a relatively noisy environment, the voice signal used to perform the voice service is switched to a voice signal with a higher signal-to-noise ratio in the current environment, ensuring that the voice signal remains clear in a high-noise environment, thereby ensuring the quality of the voice service.

[0114] S604: While performing voice services using the third voice signal, obtain a third duration based on the first voice signal, where the third duration is the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to a second energy threshold, and the second energy threshold is less than or equal to the first energy threshold.

[0115] While the third voice signal is being used to perform the voice service, a third duration is obtained based on the first voice signal, wherein the third duration is the duration during which the noise energy of the current ambient noise signal is continuously less than or equal to the second energy threshold.

[0116] The electronic device calculates the noise energy of the latest ambient noise signal and determines whether the noise energy is less than or equal to a second energy threshold. When the noise energy is less than or equal to the second energy threshold, the electronic device records a third duration in the time domain during which the noise energy of the ambient noise signal is continuously less than or equal to the second energy threshold up to the current time.

[0117] Specifically, the electronic device frames the ambient noise signal in the time domain and then calculates the noise energy of each frame. The noise energy is the square of the amplitude of the ambient noise signal in that frame. The noise energy of each frame is compared with a second energy threshold. If the noise energy of the frame exceeds the second energy threshold, the value of the second counter is incremented by 1. If the noise energy of the frame exceeds the second energy threshold, the value of the first counter is reset to 0. It should be noted that the second counter needs to count according to the frame timing.

[0118] Of course, when the noise energy of the ambient noise signal has been continuously less than or equal to the second energy threshold, it can also refer to the cumulative duration in the sliding time window that the noise energy of the ambient noise signal in the time domain is less than or equal to the second energy threshold.

[0119] In this embodiment, the second energy threshold is less than or equal to the first energy threshold. The first duration threshold and the third duration threshold may be the same or different. The second duration threshold and the fourth duration threshold may be the same or different.

[0120] S605: Determine whether to switch to using the second voice signal to perform the voice service based on the third duration, when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

[0121] The third duration is a key criterion for determining whether to switch the voice signal used for voice services to the second voice signal. If the second duration is greater than the first duration threshold, it indicates that the current ambient noise level is relatively stable and low, preventing frequent voice signal switching in complex ambient noise environments from affecting voice coherence.

[0122] In one possible implementation, the third duration being greater than a third duration threshold can be used as the sole condition for currently satisfying the voice switching condition. Specifically, the electronic device determines whether the third duration is greater than the third duration threshold. If the third duration is greater than the third duration threshold, the electronic device determines to switch to using the second voice signal to perform the voice service. If the third duration is less than or equal to the third duration threshold, the electronic device determines to continue using the third voice signal to perform the voice service. In one possible implementation, the current value p of the second counter (p is a natural number indicating that the noise energy of p consecutive frames in the time domain is greater than the first energy threshold. Since the frame length is fixed, the value p can represent the third duration) can be compared with a second preset value q (q is a positive integer indicating the number of consecutive frames greater than the second energy threshold. Since the frame length is fixed, q can also represent the third duration threshold). When p is greater than q, the third duration can be considered greater than the third duration threshold. When p is less than or equal to q, the third duration can be considered less than or equal to the third duration threshold.

[0123] In another possible implementation, a third duration greater than a third duration threshold, and a fourth duration greater than a fourth duration threshold, can be used as a voice switching condition. The fourth duration is the duration that the third voice signal has been used continuously for voice service. Optionally, the fourth duration threshold is greater than the third duration threshold. The fourth duration threshold can be considered a switching protection period. That is, after switching the voice signal, the switched voice signal must remain in use for a period greater than the fourth duration threshold before the voice signal is allowed to switch again. This prevents frequent voice signal switching that affects voice service quality. Specifically, the electronic device compares the third duration with the third duration threshold, and the fourth duration with the fourth duration threshold. If the third duration is greater than the third duration threshold and the fourth duration is greater than the fourth duration threshold, the voice switching condition is considered met and the voice service is switched to the third voice signal. If the third duration is less than or equal to the third duration threshold and / or the fourth duration is less than or equal to the fourth duration threshold, the voice switching condition is considered not met and the voice service is continued to be performed using the third voice signal.

[0124] The third duration threshold is used to ensure that the voice signal is not switched simply because the noise energy of a frame is less than or equal to the second energy threshold, thereby ensuring the stability and continuity of voice services. The fourth duration threshold is used to ensure that when switching to the third voice signal for voice services, a switch to the second voice signal is not performed within a short period of time, thereby ensuring the stability and continuity of voice services.

[0125] In this embodiment, when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal. That is, in a low-noise environment, the voice signal used to perform the voice service is switched to a voice signal with a higher signal-to-noise ratio in the current environment, ensuring that the voice signal remains clear in a high-noise environment, thereby ensuring the quality of the voice service.

[0126] In this embodiment, at least two different microphones are provided on the wearable device. The second microphone has a higher signal-to-noise ratio in a low-noise environment, and the third microphone has a higher signal-to-noise ratio in a high-noise environment. The duration for which the current environment is continuously in a certain state is obtained according to the environmental noise signal, and whether the current environment is a high-noise environment or a low-noise environment is determined according to the duration, and then the voice signal used to perform the voice service is adaptively switched, and two-way switching is supported to ensure that the voice signal in a complex environment has a high signal-to-noise ratio, the voice is clear and reliable, and the stability of the voice service is ensured. Furthermore, this embodiment also sets a switching protection time interval to avoid frequent switching of voice signals affecting the quality of the voice service. Moreover, this embodiment does not require a complex noise reduction algorithm, that is, it can ensure that voice services can be processed based on voice signals with a high signal-to-noise ratio in all scenarios, can reduce the computing power requirements for electronic equipment, and reduce the requirements for noise reduction algorithms, and can ensure the stability of voice services in a low-cost manner.

[0127] Optionally, the electronic device has an adaptive switching function (i.e., the method in Figure 6) and a forced switching function. When the forced switching function is turned on and the adaptive switching function is turned off, the voice signal used to perform the voice service is switched according to the forced switching instruction. When the forced function is turned on, the adaptive switching function starts, and the forced switching instruction is executed first. Specifically, when the user issues a forced switching instruction due to actual needs, the forced switching instruction instructs to switch to the target voice signal among the first voice signal and the second voice signal, and the forced switching instruction is executed first. When the voice signal currently used to perform the voice service is inconsistent with the target voice signal, the target voice signal is switched to perform the voice service. After the forced switching is executed, when the switching conditions are met, the voice signal is switched according to the adaptive switching instruction. When the forced function is turned off and the adaptive switching function is turned on, the voice signal is switched only according to the adaptive switching instruction. When both the forced switching function and the adaptive switching function are turned off, the current voice signal is maintained to perform the voice service without any switching.

[0128] Optionally, the above describes how to adaptively switch the voice signal used to perform the voice service while the voice service is in progress. At the start of the voice service, the initial voice signal used to perform the voice service can be selected using the following method. For example, a default initial voice signal can be set, and at the start of the voice service, the default voice signal is used to perform the voice service. The default initial voice signal can be the second voice signal or the third voice signal. For another example, the voice signal to be used to perform the voice service can be determined based on the noise energy of the ambient noise signal at the start of the service. Specifically, when the initial noise energy of the ambient noise signal is greater than a third energy threshold, the third voice signal is selected to perform the voice service. When the initial noise energy is less than or equal to the third energy threshold, the second voice signal is selected to perform the voice service. The third energy threshold can be greater than or equal to the second energy threshold and less than or equal to the first energy threshold. Of course, the third energy threshold can also be greater than the first energy threshold or less than the second energy threshold, and this is not limited here.

[0129] Optionally, when the second-channel voice signal is used to perform a voice task, the first-channel voice signal can be used to perform noise reduction processing on the second-channel voice signal. The noise reduction processing of the first-channel voice signal on the second-channel voice signal is shown in the following formula. This processing is performed in the frequency domain, that is, y (the first-channel voice signal) and x (the second-channel voice signal) are first transformed by fast Fourier transform (FFT), and then transformed to the frequency domain before processing. It can be understood that when processing the first-channel voice signal and the second-channel voice signal, each frame of data of the first-channel voice signal and the second-channel voice signal is processed separately after framing and windowing.

[0130] Where s(w) is the denoised speech signal after denoising, which is a frequency domain signal. x(w) is the frequency domain signal corresponding to the second-channel speech signal, and y(w) is the frequency domain signal corresponding to the first-channel speech signal. Represents the frequency domain signal related to x(w) and y(w). This formula means that the frequency domain signal related to x(w) in y(w) (i.e., the frequency domain signal of the noise) is subtracted from x(w) to obtain the denoised speech signal after noise reduction in the frequency domain.

[0131] That is, the part of the second-channel voice signal that has a strong correlation with the first-channel voice signal is filtered out. Since the first-channel voice signal is mainly a noise signal, and the first-channel voice signal and the second-channel voice signal are voice signals collected in the same environment, the part of the second-channel voice signal that has a strong correlation with the first-channel voice signal can be considered as noise. After filtering out the noise, the remaining signal s(w) is mainly a human voice signal with a higher signal-to-noise ratio, which is used as the noise-reduced voice signal for subsequent business processing.

[0132] In a call service scenario, subsequent processing includes acoustic processing such as echo cancellation and reverberation removal. After undergoing these acoustic processing steps, the voice signal is encoded and transmitted to the peer end via the wireless network. In a recording service scenario, subsequent processing includes, for example, storing the noise-reduced voice signal. In a voice interaction scenario, subsequent processing includes, for example, performing speech recognition on the noise-reduced voice signal and executing related tasks.

[0133] In order to make this solution easier to understand, for example, as shown in Figure 7, Figure 7 is a simulation diagram of voice switching based on the solution provided by this application. Figure 7 takes the second microphone as an air conduction microphone and the second microphone as a bone conduction microphone as an example. The first capability threshold is E2, the second energy threshold is E1, and the voice service is performed using the voice collected by the air conduction microphone at the start of the voice service. The timing time T1 is the first duration threshold, the protection time T2 is the second duration threshold, the duration T3 is the third duration threshold, and the protection time T4 is the fourth duration threshold. It can be seen that after the noise energy (ambient sound energy) of the ambient noise signal reaches the first duration of the threshold E2 (the number of consecutive frames in which the ambient sound energy in the time domain is greater than the threshold E2 satisfies the count A1) and is greater than the duration T1, the voice signal collected by the bone conduction microphone is switched to perform the voice service. After switching to the bone conduction microphone, if the second duration (satisfying count A2) of voice service using the bone conduction microphone is greater than the protection time T2, the system can switch to the air conduction microphone. However, at this time, the ambient sound energy condition is not met, and the voice service is still performed using the bone conduction microphone. After a frame with ambient sound energy less than threshold E1 is found, the system starts counting the number of consecutive frames with ambient sound energy less than threshold E1. When the third duration is reached (count A3 is met), the system switches to using the voice signal collected by the air conduction microphone for voice service. After switching to the air conduction microphone, if the fourth duration (satisfying count A4) of voice service using the air conduction microphone is greater than the protection time T4, the system can switch to the bone conduction microphone. However, at this time, the ambient sound energy condition is not met, and the voice service is still performed using the voice signal collected by the air conduction microphone.

[0134] Based on the same inventive concept, as shown in Figure 8, the present application also provides a speech processing device. The speech processing device 800 includes a processing module 801. The speech processing device 800 can be applied to wearable devices, such as headphones, smart glasses, AR head-mounted display devices, VR head-mounted display devices, or neck massagers. The speech processing device 800 can be a wearable device, or a functional module in a wearable device, or a hardware module (such as a chip, etc.). The speech processing device 800 can be applied to smart devices, such as mobile phones, tablet computers, computers, and car computers. The speech processing device 800 can be a smart device, or a functional module in a smart device, or a hardware module (such as a chip, etc.).

[0135] Processing module 801 is configured to obtain a first voice signal collected by a first microphone, the first voice signal including an ambient noise signal indicating the ambient noise of the wearable device's environment. The first microphone is configured to collect ambient noise, and the ambient noise in the first voice signal can accurately represent the ambient noise of the wearable device's environment. While performing a voice service using a second voice signal collected by a second microphone of the wearable device, the processing module is configured to obtain a first duration based on the first voice signal, the first duration being the duration during which the noise energy of the current ambient noise signal has been continuously greater than a first energy threshold. The processing module is configured to determine, based on the first duration, whether to switch to performing the voice service using a third voice signal, the third voice signal collected by a third microphone of the wearable device. When the noise energy of the ambient noise exceeds the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

[0136] In one possible implementation, processing module 801 is configured to determine, when the first duration is greater than a first duration threshold, to switch to using the third voice signal to perform the voice service. Processing module 801 is configured to determine, when the first duration is less than or equal to the first duration threshold, to continue using the second voice signal to perform the voice service.

[0137] In a possible implementation, the processing module 801 is configured to determine whether to switch to using a third voice signal to perform voice services based on a first duration and a second duration, where the second duration is the duration for which the second voice signal has been used to perform voice services.

[0138] In one possible implementation, processing module 801 is configured to determine to switch to using the third voice signal to perform the voice service when the first duration is greater than a first duration threshold and the second duration is greater than a second duration threshold. Processing module 801 is configured to determine to continue using the second voice signal to perform the voice service when the first duration is less than or equal to the first duration threshold and / or the second duration is less than or equal to the second duration threshold. Optionally, the second duration threshold is greater than the first duration threshold.

[0139] In one possible implementation, processing module 801 is configured to, while performing voice services using a third voice signal, obtain a third duration based on the first voice signal, where the third duration is the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to a second energy threshold, where the second energy threshold is less than or equal to the first energy threshold. Processing module 801 is configured to determine, based on the third duration, whether to switch to performing voice services using the second voice signal, where the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal when the noise energy of the ambient noise is less than or equal to the second energy threshold.

[0140] In one possible implementation, processing module 801 is configured to determine, when the third duration is greater than a third duration threshold, to switch to using the second voice signal to perform the voice service. Processing module 801 is configured to determine, when the third duration is less than or equal to the third duration threshold, to continue using the third voice signal to perform the voice service.

[0141] In one possible implementation, the processing module 801 is configured to determine whether to switch to using the third voice signal to perform the voice service based on the third duration and the fourth duration, where the fourth duration is the duration for which the third voice signal has been continuously used to perform the voice service.

[0142] In one possible implementation, processing module 801 is configured to determine to switch to using the second voice signal to perform the voice service when the third duration is greater than a third duration threshold and the fourth duration is greater than a fourth duration threshold. Processing module 801 is configured to determine to continue using the third voice signal to perform the voice service when the third duration is less than or equal to the third duration threshold and / or the fourth duration is less than or equal to the fourth duration threshold.

[0143] In one possible implementation, processing module 801 is configured to obtain a forced switching instruction, where the forced switching instruction instructs to use a target voice signal to perform a voice service, where the target voice signal is one of the second voice signal and the third voice signal. Processing module 801 is configured to switch to using the target voice signal to perform the voice service when the voice signal currently used to perform the voice service differs from the target voice signal.

[0144] In one possible implementation, the processing module 801 is used to obtain a fifth duration based on the first voice signal when the target voice signal is the second voice signal, and determine whether to switch to using the third voice signal to perform the voice service based on the fifth duration, and the fifth duration is the duration during which the noise energy of the current environmental noise signal has been continuously greater than the first energy threshold. The processing module 801 is used to obtain a sixth duration based on the first voice signal when the target voice signal is the third voice signal, and determine whether to switch to using the second voice signal to perform the voice service based on the sixth duration, and the sixth duration is the duration during which the noise energy of the current environmental noise signal has been continuously less than or equal to the second energy threshold. The combination of forced switching and adaptive switching can both meet user needs and ensure the stability of voice services.

[0145] In one possible implementation, processing module 801 is configured to obtain the initial noise energy of the ambient noise based on the first voice signal at the start of a voice service. Processing module 801 is configured to determine whether to use the third voice signal for the voice service when the initial noise energy is greater than a fifth energy threshold. Processing module 801 is configured to determine whether to use the second voice signal for the voice service when the initial noise energy is less than or equal to the fifth energy threshold. Selecting the voice signal for the voice service based on the initial noise energy at the start of a voice service ensures a high signal-to-noise ratio (SNR) at the start of the voice service, thereby guaranteeing the quality of the voice service.

[0146] In one possible implementation, processing module 801 is configured to extract non-speech segments from the first voice signal to obtain an ambient noise signal. Processing module 801 is configured to obtain noise energy from the ambient noise signal. The first voice signal may also contain a user voice signal. Extracting the non-speech segments from the first voice signal provides a relatively pure ambient noise signal. Thus, the ambient noise signal accurately represents the current ambient noise state, making voice switching decisions based on the ambient noise state more timely and accurate.

[0147] In a possible implementation, the first microphone is provided as a microphone on the wearable device; or the first microphone is provided as a microphone on a smart device connected to the wearable device.

[0148] In one possible implementation, processing module 801 is configured to perform noise reduction processing on a second voice signal using the first voice signal to obtain a noise-reduced voice signal. Processing module 801 is configured to perform voice services using the noise-reduced voice signal. The first voice signal can more accurately represent the current environmental noise state, and the first voice signal and the second voice signal are collected in the same environment. Therefore, using the first voice signal to perform noise reduction processing on the second voice signal can more accurately filter out the environmental noise in the second voice signal, thereby obtaining a noise-reduced voice signal with a higher signal-to-noise ratio, thereby improving user voice clarity and ensuring voice service quality.

[0149] In one possible implementation, the second microphone is an air conduction microphone, and the third microphone is a bone conduction microphone; or the second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

[0150] Optionally, the speech processing device 800 may further include a transceiver module 802. When the speech processing device 800 is applied to a wearable device, the transceiver module 802 is configured to send the second or third speech signal to the smart device. Optionally, the transceiver module 802 is also configured to send the first speech signal to the smart device.

[0151] When the voice processing apparatus 800 is applied to a smart device, the transceiver module 802 is used to receive the second voice signal or the third voice signal sent by the wearable device. Optionally, the transceiver module 802 is also used to receive the first voice signal sent by the wearable device.

[0152] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the signal processing method flow of any of the above method embodiments.

[0153] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical or other forms.

[0155] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A speech signal processing method, characterized in that: The method comprises: Acquire a first voice signal collected by a first microphone, wherein the first voice signal includes an environmental noise signal, and the environmental noise signal indicates environmental noise of an environment where the wearable device is located; During the execution of the voice service using the second voice signal collected by the second microphone of the wearable device, a first duration is obtained according to the first voice signal, where the first duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than a first energy threshold; Determine whether to switch to using a third voice signal to perform the voice service according to the first duration, where the third voice signal is collected by a third microphone of the wearable device, and when the noise energy of the ambient noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

2. The method according to claim 1, characterized in that The determining, according to the first duration, whether to switch to using a third voice signal to perform the voice service includes: When the first duration is greater than the first duration threshold, determining to switch to using the third voice signal to perform the voice service; When the first duration is less than or equal to the first duration threshold, it is determined that the second voice signal is still used to perform the voice service.

3. The method according to claim 1, characterized in that The determining, according to the first duration, whether to switch to using a third voice signal to perform the voice service includes: Whether to switch to using the third voice signal to perform the voice service is determined based on the first duration and the second duration, where the second duration is the duration for which the second voice signal has been continuously used to perform the voice service.

4. The method according to claim 3, characterized in that The determining, according to the first duration and the second duration, whether to switch to using the third voice signal to perform the voice service comprises: When the first duration is greater than the first duration threshold, and the second duration is greater than the second duration threshold, determining to switch to using the third voice signal to perform the voice service; When the first duration is less than or equal to the first duration threshold, and / or the second duration is less than or equal to the second duration threshold, it is determined that the second voice signal is still used to perform the voice service.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: During the period of performing the voice service using the third voice signal, obtaining a third duration according to the first voice signal, the third duration being the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to a second energy threshold, and the second energy threshold is less than or equal to the first energy threshold; Determine whether to switch to using the second voice signal to perform the voice service based on the third duration, and when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

6. The method according to claim 5, characterized in that The determining, according to the third duration, whether to switch to using the second voice signal to perform the voice service includes: When the third duration is greater than a third duration threshold, determining to switch to using the second voice signal to perform the voice service; When the third duration is less than or equal to the third duration threshold, it is determined that the voice service is still performed using the third voice signal.

7. The method according to claim 5, characterized in that The determining, according to the third duration, whether to switch to using the second voice signal to perform the voice service includes: Whether to switch to using the third voice signal to perform the voice service is determined according to the third duration and the fourth duration, and the fourth duration is the duration for which the third voice signal has been continuously used to perform the voice service.

8. The method according to claim 7, characterized in that The determining, according to the third duration and the fourth duration, whether to switch to using the third voice signal to perform the voice service comprises: When the third duration is greater than the third duration threshold, and the fourth duration is greater than the fourth duration threshold, determining to switch to using the second voice signal to perform the voice service; When the third duration is less than or equal to the third duration threshold, and / or the fourth duration is less than or equal to the fourth duration threshold, it is determined that the third voice signal is still used to perform the voice service.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Obtaining a forced switching instruction, wherein the forced switching instruction indicates that the voice service is performed using a target voice signal, wherein the target voice signal is one of the second voice signal and the third voice signal; When the voice signal currently used to perform the voice service is different from the target voice signal, the target voice signal is switched to be used to perform the voice service.

10. The method according to claim 9, characterized in that After the switching is to use the target voice signal to perform the voice service, the method further includes: When the target voice signal is the second voice signal, a fifth duration is obtained according to the first voice signal, and according to the fifth duration, it is determined whether to switch to using the third voice signal to perform the voice service, wherein the fifth duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than the first energy threshold; When the target voice signal is the third voice signal, a sixth duration is obtained according to the first voice signal, and according to the sixth duration, it is determined whether to switch to using the second voice signal to perform the voice service, wherein the sixth duration is the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to the second energy threshold.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: When the voice service starts, obtaining the initial noise energy of the environmental noise according to the first voice signal; When the initial noise energy is greater than a fifth energy threshold, determining to use the third voice signal to perform the voice service; When the initial noise energy is less than or equal to the fifth energy threshold, it is determined to use the second voice signal to perform the voice service.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Extracting a non-speech segment from the first speech signal to obtain the environmental noise signal; The noise energy of the environmental noise signal is obtained.

13. The method according to any one of claims 1 to 12, characterized in that The first sound pickup is a sound pickup arranged on the wearable device; or The first microphone is a microphone arranged on a smart device connected to the wearable device.

14. The method according to any one of claims 1 to 13, characterized in that The performing of the voice service using the second voice signal collected by the second microphone of the wearable device includes: Using the first voice signal to perform noise reduction processing on the second voice signal to obtain a noise-reduced voice signal; The voice service is performed using the noise reduced voice signal.

15. The method according to any one of claims 1 to 14, characterized in that The second sound pickup is an air conduction microphone, and the third sound pickup is a bone conduction microphone; or The second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

16. A speech signal processing device, characterized in that: The device comprises a processing module, and the processing module is used to execute the speech signal processing method according to any one of claims 1 to 15.

17. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor is coupled to the memory, and the processor is configured to execute the speech signal processing method according to any one of claims 1 to 15 based on instructions stored in the memory.

18. The electronic device according to claim 17, characterized in that: The electronic device is the wearable device, and the electronic device further includes the second microphone and the third microphone, the second microphone is used to collect the second voice signal, and the third microphone is used to collect the third voice signal.

19. The electronic device according to claim 17 or 18, characterized in that: The electronic device further includes the first microphone, which is used to collect the first voice signal.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when the computer-readable storage medium is executed on a computer, causes the computer to execute the speech signal processing method according to any one of claims 1 to 15.

Images