Voice signal processing method and related equipment

Abstract

The invention discloses a voice signal processing method and related equipment, so as to ensure that wearable equipment can realize a good pickup signal-to-noise ratio in different noise scenes. The method comprises the steps that a first voice signal collected by a first pickup is acquired, the first voice signal comprises an environment noise signal, and the environment noise signal indicates the environment noise of the environment where the wearable device is located; when a second voice signal collected by a second sound pickup of the wearable device is adopted to execute the voice service, according to the first voice signal, a first time length is obtained, and the first time length is a time length in which the noise energy of the current environmental noise signal is continuously greater than a first energy threshold value; and according to the first duration, determining whether to switch to a third path of voice signal collected by a third pickup of the wearable device to execute the voice service. Wherein when the noise energy of the environmental noise is greater than the first energy threshold value, the signal-to-noise ratio of the third path of voice signal is greater than the signal-to-noise ratio of the second path of voice signal.

Description

Technical Field

[0001] The present application relates to the field of acoustic technology, and in particular to a speech signal processing method and related equipment. Background Art

[0002] When making a call, the microphone on the electronic device is generally required to collect clear voice signals to ensure the call quality. However, the real environment is often more complicated, especially in scenes with relatively loud environmental noise such as roadsides, workshops, and factory and mining areas. The environmental noise is extremely high, which poses a high challenge to the clear call of the headset.

[0003] Bone conduction microphones can effectively pick up voice signals in high-noise environments, and can ensure call quality when used in high-noise environments. The working principle of bone conduction microphones is that when people speak, their skin and jaws vibrate, and the bone conduction microphone picks up the vibration signals of the skin and jaws and converts them into voice signals.

[0004] However, the voice signals collected by bone conduction microphones are severely distorted, and the clarity of human voices is poor, which limits the application scenarios. Therefore, there is an urgent need for a technical solution that can take into account various noise scenarios and ensure clear sound pickup and clear calls in various complex noise scenarios. Summary of the invention

[0005] The present application provides a speech signal processing method and related equipment to solve the problem in the related art that it is difficult for headphones to obtain stable speech signals with a high signal-to-noise ratio in complex noise scenarios.

[0006] The first aspect provides a method for processing a speech signal. The method includes: obtaining a first voice signal collected by a first microphone, wherein the first voice signal includes an environmental noise signal, and the environmental noise signal indicates the environmental noise of the environment in which the wearable device is located. Among them, the first microphone is a microphone for collecting environmental noise, and the environmental noise in the first voice signal can accurately characterize the environmental noise of the environment in which the wearable device is located. During the period when the second voice signal collected by the second microphone of the wearable device is used to perform a voice service, a first duration is obtained according to the first voice signal, and the first duration is the duration during which the noise energy of the current environmental noise signal has been continuously greater than the first energy threshold. According to the first duration, it is determined whether to switch to using a third voice signal to perform a voice service, the third voice signal is collected by the third microphone of the wearable device, and when the noise energy of the environmental noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

[0007] By setting at least two different microphones on the wearable device, the third microphone has a higher signal-to-noise ratio in a high-noise environment. According to the first voice signal, the first duration of the environmental noise energy in the current environment being continuously greater than the first energy threshold is obtained, so that it is possible to judge whether the current environment is a high-noise environment or a low-noise environment according to the first duration, and whether the environmental noise state is stable in a high-noise state, and then the voice signal used to perform the voice service can be adaptively switched according to the first duration, ensuring that the voice signal in a complex environment has a high signal-to-noise ratio, the voice is clear and reliable, and the stability of the voice service is guaranteed.

[0008] In one possible implementation, determining whether to switch to using a third voice signal to perform voice services is based on the first duration, including: when the first duration is greater than a first duration threshold, determining to switch to using a third voice signal to perform voice services; when the first duration is less than or equal to the first duration threshold, determining to still use the second voice signal to perform voice services.

[0009] If the first duration is greater than the first duration threshold, it means that the environment is currently in a relatively stable high-noise state, and the voice signal used to perform the voice service can be switched to a third voice signal with a higher signal-to-noise ratio in the environment, thereby improving voice clarity and ensuring the stability of the voice service. If the first duration is less than or equal to the first duration threshold, it means that the current environmental noise is still in an unstable state. In order to avoid frequent switching of voice signals, it is possible not to switch temporarily and keep the currently used voice signal.

[0010] In one possible implementation, determining whether to switch to using a third voice signal to perform voice services is based on a first duration, including: determining whether to switch to using a third voice signal to perform voice services is based on a first duration and a second duration, the second duration being the duration for which the second voice signal has been continuously used to perform voice services.

[0011] Including the duration of the current continuous use of the second voice signal to perform the voice service into the switching condition can avoid frequent switching of voice signals and improve the stability of the voice service.

[0012] In a possible implementation, determining whether to switch to using a third voice signal to perform voice services according to the first duration and the second duration includes: when the first duration is greater than a first duration threshold, and the second duration is greater than a second duration threshold, determining to switch to using the third voice signal to perform voice services; when the first duration is less than or equal to the first duration threshold, and / or the second duration is less than or equal to the second duration threshold, determining to still use the second voice signal to perform voice services. Optionally, the second duration threshold is greater than the first duration threshold.

[0013] Switching the voice signal when the first duration is greater than the first duration threshold and the second duration is greater than the second duration threshold is satisfied at the same time; switching the voice signal after the environmental noise is stable and the voice service is performed for a period of time using the second duration threshold can avoid frequent switching of the voice signal and ensure the stability of the voice service; and switching to a voice signal with a higher signal-to-noise ratio can improve the quality of the voice service.

[0014] In a possible implementation, while a third voice signal is used to perform voice services, a third duration is obtained based on the first voice signal, where the third duration is the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to a second energy threshold, and the second energy threshold is less than or equal to the first energy threshold; based on the third duration, it is determined whether to switch to using the second voice signal to perform voice services, and when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

[0015] In one possible implementation, determining whether to switch to using the second voice signal to perform voice services is based on the third duration, including: when the third duration is greater than a third duration threshold, determining to switch to using the second voice signal to perform voice services; when the third duration is less than or equal to the third duration threshold, determining to still use the third voice signal to perform voice services.

[0016] In one possible implementation, determining whether to switch to using the second voice signal to perform voice services is based on the third duration, including: determining whether to switch to using the third voice signal to perform voice services is based on the third duration and a fourth duration, the fourth duration being the duration for which the third voice signal has been continuously used to perform voice services.

[0017] In one possible implementation, determining whether to switch to using a third voice signal to perform voice services is based on the third duration and the fourth duration, including: when the third duration is greater than the third duration threshold, and the fourth duration is greater than the fourth duration threshold, determining to switch to using the second voice signal to perform voice services; when the third duration is less than or equal to the third duration threshold, and / or the fourth duration is less than or equal to the fourth duration threshold, determining to still use the third voice signal to perform voice services.

[0018] In a possible implementation, the method also includes: obtaining a forced switching instruction, the forced switching instruction indicating the use of a target voice signal to perform voice services, the target voice signal being one of a second voice signal and a third voice signal; when the voice signal currently used to perform the voice service is different from the target voice signal, switching to use the target voice signal to perform the voice service.

[0019] Allows users to forcibly switch voice signals used for voice services when needed, making the switching more flexible and more adaptable to user needs.

[0020] In a possible implementation, after switching to use the target voice signal to perform voice services, it also includes: when the target voice signal is the second voice signal, the fifth duration is obtained according to the first voice signal, and it is determined whether to switch to use the third voice signal to perform voice services according to the fifth duration, and the fifth duration is the duration that the noise energy of the current environmental noise signal has been continuously greater than the first energy threshold; when the target voice signal is the third voice signal, the sixth duration is obtained according to the first voice signal, and it is determined whether to switch to use the second voice signal to perform voice services according to the sixth duration, and the sixth duration is the duration that the noise energy of the current environmental noise signal has been continuously less than or equal to the second energy threshold. The combination of forced switching and adaptive switching can not only meet user needs but also ensure the stability of voice services.

[0021] In a possible implementation, the method further includes: when the voice service starts, obtaining the initial noise energy of the environmental noise according to the first voice signal; when the initial noise energy is greater than the fifth energy threshold, determining to use the third voice signal to perform the voice service; when the initial noise energy is less than or equal to the fifth energy threshold, determining to use the second voice signal to perform the voice service. When the voice service starts, the voice signal used to perform the voice service is selected according to the initial noise energy, which can ensure that the voice signal has a high signal-to-noise ratio at the beginning of the voice service, thereby ensuring the quality of the voice service.

[0022] In a possible implementation, the method further includes: extracting a non-voice segment from the first voice signal to obtain an environmental noise signal; and obtaining noise energy of the environmental noise signal. The first voice signal may also contain a user voice signal, and the non-voice segment extracted from the first voice signal is a relatively pure environmental noise signal, so that the environmental noise signal can accurately represent the current environmental noise state, and therefore the voice switching decision based on the environmental noise state is more timely and accurate.

[0023] In a possible implementation, the first microphone is a microphone disposed on the wearable device; or the first microphone is a microphone disposed on a smart device connected to the wearable device.

[0024] In a possible implementation, performing voice services using a second voice signal collected by a second microphone of a wearable device includes: using a first voice signal to perform noise reduction processing on the second voice signal to obtain a noise-reduced voice signal; and performing voice services using the noise-reduced voice signal. The first voice signal can more accurately characterize the current environmental noise state, and the first voice signal and the second voice signal are collected in the same environment. Therefore, using the first voice signal to perform noise reduction processing on the second voice signal can more accurately filter the environmental noise in the second voice signal, thereby obtaining a noise-reduced voice signal with a higher signal-to-noise ratio, which can improve the clarity of user voices and ensure the quality of voice services.

[0025] In a possible implementation, the second microphone is an air conduction microphone, and the third microphone is a bone conduction microphone; or the second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

[0026] A second aspect provides a speech signal processing device, which includes a processing module, and the processing module is used to execute the speech signal processing method in the first aspect or any possible implementation of the first aspect.

[0027] Specifically, the processing module is used to obtain a first voice signal collected by a first microphone, wherein the first voice signal includes an environmental noise signal, and the environmental noise signal indicates the environmental noise of the environment where the wearable device is located. Among them, the first microphone is a microphone for collecting environmental noise, and the environmental noise in the first voice signal can accurately characterize the environmental noise of the environment where the wearable device is located. During the period when the second voice signal collected by the second microphone of the wearable device is used to perform a voice service, the processing module is used to obtain a first duration according to the first voice signal, and the first duration is the duration that the noise energy of the current environmental noise signal has been continuously greater than the first energy threshold. The processing module is used to determine whether to switch to using a third voice signal to perform a voice service according to the first duration, and the third voice signal is collected by the third microphone of the wearable device. When the noise energy of the environmental noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

[0028] In one possible implementation, the processing module is used to: when the first duration is greater than a first duration threshold, determine to switch to using a third voice signal to perform voice services; when the first duration is less than or equal to the first duration threshold, determine to still use the second voice signal to perform voice services.

[0029] In a possible implementation, the processing module is used to determine whether to switch to using a third voice signal to perform voice services according to a first duration and a second duration, where the second duration is the duration for which the second voice signal has been continuously used to perform voice services.

[0030] In a possible implementation, the processing module is used to: when the first duration is greater than the first duration threshold and the second duration is greater than the second duration threshold, determine to switch to using the third voice signal to perform the voice service; when the first duration is less than or equal to the first duration threshold and / or the second duration is less than or equal to the second duration threshold, determine to still use the second voice signal to perform the voice service. Optionally, the second duration threshold is greater than the first duration threshold.

[0031] In one possible implementation, the processing module is used to: during the period of using the third voice signal to perform voice services, obtain a third duration based on the first voice signal, the third duration being the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to the second energy threshold, and the second energy threshold is less than or equal to the first energy threshold; determine whether to switch to using the second voice signal to perform voice services based on the third duration, and when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

[0032] In one possible implementation, the processing module is used to: when the third duration is greater than a third duration threshold, determine to switch to using the second voice signal to perform voice services; when the third duration is less than or equal to the third duration threshold, determine to still use the third voice signal to perform voice services.

[0033] In a possible implementation, the processing module is used to determine whether to switch to using the third voice signal to perform the voice service according to the third duration and the fourth duration, where the fourth duration is the duration for which the third voice signal has been continuously used to perform the voice service.

[0034] In one possible implementation, the processing module is used to: when the third duration is greater than a third duration threshold, and the fourth duration is greater than a fourth duration threshold, determine to switch to using the second voice signal to perform voice services; when the third duration is less than or equal to the third duration threshold, and / or the fourth duration is less than or equal to the fourth duration threshold, determine to still use the third voice signal to perform voice services.

[0035] In one possible implementation, the processing module is used to: obtain a forced switching instruction, the forced switching instruction indicates that a target voice signal is used to perform a voice service, the target voice signal is one of a second voice signal and a third voice signal; when the voice signal currently used to perform the voice service is different from the target voice signal, switch to using the target voice signal to perform the voice service.

[0036] In a possible implementation, the processing module is used to: when the target voice signal is the second-channel voice signal, obtain the fifth duration according to the first-channel voice signal, and determine whether to switch to using the third-channel voice signal to perform voice services according to the fifth duration, and the fifth duration is the duration that the noise energy of the current environmental noise signal has been continuously greater than the first energy threshold; when the target voice signal is the third-channel voice signal, obtain the sixth duration according to the first-channel voice signal, and determine whether to switch to using the second-channel voice signal to perform voice services according to the sixth duration, and the sixth duration is the duration that the noise energy of the current environmental noise signal has been continuously less than or equal to the second energy threshold. The combination of forced switching and adaptive switching can not only meet user needs but also ensure the stability of voice services.

[0037] In a possible implementation, the processing module is used to: obtain the initial noise energy of the environmental noise according to the first voice signal at the beginning of the voice service; determine to use the third voice signal to perform the voice service when the initial noise energy is greater than the fifth energy threshold; determine to use the second voice signal to perform the voice service when the initial noise energy is less than or equal to the fifth energy threshold. At the beginning of the voice service, the voice signal used to perform the voice service is selected according to the initial noise energy, which can ensure that the voice signal has a high signal-to-noise ratio at the beginning of the voice service, thereby ensuring the quality of the voice service.

[0038] In a possible implementation, the processing module is used to: extract the non-voice segment in the first voice signal to obtain the environmental noise signal; and obtain the noise energy of the environmental noise signal. The first voice signal may also contain a user voice signal, and the non-voice segment extracted from the first voice signal is a relatively pure environmental noise signal, so that the environmental noise signal can accurately represent the current environmental noise state, and the voice switching decision based on the environmental noise state is more timely and accurate.

[0039] In a possible implementation, the first microphone is a microphone disposed on the wearable device; or the first microphone is a microphone disposed on a smart device connected to the wearable device.

[0040] In a possible implementation, the processing module is used to: use the first voice signal to perform noise reduction processing on the second voice signal to obtain a noise-reduced voice signal; and use the noise-reduced voice signal to perform voice services. The first voice signal can more accurately characterize the current environmental noise state, and the first voice signal and the second voice signal are collected in the same environment. Therefore, using the first voice signal to perform noise reduction processing on the second voice signal can more accurately filter the environmental noise in the second voice signal, thereby obtaining a noise-reduced voice signal with a higher signal-to-noise ratio, which can improve the clarity of user voice and ensure the quality of voice services.

[0041] In a possible implementation, the second microphone is an air conduction microphone, and the third microphone is a bone conduction microphone; or the second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

[0042] The third aspect provides an electronic device, characterized in that the electronic device includes a processor and a memory, the processor is coupled to the memory, and the processor is configured to execute the speech signal processing method in the first aspect or any one of the embodiments of the first aspect based on instructions stored in the memory.

[0043] In a possible implementation, the electronic device is a wearable device, and the electronic device further includes a second microphone and a third microphone, the second microphone is used to collect a second voice signal, and the third microphone is used to collect a third voice signal.

[0044] In a possible implementation, the electronic device further includes a first microphone, and the first microphone is used to collect a first voice signal.

[0045] The fourth aspect provides a computer-readable storage medium, characterized in that it includes instructions, which, when the computer-readable storage medium is run on a computer, enables the computer to execute the speech signal processing method in the first aspect or any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1a A schematic diagram of a real-time conversation between an air conduction microphone and a mobile phone provided in the present application;

[0047] Figure 1b A schematic diagram of a real-time call between a bone conduction microphone and a mobile phone provided in the present application;

[0048] Figure 2 A schematic diagram of the system architecture involved in a speech processing method provided in this application;

[0049] Figure 3 A schematic diagram of a pickup deployment method for a wearable device provided in the present application;

[0050] Figure 4 A schematic diagram of the structure of an electronic device provided in this application;

[0051] Figure 5a A schematic diagram of the logical structure of an electronic device provided in this application;

[0052] Figure 5b for Figure 5a Schematic diagram of the interaction between modules;

[0053] Figure 6 A flowchart of a speech processing method provided in this application;

[0054] Figure 7 A simulation diagram of voice switching based on the solution provided by this application;

[0055] Figure 8 A schematic diagram of the structure of a speech processing device provided in this application. DETAILED DESCRIPTION

[0056] The present application provides a voice signal processing method and related equipment to ensure the voice quality of voice services in complex scenarios.

[0057] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0058] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In the description of this application, unless otherwise specified, the meaning of "plurality" refers to two or more.

[0059] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0060] In the various embodiments of the present application, unless otherwise specified or provided in a logical conflict, the terms and / or descriptions between the different embodiments are consistent and may be referenced to each other, and the technical features in the different embodiments may be combined to form new embodiments according to their inherent logical relationships.

[0061] In voice scenarios such as calls, voice interactions, recordings, voice entertainment, and video conferences, users sometimes wear wearable devices such as headphones to collect user voice signals, so that wearable devices or electronic devices connected to wearable devices can implement related voice services based on the user voice signals. When users use wearable devices to participate in voice services, on the one hand, they can free their hands so that they can engage in other activities at the same time; on the other hand, wearable devices are closer to the user's sound source and can pick up voice signals with a higher signal-to-noise ratio, improving the quality of voice services; on the other hand, they can also reduce the risk of user privacy leakage or reduce the impact on others, bringing convenience to users' work, work, and study.

[0062] Taking the call scene as an example, the voice signal is transmitted in most cases by air conduction. The transmission process can be described as follows: Figure 1a As shown, the user on side A and the user on side B make a voice call based on a wearable device (here taking headphones as an example) and an electronic device connected to the wearable device (here taking a mobile phone as an example, it can also be a computer or tablet, etc.). Taking the speaking of side A and the listening of side B as an example, the above transmission process is: ① The user on side A speaks and sends a voice signal S; ② The air conduction microphone on the earphone on side A collects the voice signal S; ③ The earphone on side A sends the voice signal S to the mobile phone on side A; ④ The mobile phone on side A encodes and compresses the voice signal S and transmits it to the mobile phone on side B based on a wireless network; ⑤ The mobile phone on side B decodes the received signal and restores it to obtain the voice signal S; ⑥ The mobile phone on side B plays the voice signal S based on the speaker of the mobile phone on side B (or the wearable device connected to the mobile phone on side B).

[0063] When the above-mentioned air conduction voice signal is transmitted, the voice signal S will collect environmental noise in addition to the voice of the user on side A. If the environmental noise is loud, it will affect the listening effect of the user on side B. Therefore, in some high-noise scenarios, such as mining and rescue, bone conduction voice signals are generally used for information transmission. This is because the bone conduction microphone collects less environmental noise and has a higher audio signal-to-noise ratio, which can greatly ensure the call efficiency in these scenarios. The transmission process is as follows: Figure 1b As shown, the user on the A side and the user on the B side make a voice call based on the mobile phone. Taking the case where the A side speaks and the B side (assuming that the B side does not change) listens as an example, the above transmission process is: ① The user on the A side speaks and sends a voice signal S1; ② The bone conduction microphone on the head-mounted device on the A side (such as headphones) is close to the face of the user on the A side, and the audio signal S2 (also called bone conduction voice signal S2) when the user on the A side speaks is collected based on the vibration of the skin. This bone conduction voice signal S2 is sent to the mobile phone on the A side (such as sent through the Bluetooth connection between the head-mounted device and the mobile phone); ③ The mobile phone on the A side encodes and compresses the bone conduction voice signal S2 and transmits it to the mobile phone on the B side based on the wireless network; ④ The mobile phone on the B side decodes and restores the received signal to obtain the bone conduction voice signal S2; ⑤ The mobile phone on the B side plays the bone conduction voice signal S2 based on the air conduction speaker of the mobile phone on the B side.

[0064] The generation model of air-conduction speech signal and bone-conduction speech signal is the way of excitation source convolution excitation channel (i.e. excitation source * excitation channel = speech signal), but the specific generation principle of the two is different: the excitation source of air-conduction speech signal and bone-conduction speech signal is the vocal cord, but the excitation channel of air-conduction speech signal is pharynx & oral cavity, etc., while the excitation channel of bone-conduction speech signal is muscle & bone, etc. Because the excitation channels of the two are different, the final listening will be quite different: the air-conduction speech signal has less distortion and higher brightness because the excitation channel is air transmission, but the air-conduction microphone is sensitive to noise in the environment, so the air-conduction speech signal includes more environmental noise. The excitation channel of bone-conduction speech signal is solid and soft transmission, although the signal-to-noise ratio is higher, the distortion is large, the brightness is darker than that of air-conduction speech signal, and the listening comfort is poor; more seriously, it will cause the other end (such as side B) to hear the unnatural sound of side A, and in severe cases, it may not be able to distinguish who is speaking, thus affecting the call efficiency.

[0065] Therefore, there is an urgent need for a technical solution that can take into account various noise scenarios and ensure that the voice signal is clear and stable in various complex noise scenarios.

[0066] In order to solve the above technical problems, this application provides the following technical solutions.

[0067] like Figure 2 As shown, Figure 2 A schematic diagram of the system architecture involved in a speech processing method provided in this application. Figure 2 , the system may include a wearable device 201 and a smart device 202. The wearable device 201 and the smart device 202 are connected to communicate via a wired or wireless method. The wireless method may specifically be Bluetooth, wireless local area network (WLAN), ZigBee or near field communication (NFC), etc., or may also be other new generation wireless short-range communication technologies, such as NearLink, etc. In an embodiment, the wearable device 201 may be a headset, smart glasses, an augmented reality (AR) head display device, a virtual reality (VR) head display device, or a neck massager. The smart device 202 may be a mobile phone, a tablet computer, a computer, a car computer, and other devices.

[0068] In the embodiment of the present application, the wearable device 201 is used to collect voice signals and send the collected voice signals to the smart device 202. The smart device 202 is used to receive the voice signals sent by the wearable device 201 and perform corresponding voice tasks based on the received voice signals. In this embodiment, the voice service can be a call service, a voice interaction service, a recording service, a voice entertainment service (such as karaoke) and a video conferencing service, etc., which involve voice signal processing.

[0069] The system includes a first pickup (or pickup sensor, microphone, etc.) for collecting the ambient sound signal of the environment where the wearable device 201 is located. In a possible implementation, the first pickup can be a pickup set on the wearable device 201. In another possible implementation, the first pickup can also be a pickup set on the smart device 202. Since the wearable device 201 and the smart device 202 are close to each other and are in the same environment, the ambient sound signal collected by the pickup on the smart device 202 can be considered as the ambient sound signal of the environment where the wearable device 201 is located, and the smart device 202 has a certain distance from the user sound source, and the user voice signal in the collected ambient sound signal is less, which can more accurately represent the noise situation in the environment. Optionally, in a possible implementation, the system also includes other devices in the same environment and connected to the smart device, such as wearable devices (such as smart watches), smart smart devices (tablets, laptops) or IoT devices (smart home appliances, smart toys, smart robots or industrial IoT devices), etc., and the first pickup can be a pickup on other devices.

[0070] When the first sound pickup is a sound pickup on the wearable device 201, the first sound pickup may be a bone conduction microphone or an air conduction microphone. Optionally, the first sound pickup may be deployed toward the external environment, facing away from the human voice source, to collect more accurate environmental noise. When the first sound pickup is a sound pickup on the smart device 202, the first sound pickup may be an air conduction microphone. Of course, when the first sound pickup is a sound pickup on the smart device 202, the first sound pickup may also be a bone conduction microphone.

[0071] The wearable device 201 includes at least two microphones, for example, a second microphone and a third microphone. The second microphone and the third microphone are different microphones. In a low-noise environment, the signal-to-noise ratio of the voice signal collected by the second microphone is higher than that of the voice signal collected by the third microphone, that is, the human voice is clearer in the voice signal collected by the second microphone in a low-noise environment. In a high-noise environment, the signal-to-noise ratio of the voice signal collected by the third microphone is higher than that of the voice signal collected by the second microphone, that is, the human voice is clearer in the voice signal collected by the third microphone in a high-noise environment. In one possible implementation, the second microphone is an air conduction microphone, and the third microphone is a bone conduction microphone. In another possible implementation, the second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

[0072] The voice signal collected by the first microphone in real time is called the first voice signal, and the first voice signal includes the environmental noise signal. The voice signal collected by the second microphone in real time is called the second voice signal. The voice signal collected by the third microphone in real time is called the third voice signal. The second microphone and the third microphone are the main microphones, that is, the second voice signal and the third voice signal are used to perform voice services. In this embodiment, the first microphone is the auxiliary microphone, and the first voice signal collected by it is mainly used to analyze the environmental noise situation. The environmental noise situation can be used to decide which voice signal of the second voice signal and the third voice signal is used to perform voice services, so as to realize adaptive switching of voice signals. When performing voice services, at most one voice signal of the second voice signal and the third voice signal is used for processing at the same time. When the preset voice switching condition is met, the voice signal currently used to perform voice services can be switched from one voice signal to another voice signal.

[0073] When the first microphone is a microphone on the wearable device 201, in a possible implementation, the first microphone and the second microphone can be the same microphone, and the first voice signal and the second voice signal are the same voice signal, thereby reducing the number of microphones on the wearable device 201 and reducing the cost of the wearable device 201.

[0074] When the first microphone is a microphone on the wearable device 201, in another possible implementation, the first microphone, the second microphone and the third microphone on the wearable device 201 are three different microphones. By setting a first microphone dedicated to collecting ambient noise, the accuracy of the energy of the estimated ambient noise based on the first voice signal can be improved, thereby ensuring that the timing of voice switching can be grasped in a timely and accurate manner, thereby improving the efficiency of voice switching. Optionally, a microphone with a lower sensitivity than the second microphone can be selected as the first microphone to reduce the influence of the user's voice signal on the first voice signal, increase the proportion of ambient noise, and enable the first voice signal to more accurately characterize the ambient noise situation. Figure 3 As shown, Figure 3 A schematic diagram of a wearable device provided for this application. Figure 3 Taking the wearable device as headphones as an example, it can be understood that the shape of the headphones here is only for illustration, and the headphones can also be in other forms, such as in-ear or neck-hanging. The second microphone and the third microphone can be set at a position close to the user and close to the user's vocal part, so that the second microphone and the third microphone can collect the user's voice signal more clearly, and improve the signal-to-noise ratio of the second voice signal and the third voice signal. The position and orientation of the first microphone on the wearable device 201 can be reasonably designed. For example, the first microphone is set on the outside of the wearable device 21, and the first microphone is facing away from the user, so that when the user wears the wearable device, the first microphone is far away from the user's sound source, and can better collect environmental noise signals, while collecting as little of the user's own voice signal as possible. Figure 3 In the figure, only possible settings of the first microphone, the second microphone and the third microphone of the wearable device 201 are shown, which should not be understood as a limitation to the present application. The actual implementation may depend on the product form of the wearable device 201.

[0075] In a possible implementation, the wearable device 201 can determine in real time whether the voice switching condition is currently met based on the first voice signal, and when the voice switching condition is currently met, determine to switch to using the other voice signal of the second voice signal and the third voice signal to perform the voice service. In this case, after determining to switch the voice signal used to perform the voice service, the wearable device 201 can only send the switched voice signal for performing the voice service to the smart device 202, without sending the voice signal used to perform the voice service before the switch, and the smart device 202 uses the switched voice signal to perform the voice service. At this time, the second microphone and the third microphone can both be in the on state, that is, the second microphone and the third microphone are still collecting voice signals in real time, but the wearable device 202 only sends the switched voice signal to the smart device 202. Alternatively, after determining to switch the voice signal for performing the voice service, the wearable device 202 turns off the microphone corresponding to the voice signal for performing the voice service before the switch, and turns on the microphone corresponding to the voice signal for performing the voice service after the switch, thereby turning on the microphone currently required in the second microphone and the third microphone, and temporarily turning off the other microphone that is not currently required, which can reduce the power consumption of the wearable device 202. Of course, the wearable device 201 can also send the first voice signal and the second voice signal to the smart device 202 at the same time. After determining to switch the voice signal for performing the voice service, the wearable device 201 can send a switching instruction to the smart device 202, and the smart device 202 switches the voice signal for performing the voice service according to the switching instruction, and uses the switched voice signal to perform the voice service.

[0076] In another possible implementation, the smart device 202 can determine in real time whether the voice switching condition is currently met based on the first voice signal. When the first microphone is a microphone on the wearable device 201, the wearable device sends the first voice signal to the smart device 202 in real time. When the first microphone is a microphone on the smart device 202, the smart device 202 collects the first voice signal in real time through the first microphone. In the case where the smart device 202 determines whether the voice switching condition is currently met, the wearable device 201 can simultaneously send the first voice signal and the second voice signal to the smart device 202. When the smart device 202 determines that the voice switching condition is currently met, it is determined to switch to another voice signal in the second voice signal and the third voice signal to perform the voice service, and the switched voice signal is used to perform the voice service. Switching the voice signal used to perform the voice service at the back end (smart device 202) can reduce the switching delay and reduce the voice jamming and fragmentation introduced by the switching. Alternatively, when the smart device 202 determines whether the voice switching condition is currently met, when the smart device 202 determines that the voice switching condition is currently met, it sends a switching instruction to the wearable device 201, and the wearable device 201 only sends the switched voice signal to the smart device 202.

[0077] The specific implementation of determining in real time whether the voice switching condition is currently met based on the first voice signal will be described in the method embodiment below.

[0078] By setting at least two different microphones on the wearable device 201, when the ambient noise changes, the appropriate voice signal can be adaptively adjusted to execute the voice service, ensuring that the voice service is executed with a higher signal-to-noise ratio and clearer voice signal regardless of whether it is in a high-noise environment or a low-noise environment, or when the ambient noise changes dramatically, thereby improving the quality of the voice service.

[0079] It should be noted that the system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0080] Please refer to Figure 4 , Figure 4 The following is a schematic diagram of the structure of an electronic device provided in the present application. Optionally, the electronic device is Figure 2 The wearable device 201 or smart device 202 shown in FIG. The electronic device includes one or more processors 401 , a communication bus 402 , a memory 403 , and one or more communication interfaces 404 .

[0081] The processor 401 is a general-purpose central processing unit (CPU), a network processing (NP), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. Optionally, the PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0082] The communication bus 402 is used to transmit information between the above components. Optionally, the communication bus 402 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0083] Optionally, the memory 403 is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 403 exists independently and is connected to the processor 401 via the communication bus 402, or the memory 403 is integrated with the processor 401.

[0084] The communication interface 404 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 404 includes a wired communication interface, and optionally, also includes a wireless communication interface. Among them, the wired communication interface is, for example, an Ethernet interface. Optionally, the Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a Bluetooth interface, a ZigBee interface, an NFC interface, a StarFlash interface, a cellular network communication interface, or a combination thereof.

[0085] Optionally, in some embodiments, the electronic device includes multiple processors, such as Figure 4 401 and processor 405 shown in . Each of these processors is a single-core processor, or a multi-core processor. Optionally, the processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0086] When electronic equipment is Figure 2 When the wearable device 201 shown in the figure also includes a second microphone (not shown) and a third microphone (not shown). The second microphone is used to collect the second voice signal, and the third microphone is used to collect the third voice signal. The communication interface 404 is used to send the second voice signal and / or the third voice signal to the smart device. Optionally, the electronic device may further include a first microphone, and the first microphone is used to collect the first voice signal. Thus, the electronic device can judge whether the voice switching condition is currently met according to the first voice signal, so as to determine whether the voice signal used to perform the voice service is switched from the second voice signal to the third voice signal, or from the third voice signal to the second voice signal. Optionally, the communication interface 404 is also used to send the first voice signal to the smart device. Thus, the smart device can judge whether the voice switching condition is currently met according to the first voice signal, so as to determine whether the voice signal used to perform the voice service is switched from the second voice signal to the third voice signal, or from the third voice signal to the second voice signal.

[0087] When electronic equipment is Figure 2When the smart device 202 shown in the figure is used, optionally, the electronic device also includes an output device (not shown) and an input device (not shown). The output device communicates with the processor and can display information in a variety of ways. For example, the output device is a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device or a projector, etc. The input device communicates with the processor and can receive user input in a variety of ways. For example, the input device 407 includes one or more of a mouse, a keyboard, a touch screen device or a sensor device, etc. The communication interface 404 is used to receive a second voice signal and / or a third voice signal sent by the wearable device. Optionally, the communication interface 404 is used to receive a first voice signal sent by the wearable device. Optionally, the electronic device may also include a first microphone, which is used to collect the first voice signal. Thus, the smart device can determine whether the voice switching condition is currently met based on the first voice signal to determine whether the voice signal used to perform the voice service is switched from the second voice signal to the third voice signal, or from the third voice signal to the second voice signal.

[0088] In some embodiments, the memory 403 is used to store the program code 406 for executing the solution of the present application, and the processor 401 can execute the program code 406 stored in the memory 403. The program code 406 includes one or more software modules. The electronic device can implement the following by the processor 401 and the program code 406 in the memory 403. Figure 6 The embodiment provides a speech processing method.

[0089] like Figure 5a and Figure 5b As shown, Figure 5a A schematic diagram of the logical structure of an electronic device provided in this application; Figure 5b for Figure 5a In this embodiment, the electronic device includes a first voice signal acquisition module, a second voice signal acquisition module, a third voice signal acquisition module, an environmental noise energy calculation module, a switching judgment module, a switching control module and a voice service processing module.

[0090] The first voice signal acquisition module is used to acquire the first voice signal collected by the first microphone in real time. The second voice acquisition module is used to acquire the second voice signal collected by the second microphone in real time. The third voice acquisition module is used to acquire the third voice signal collected by the third microphone in real time.

[0091] The environmental noise energy calculation module is used to calculate the noise energy of the environmental noise signal in the first-channel voice signal in real time according to the first-channel voice signal.

[0092] During the current use of the second voice signal to perform voice services, the switching judgment module is used to compare the noise energy with the first noise threshold after each noise energy is obtained, and to count the first duration that the noise energy is continuously greater than the first noise threshold. In one possible implementation, the switching judgment module compares the first duration with the first duration threshold, and when the first duration is greater than the first duration threshold, determines to switch to using the third voice signal to perform voice services, and generates a switching instruction. In another possible implementation, the switching judgment module compares the first duration with the first duration threshold, and also compares the second duration with the second duration threshold, and when the first duration is greater than the first duration threshold, and the second duration is greater than the second duration threshold, determines to switch to using the third voice signal to perform voice services. Wherein, the second duration is the duration of the current continuous use of the second voice signal to perform the voice service, and generates a switching instruction.

[0093] During the current use of the third voice signal to perform voice services, the switching judgment module is used to compare the noise energy with the second noise threshold after each acquisition of noise energy, and to count the third duration of time when the noise energy is continuously greater than the second noise threshold. In one possible implementation, the switching judgment module compares the third duration with the third duration threshold, and when the third duration is greater than the third duration threshold, determines to switch to using the second voice signal to perform voice services, and generates a switching instruction. In another possible implementation, the switching judgment module compares the third duration with the third duration threshold, and also compares the fourth duration with the fourth duration threshold, and when the third duration is greater than the third duration threshold, and the fourth duration is greater than the fourth duration threshold, determines to switch to using the second voice signal to perform voice services. Wherein, the fourth duration is the duration of time that the third voice signal has been continuously used to perform the voice service, and generates a switching instruction.

[0094] The switching control module switches the voice signal used to perform the voice service from the second voice signal to the third voice signal according to the switching instruction, or switches the voice signal used to perform the voice service from the third voice signal to the second voice signal.

[0095] The voice service processing module is used to perform voice services according to the switched voice signals. The voice service processing module can perform noise reduction, echo reduction and encoding on the voice signals, and then perform corresponding voice services based on the processed voice signals.

[0096] like Figure 6 As shown, Figure 6 A flow chart of a speech processing method provided by the present application. Figure 4 The electronic device in the embodiment can be Figure 2 Wearable devices in Figure 2 The present embodiment comprises the following steps:

[0097] S601: Acquire a first voice signal collected by a first microphone, where the first voice signal includes an environmental noise signal, and the environmental noise signal indicates environmental noise in an environment where the wearable device is located.

[0098] The first microphone can be a microphone on a wearable device or a microphone on a smart device connected to the wearable device. The first microphone is a microphone used to collect ambient noise of the environment in which the wearable device is located. Therefore, the first voice signal collected by the first microphone includes an ambient noise signal. The first voice signal is a voice signal collected by the first microphone in real time, and the electronic device also performs noise analysis on the latest voice signal collected by the first microphone in real time.

[0099] In a possible implementation, the first voice signal may be used as an environmental noise signal.

[0100] In another possible implementation, a non-speech segment in the first voice signal can be extracted as an ambient sound signal. For example, voice activity detection (VAD) can be used to extract the non-speech segment in the first voice. Thus, the ambient sound signal can more accurately characterize the noise situation in the environment, ensuring the accuracy and efficiency of the voice switching decision.

[0101] S602: While performing a voice service using a second voice signal collected by a second microphone of the wearable device, a first duration is obtained according to the first voice signal, where the first duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than a first energy threshold.

[0102] During the period of using the second voice signal to perform the voice service, a first duration is obtained according to the first voice signal, wherein the first duration is the duration during which the noise energy of the current environmental noise signal has been continuously greater than the first energy threshold.

[0103] The electronic device calculates the noise energy of the latest environmental noise signal and determines whether the noise energy is greater than a first noise threshold. When the noise energy is greater than the first noise threshold, the electronic device records the first duration in the time domain during which the noise energy of the environmental noise signal is continuously greater than the first energy threshold up to the present time.

[0104] Specifically, the electronic device frames the ambient noise signal in the time domain, and then calculates the noise energy of each frame of the ambient noise signal. The noise energy may be the square value of the amplitude of the ambient noise signal of the frame. In another implementation, the noise energy may also refer to a parameter characterizing the intensity of the ambient noise, such as the loudness, power, sound intensity or sound pressure of the noise. The noise energy of each frame is compared with the first energy threshold. If the noise energy of the frame is greater than the first energy threshold, the value of the first counter is increased by 1. If the noise energy of the frame is less than or equal to the first energy threshold, the value of the first counter is cleared to 0. It should be noted that the first counter needs to count according to the timing of the frame. Exemplarily, a group of continuous frames in the time domain, (frame 1, frame 2, frame 3, frame 4), frame 1 is the earliest frame in the time domain, frame 4 is the latest frame in the time domain, the noise energy of frame 1 is greater than the first energy threshold, then the value of the first counter is increased by 1, the noise energy of frame 2 is greater than the first energy threshold, then the value of the first counter is increased by 1 again, the noise energy of frame 3 is less than the first energy threshold, then the value of the first counter is cleared to 0, at this time the first duration is 0, the noise energy of frame 4 is greater than the first energy threshold, then the value of the first counter is increased by 1, at this time the value of the first counter is 1.

[0105] Of course, the duration for which the noise energy of the ambient noise signal has been continuously greater than the first energy threshold may also refer to the cumulative duration for which the noise energy of the ambient noise signal in the time domain is greater than the first energy threshold within the sliding time window.

[0106] S603: Determine whether to switch to using a third voice signal to perform voice services based on the first duration, where the third voice signal is collected by a third microphone of the wearable device, and when the noise energy of the ambient noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

[0107] The first duration is a key condition for determining whether to switch the voice signal used to perform the voice service to the third voice signal. If the first duration is greater than the first duration threshold, it means that the current environmental noise is in a relatively stable high noise state, which can avoid frequent switching of voice signals in complex environmental noises that affects the coherence of the voice.

[0108] In a possible implementation, the first duration being greater than the first duration threshold can be used as the only condition for satisfying the voice switching condition at present. Specifically, the electronic device determines whether the first duration is greater than the first duration threshold, and when the first duration is greater than the first duration threshold, determines to switch to using the third voice signal to perform the voice service. When the first duration is less than or equal to the first duration threshold, it is determined to still use the second voice signal to perform the voice service. In a possible implementation, the current value m of the first counter (m is a natural number, indicating that the noise energy of m frames in the time domain is greater than the first energy threshold continuously, and since the frame length is fixed, the value m can represent the first duration) can be compared with the first preset value n (n is a positive integer, n represents the number threshold of frames continuously greater than the first energy threshold, and since the length of the frame is fixed, n can also represent the first duration threshold). When m is greater than n, it can be considered that the first duration is greater than the first duration threshold, and when m is less than or equal to n, it can be considered that the first duration is less than or equal to the first duration threshold.

[0109] In another possible implementation, the first duration greater than the first duration threshold, and the second duration greater than the second duration threshold can be used as a voice switching condition. The second duration is the duration for which the second voice signal has been continuously used to perform voice services. Optionally, the second duration threshold is greater than the first duration threshold. The second duration threshold can be regarded as a switching protection duration, that is, after switching the voice signal, the time length of the switched voice signal needs to be kept greater than the second duration threshold before the voice signal is allowed to be switched again, so as to avoid frequent switching of the voice signal and affecting the quality of the voice service. Specifically, the electronic device compares the size of the first duration and the first duration threshold, and compares the size of the second duration and the second duration threshold. When the first duration is greater than the first duration threshold, and the second duration is greater than the second duration threshold, it can be considered that the voice switching condition is currently met, and it is determined to switch to the third voice signal to perform the voice service. When the first duration is less than or equal to the first duration threshold, and / or the second duration is less than or equal to the second duration threshold, it can be considered that the voice switching condition is not currently met, and it is determined that the second voice signal is still used to perform the voice service.

[0110] The first duration threshold is used to ensure that the voice signal will not be switched because the noise energy of a certain frame is greater than the first energy threshold, thereby ensuring the stability and continuity of the voice service. The second duration threshold can ensure that when switching to the second voice signal to perform the voice service, the voice signal will not be immediately switched to the third voice signal in a short time, thereby ensuring the stability and continuity of the voice service.

[0111] In this embodiment, when the noise energy of the ambient noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal, that is, in a relatively noisy environment, the voice signal used to perform the voice service is switched to a voice signal with a higher signal-to-noise ratio in the current environment, ensuring that the voice signal is still clear in a high-noise environment, thereby ensuring the quality of the voice service.

[0112] S604: While performing voice services using a third voice signal, a third duration is obtained based on the first voice signal, where the third duration is the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to a second energy threshold, and the second energy threshold is less than or equal to the first energy threshold.

[0113] During the period of using the third voice signal to perform the voice service, a third duration is obtained according to the first voice signal, wherein the third duration is the duration during which the noise energy of the current environmental noise signal has been continuously less than or equal to the second energy threshold.

[0114] The electronic device calculates the noise energy of the latest ambient noise signal and determines whether the noise energy is less than or equal to a second energy threshold. When the noise energy is less than or equal to the second energy threshold, the electronic device records the ambient noise signal in the time domain up to the current third time duration in which its noise energy is continuously less than or equal to the second energy threshold.

[0115] Specifically, the electronic device frames the ambient noise signal in the time domain, and then calculates the noise energy of each frame of the ambient noise signal. The noise energy is the square of the amplitude of the ambient noise signal of the frame. The noise energy of each frame is compared with the second energy threshold. If the noise energy of the frame is greater than the second energy threshold, the value of the second counter is increased by 1. If the noise energy of the frame is greater than the second energy threshold, the value of the first counter is cleared to 0. It should be noted that the second counter needs to count according to the timing of the frame.

[0116] Of course, when the noise energy of the ambient noise signal has been continuously less than or equal to the second energy threshold, it may also refer to the cumulative duration in the sliding time window that the noise energy of the ambient noise signal in the time domain is less than or equal to the second energy threshold.

[0117] In this embodiment, the second energy threshold is less than or equal to the first energy threshold. The first duration threshold and the third duration threshold may be the same or different. The second duration threshold and the fourth duration threshold may be the same or different.

[0118] S605: Determine whether to switch to using the second voice signal to perform voice services according to the third duration, when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

[0119] The third duration is a key condition for determining whether to switch the voice signal used to perform voice services to the second voice signal. If the second duration is greater than the first duration threshold, it means that the current environmental noise is in a relatively stable low-noise state, which can avoid frequent switching of voice signals in complex environmental noises that affects the coherence of the voice.

[0120] In a possible implementation, the third duration being greater than the third duration threshold can be used as the only condition that currently satisfies the voice switching condition. Specifically, the electronic device determines whether the third duration is greater than the third duration threshold, and when the third duration is greater than the third duration threshold, determines to switch to using the second voice signal to perform the voice service. When the third duration is less than or equal to the third duration threshold, it is determined that the third voice signal is still used to perform the voice service. In a possible implementation, the current value p of the second counter (p is a natural number, indicating that the noise energy of p frames in the time domain is continuously greater than the first energy threshold, and since the frame length is fixed, the value p can represent the third duration) can be compared with the second preset value q (q is a positive integer, q represents the number threshold of frames continuously greater than the second energy threshold, and since the length of the frame is fixed, q can also represent the third duration threshold). When p is greater than q, it can be considered that the third duration is greater than the third duration threshold, and when p is less than or equal to q, it can be considered that the third duration is less than or equal to the third duration threshold.

[0121] In another possible implementation, the third duration greater than the third duration threshold, and the fourth duration greater than the fourth duration threshold can be used as a voice switching condition. The fourth duration is the duration for which the third voice signal has been used to perform voice services. Optionally, the fourth duration threshold is greater than the third duration threshold. The fourth duration threshold can be regarded as a switching protection duration, that is, after switching the voice signal, the time length of the switched voice signal needs to be kept greater than the fourth duration threshold before the voice signal is allowed to be switched again, so as to avoid frequent switching of the voice signal and affecting the quality of the voice service. Specifically, the electronic device compares the size of the third duration and the third duration threshold, and compares the size of the fourth duration and the fourth duration threshold. When the third duration is greater than the third duration threshold, and the fourth duration is greater than the fourth duration threshold, it can be considered that the voice switching condition is currently met, and it is determined to switch to the third voice signal to perform the voice service. When the third duration is less than or equal to the third duration threshold, and / or the fourth duration is less than or equal to the fourth duration threshold, it can be considered that the voice switching condition is not currently met, and it is determined that the third voice signal is still used to perform the voice service.

[0122] The third duration threshold is used to ensure that the voice signal will not be switched because the noise energy of a certain frame is less than or equal to the second energy threshold, thereby ensuring the stability and continuity of the voice service. The fourth duration threshold can ensure that when switching to the third voice signal to perform the voice service, the second voice signal will not be immediately switched in a short time, thereby ensuring the stability and continuity of the voice service.

[0123] In this embodiment, when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal, that is, in a low-noise environment, the voice signal used to perform the voice service is switched to a voice signal with a higher signal-to-noise ratio in the current environment, ensuring that the voice signal is still clear in a high-noise environment, thereby ensuring the quality of the voice service.

[0124] In this embodiment, at least two different microphones are provided on the wearable device. The second microphone has a higher signal-to-noise ratio in a low-noise environment, and the third microphone has a higher signal-to-noise ratio in a high-noise environment. The duration of time that the current environment is continuously in a certain state is obtained according to the environmental noise signal, and whether the current environment is a high-noise environment or a low-noise environment is determined according to the duration, and then the voice signal used to perform the voice service is adaptively switched, and two-way switching is supported to ensure that the voice signal in a complex environment has a high signal-to-noise ratio, the voice is clear and reliable, and the stability of the voice service is ensured. Furthermore, this embodiment also sets a switching protection time interval to avoid frequent switching of voice signals affecting the quality of voice services. In addition, this embodiment does not require a complex noise reduction algorithm, that is, it can ensure that voice services can be processed based on voice signals with a high signal-to-noise ratio in all scenarios, can reduce the computing power requirements for electronic equipment, and reduce the requirements for noise reduction algorithms, and can ensure the stability of voice services in a low-cost manner.

[0125] Optionally, the electronic device has an adaptive switching function (i.e. Figure 6 The method in (a) and the forced switching function. When the forced switching function is turned on and the adaptive switching function is turned off, the voice signal used to perform the voice service is switched according to the forced switching instruction. When the forced function is turned on and the adaptive switching function starts, the forced switching instruction is executed first. Specifically, when the user issues a forced switching instruction due to actual needs, the forced switching instruction indicates switching to the target voice signal in the first voice signal and the second voice signal, and the forced switching instruction is executed first. When the voice signal currently used to perform the voice service is inconsistent with the target voice signal, the voice service is performed using the target voice signal. After the forced switching is executed, when the switching conditions are met, the voice signal is switched according to the adaptive switching instruction. When the forced function is turned off and the adaptive switching function is turned on, the voice signal is switched only according to the adaptive switching instruction. When both the forced switching function and the adaptive switching function are turned off, the current voice signal is maintained to perform the voice service without any switching.

[0126] Optionally, the above describes how to adaptively switch the voice signal used to perform the voice service during the voice service. At the beginning of the voice service, the initial voice signal for performing the voice service can be selected by the following method. For example, a default initial voice signal can be set, and at the beginning of the voice service, the default voice signal is used to perform the voice service. The default initial voice signal can be a second-channel voice signal or a third-channel voice signal. For example, it can be determined which voice signal is used to perform the voice service based on the noise energy of the ambient noise signal at the beginning of the service. Specifically, when the initial noise energy of the ambient noise signal is greater than the third energy threshold, the third-channel voice signal is selected to perform the voice service, and when the initial noise energy is less than or equal to the third energy threshold, the second-channel voice signal is selected to perform the voice service. The third energy threshold can be greater than or equal to the second energy threshold, and less than or equal to the first energy threshold. Of course, the third energy threshold can also be greater than the first energy threshold, or less than the second energy threshold, which is not limited here.

[0127] Optionally, when the second-channel voice signal is used to perform a voice task, the first-channel voice signal can be used to perform noise reduction processing on the second-channel voice signal. The first-channel voice signal performs noise reduction processing on the second-channel voice signal as shown in the following formula. The processing is performed in the frequency domain, that is, y (the first-channel voice signal) and x (the second-channel voice signal) are first transformed by fast Fourier transform (FFT), and then transformed to the frequency domain for processing. It can be understood that when processing the first-channel voice signal and the second-channel voice signal, each frame of data after framing and windowing of the first-channel voice signal and the second-channel voice signal is processed respectively.

[0128]

[0129] Among them, s(w) is the denoised speech signal after denoising, which is a frequency domain signal. x(w) is the frequency domain signal corresponding to the second voice signal, and y(w) is the frequency domain signal corresponding to the first voice signal. Represents the frequency domain signal related to x(w) and y(w). The meaning of this formula is to subtract the frequency domain signal related to x(w) in y(w) (i.e. the frequency domain signal of the noise) from x(w) to obtain the denoised speech signal after denoising in the frequency domain.

[0130] That is, the part of the second voice signal that is highly correlated with the first voice signal is filtered out. Since the first voice signal is mainly a noise signal, and the first voice signal and the second voice signal are voice signals collected under the same environment, the part of the second voice signal that is highly correlated with the first voice signal can be considered as noise. The remaining signal s(w) after filtering out the noise is mainly a human voice signal with a higher signal-to-noise ratio, which is used as a noise-reduced voice signal for subsequent business processing.

[0131] In the call service scenario, subsequent service processing includes acoustic processing such as echo cancellation and reverberation cancellation. After a series of acoustic processing, the voice signal is encoded and sent to the other end through the wireless network. In the recording service scenario, subsequent service processing includes, for example, storing the voice signal after noise reduction. In the voice interaction scenario, subsequent processing includes, for example, performing voice recognition processing on the voice signal after noise reduction processing and executing related tasks.

[0132] In order to make this solution easier to understand, for example, Figure 7 As shown, Figure 7 This is a simulation diagram of voice switching based on the solution provided by this application. Figure 7 Taking the second microphone as an air conduction microphone and the second microphone as a bone conduction microphone as an example, the first capability threshold is E2, the second energy threshold is E1, and the voice service is performed using the voice collected by the air conduction microphone at the start of the voice service. The timing time T1 is the first duration threshold, the protection time T2 is the second duration threshold, the duration T3 is the third duration threshold, and the protection time T4 is the fourth duration threshold. It can be seen that after the noise energy (ambient sound energy) of the ambient noise signal reaches the first duration of the threshold E2 (the number of consecutive frames in which the ambient sound energy in the time domain is greater than the threshold E2 satisfies the count A1) and is greater than the duration T1, the voice service is performed by switching to the voice signal collected by the bone conduction microphone. After switching to the bone conduction microphone, after the second duration (satisfying the count A2) of the voice service is continuously performed using the voice signal collected by the bone conduction microphone for greater than the protection time T2, it is allowed to switch to the air conduction microphone, but at this time the ambient sound energy condition is not met, and the voice service is still performed using the voice signal collected by the bone conduction microphone. When the ambient sound energy of the existing frame is less than the threshold E1, the number of consecutive frames whose ambient sound energy is less than the threshold E1 is counted. When the third duration is reached (the count satisfies A3), the voice service is switched to the voice signal collected by the air conduction microphone. After switching to the air conduction microphone, the voice service is performed by the voice signal collected by the air conduction microphone for the fourth duration (satisfying the count A4) greater than the protection time T4, and then the switch to the bone conduction microphone is allowed. However, at this time, the ambient sound energy condition is not met, and the voice service is still performed by the voice signal collected by the air conduction microphone.

[0133] Based on the same inventive concept, Figure 8 As shown, the present application also provides a speech processing device. The speech processing device 800 includes a processing module 801. The speech processing device 800 can be applied to wearable devices, such as headphones, smart glasses, AR head-mounted display devices, VR head-mounted display devices, or neck massagers. The speech processing device 800 can be a wearable device, and can also be a functional module in a wearable device, or a hardware module (such as a chip, etc.). The speech processing device 800 can be applied to smart devices, such as mobile phones, tablet computers, computers, and car computers. The speech processing device 800 can be a smart device, and can also be a functional module in a smart device, or a hardware module (such as a chip, etc.).

[0134] The processing module 801 is used to obtain a first voice signal collected by a first microphone, wherein the first voice signal includes an environmental noise signal, and the environmental noise signal indicates the environmental noise of the environment where the wearable device is located. The first microphone is a microphone for collecting environmental noise, and the environmental noise in the first voice signal can accurately characterize the environmental noise of the environment where the wearable device is located. During the period when the second voice signal collected by the second microphone of the wearable device is used to perform a voice service, the processing module is used to obtain a first duration according to the first voice signal, and the first duration is the duration during which the noise energy of the current environmental noise signal has been continuously greater than the first energy threshold. The processing module is used to determine whether to switch to using a third voice signal to perform a voice service according to the first duration, wherein the third voice signal is collected by the third microphone of the wearable device, and when the noise energy of the environmental noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

[0135] In a possible implementation, the processing module 801 is used to determine to switch to using the third voice signal to perform the voice service when the first duration is greater than the first duration threshold. The processing module 801 is used to determine to still use the second voice signal to perform the voice service when the first duration is less than or equal to the first duration threshold.

[0136] In a possible implementation, the processing module 801 is used to determine whether to switch to using a third voice signal to perform voice services according to a first duration and a second duration, where the second duration is the duration for which the second voice signal has been continuously used to perform voice services.

[0137] In a possible implementation, the processing module 801 is used to determine to switch to using the third voice signal to perform the voice service when the first duration is greater than the first duration threshold and the second duration is greater than the second duration threshold. The processing module 801 is used to determine to still use the second voice signal to perform the voice service when the first duration is less than or equal to the first duration threshold and / or the second duration is less than or equal to the second duration threshold. Optionally, the second duration threshold is greater than the first duration threshold.

[0138] In a possible implementation, the processing module 801 is used to obtain a third duration according to the first voice signal during the period of using the third voice signal to perform the voice service, the third duration being the duration during which the noise energy of the current environmental noise signal has been continuously less than or equal to the second energy threshold, and the second energy threshold is less than or equal to the first energy threshold. The processing module 801 is used to determine whether to switch to using the second voice signal to perform the voice service according to the third duration, and when the noise energy of the environmental noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

[0139] In a possible implementation, the processing module 801 is used to determine to switch to using the second voice signal to perform the voice service when the third duration is greater than the third duration threshold. The processing module 801 is used to determine to still use the third voice signal to perform the voice service when the third duration is less than or equal to the third duration threshold.

[0140] In a possible implementation, the processing module 801 is used to determine whether to switch to using the third voice signal to perform voice services according to the third duration and the fourth duration, where the fourth duration is the duration for which the third voice signal has been continuously used to perform voice services.

[0141] In a possible implementation, the processing module 801 is used to determine to switch to using the second voice signal to perform the voice service when the third duration is greater than the third duration threshold and the fourth duration is greater than the fourth duration threshold. The processing module 801 is used to determine to still use the third voice signal to perform the voice service when the third duration is less than or equal to the third duration threshold and / or the fourth duration is less than or equal to the fourth duration threshold.

[0142] In a possible implementation, the processing module 801 is used to obtain a forced switching instruction, the forced switching instruction indicates that a target voice signal is used to perform a voice service, and the target voice signal is one of a second voice signal and a third voice signal. The processing module 801 is used to switch to using the target voice signal to perform the voice service when the voice signal currently used to perform the voice service is different from the target voice signal.

[0143] In one possible implementation, processing module 801 is used to obtain a fifth duration according to the first voice signal when the target voice signal is the second voice signal, and determine whether to switch to using the third voice signal to perform voice services according to the fifth duration, and the fifth duration is the duration that the noise energy of the current environmental noise signal has been continuously greater than the first energy threshold. Processing module 801 is used to obtain a sixth duration according to the first voice signal when the target voice signal is the third voice signal, and determine whether to switch to using the second voice signal to perform voice services according to the sixth duration, and the sixth duration is the duration that the noise energy of the current environmental noise signal has been continuously less than or equal to the second energy threshold. The combination of forced switching and adaptive switching can both meet user needs and ensure the stability of voice services.

[0144] In a possible implementation, the processing module 801 is used to obtain the initial noise energy of the environmental noise according to the first voice signal at the beginning of the voice service. The processing module 801 is used to determine to use the third voice signal to perform the voice service when the initial noise energy is greater than the fifth energy threshold. The processing module 801 is used to determine to use the second voice signal to perform the voice service when the initial noise energy is less than or equal to the fifth energy threshold. At the beginning of the voice service, the voice signal used to perform the voice service is selected according to the initial noise energy, which can ensure that the voice signal has a high signal-to-noise ratio at the beginning of the voice service, thereby ensuring the quality of the voice service.

[0145] In a possible implementation, the processing module 801 is used to extract the non-voice segment in the first voice signal to obtain the environmental noise signal. The processing module 801 is used to obtain the noise energy of the environmental noise signal. The first voice signal may also contain a user voice signal. The non-voice segment extracted from the first voice signal is a relatively pure environmental noise signal, so that the environmental noise signal can accurately represent the current environmental noise state, and therefore the voice switching decision based on the environmental noise state is more timely and accurate.

[0146] In a possible implementation, the first microphone is a microphone disposed on the wearable device; or the first microphone is a microphone disposed on a smart device connected to the wearable device.

[0147] In a possible implementation, the processing module 801 is used to perform noise reduction processing on the second voice signal using the first voice signal to obtain a noise-reduced voice signal. The processing module 801 is used to perform voice services using the noise-reduced voice signal. The first voice signal can more accurately characterize the current environmental noise state, and the first voice signal and the second voice signal are collected in the same environment. Therefore, using the first voice signal to perform noise reduction processing on the second voice signal can more accurately filter the environmental noise in the second voice signal, thereby obtaining a noise-reduced voice signal with a higher signal-to-noise ratio, which can improve the user's voice clarity and ensure the quality of voice services.

[0148] In a possible implementation, the second microphone is an air conduction microphone, and the third microphone is a bone conduction microphone; or the second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

[0149] Optionally, the speech processing device 800 may further include a transceiver module 802. When the speech processing device 800 is applied to a wearable device, the transceiver module 802 is used to send the second voice signal or the third voice signal to the smart device. Optionally, the transceiver module 802 is also used to send the first voice signal to the smart device.

[0150] When the voice processing device 800 is applied to a smart device, the transceiver module 802 is used to receive the second voice signal or the third voice signal sent by the wearable device. Optionally, the transceiver module 802 is also used to receive the first voice signal sent by the wearable device.

[0151] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, the signal processing method flow of any of the above method embodiments is implemented.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A speech signal processing method, characterized in that: The method comprises: Acquire a first voice signal collected by a first microphone, wherein the first voice signal includes an environmental noise signal, and the environmental noise signal indicates environmental noise of an environment where the wearable device is located; During the execution of the voice service using the second voice signal collected by the second microphone of the wearable device, a first duration is obtained according to the first voice signal, where the first duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than a first energy threshold; Determine whether to switch to using a third voice signal to perform the voice service according to the first duration, where the third voice signal is collected by a third microphone of the wearable device, and when the noise energy of the ambient noise is greater than the first energy threshold, the signal-to-noise ratio of the third voice signal is greater than the signal-to-noise ratio of the second voice signal.

2. The method according to claim 1, characterized in that The determining, according to the first duration, whether to switch to using a third voice signal to perform the voice service includes: When the first duration is greater than the first duration threshold, determining to switch to using the third voice signal to perform the voice service; When the first duration is less than or equal to the first duration threshold, it is determined that the second voice signal is still used to perform the voice service.

3. The method according to claim 1, characterized in that The determining, according to the first duration, whether to switch to using a third voice signal to perform the voice service includes: Whether to switch to using the third voice signal to perform the voice service is determined based on the first duration and the second duration, where the second duration is the duration for which the second voice signal has been continuously used to perform the voice service.

4. The method according to claim 3, characterized in that The determining, according to the first duration and the second duration, whether to switch to using the third voice signal to perform the voice service comprises: When the first duration is greater than the first duration threshold, and the second duration is greater than the second duration threshold, determining to switch to using the third voice signal to perform the voice service; When the first duration is less than or equal to the first duration threshold, and / or the second duration is less than or equal to the second duration threshold, it is determined that the second voice signal is still used to perform the voice service.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: During the period of performing the voice service using the third voice signal, obtaining a third duration according to the first voice signal, the third duration being the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to a second energy threshold, and the second energy threshold is less than or equal to the first energy threshold; Determine whether to switch to using the second voice signal to perform the voice service based on the third duration, and when the noise energy of the ambient noise is less than or equal to the second energy threshold, the signal-to-noise ratio of the second voice signal is greater than the signal-to-noise ratio of the third voice signal.

6. The method according to claim 5, characterized in that The determining, according to the third duration, whether to switch to using the second voice signal to perform the voice service includes: When the third duration is greater than a third duration threshold, determining to switch to using the second voice signal to perform the voice service; When the third duration is less than or equal to the third duration threshold, it is determined that the voice service is still performed using the third voice signal.

7. The method according to claim 5, characterized in that The determining, according to the third duration, whether to switch to using the second voice signal to perform the voice service includes: Whether to switch to using the third voice signal to perform the voice service is determined according to the third duration and the fourth duration, and the fourth duration is the duration for which the third voice signal has been continuously used to perform the voice service.

8. The method according to claim 7, characterized in that The determining, according to the third duration and the fourth duration, whether to switch to using the third voice signal to perform the voice service comprises: When the third duration is greater than the third duration threshold, and the fourth duration is greater than the fourth duration threshold, determining to switch to using the second voice signal to perform the voice service; When the third duration is less than or equal to the third duration threshold, and / or the fourth duration is less than or equal to the fourth duration threshold, it is determined that the third voice signal is still used to perform the voice service.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Obtaining a forced switching instruction, wherein the forced switching instruction indicates that the voice service is performed using a target voice signal, wherein the target voice signal is one of the second voice signal and the third voice signal; When the voice signal currently used to perform the voice service is different from the target voice signal, the target voice signal is switched to be used to perform the voice service.

10. The method according to claim 9, characterized in that After the switching is to use the target voice signal to perform the voice service, the method further includes: When the target voice signal is the second voice signal, a fifth duration is obtained according to the first voice signal, and according to the fifth duration, it is determined whether to switch to using the third voice signal to perform the voice service, wherein the fifth duration is the duration during which the noise energy of the current ambient noise signal has been continuously greater than the first energy threshold; When the target voice signal is the third voice signal, a sixth duration is obtained according to the first voice signal, and according to the sixth duration, it is determined whether to switch to using the second voice signal to perform the voice service, wherein the sixth duration is the duration during which the noise energy of the current ambient noise signal has been continuously less than or equal to the second energy threshold.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: When the voice service starts, obtaining the initial noise energy of the environmental noise according to the first voice signal; When the initial noise energy is greater than a fifth energy threshold, determining to use the third voice signal to perform the voice service; When the initial noise energy is less than or equal to the fifth energy threshold, it is determined to use the second voice signal to perform the voice service.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Extracting a non-speech segment from the first speech signal to obtain the environmental noise signal; The noise energy of the environmental noise signal is obtained.

13. The method according to any one of claims 1 to 12, characterized in that The first sound pickup is a sound pickup arranged on the wearable device; or The first microphone is a microphone arranged on a smart device connected to the wearable device.

14. The method according to any one of claims 1 to 13, characterized in that The performing of the voice service using the second voice signal collected by the second microphone of the wearable device includes: Using the first voice signal to perform noise reduction processing on the second voice signal to obtain a noise-reduced voice signal; The voice service is performed using the noise reduced voice signal.

15. The method according to any one of claims 1 to 14, characterized in that The second sound pickup is an air conduction microphone, and the third sound pickup is a bone conduction microphone; or The second microphone is an omnidirectional air conduction microphone, and the third microphone is a directional air conduction microphone.

16. A speech signal processing device, characterized in that: The device comprises a processing module, and the processing module is used to execute the speech signal processing method according to any one of claims 1 to 15.

17. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor is coupled to the memory, and the processor is configured to execute the speech signal processing method according to any one of claims 1 to 15 based on instructions stored in the memory.

18. The electronic device according to claim 17, characterized in that: The electronic device is the wearable device, and the electronic device further includes the second microphone and the third microphone, the second microphone is used to collect the second voice signal, and the third microphone is used to collect the third voice signal.

19. The electronic device according to claim 17 or 18, characterized in that: The electronic device further includes the first microphone, which is used to collect the first voice signal.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when the computer-readable storage medium is executed on a computer, causes the computer to execute the speech signal processing method according to any one of claims 1 to 15.

Images