Sharing method of smart home appliance microphone, smart home appliance and readable storage medium

Abstract

The present invention discloses a method for sharing microphones of smart home appliances, a smart home appliance and a readable storage medium. The method processes and calculates audio information picked up by multiple microphones, calculates the original amplitude, spectral density, processed average amplitude and signal-to-noise ratio of the audio information by weighted dot product, obtains the confidence value of each audio information, adopts the audio information of the microphone device with the highest confidence value, and finds the audio information that best reflects the actual voice information of the user among the multiple microphones, so that the multiple microphones around the user can work together, and solves the problem that the microphones are far away from the user and the noise near some microphones is loud and further voice interaction is impossible.

Description

Technical Field

[0001] The present invention relates to the field of noise reduction, and in particular to a method for sharing a microphone of a smart home appliance, a smart home appliance and a readable storage medium. Background Art

[0002] Smart homes often involve voice interaction, where users issue voice commands at home and smart devices respond accordingly. Voice interaction involves microphones capturing voice commands. Microphones of different devices are generally independent of each other, and users are generally not located in fixed positions at home. When users are far away from the target device that requires voice interaction, or when users are at any location in the home, or even more so, when users issue voice commands while moving around the home, existing devices cannot meet this requirement. Users need to issue commands toward the microphone of the target device as much as possible, which is inconvenient and reduces the user experience. Summary of the invention

[0003] In order to solve the problems in the prior art, an object of the present invention is to provide a method for sharing a microphone of a smart home appliance, a smart home appliance and a readable storage medium.

[0004] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention provides a method for sharing a microphone of a smart home appliance, comprising the following steps:

[0005] Get the audio information picked up by multiple microphones;

[0006] Determine a corresponding frequency spectrum and energy spectrum according to each of the audio information, and determine an original amplitude corresponding to the frequency spectrum and a spectral density corresponding to the energy spectrum;

[0007] Determine the human voice information corresponding to each of the audio information by using noise reduction processing and endpoint detection;

[0008] Calculate the average amplitude and signal-to-noise ratio corresponding to each of the human voice information;

[0009] Calculate the dot product of the original amplitude, the spectral density, the average amplitude and the signal-to-noise ratio of each piece of audio information with their corresponding weight values ​​to obtain a confidence value of each piece of audio information;

[0010] The microphone corresponding to the audio information with the highest confidence value is set as the preferred microphone.

[0011] As a further improvement of the present invention, the audio information includes a direct audio signal, an echo signal, a reverberation signal and a background noise audio signal, and the energy spectrum is set to the energy spectrum of the direct audio signal.

[0012] As a further improvement of the present invention, the original amplitude includes the audio amplitude of the direct audio signal, the audio amplitude of the echo signal, the audio amplitude of the reverberation signal and the audio amplitude of the background noise audio signal.

[0013] As a further improvement of the present invention, the steps are also included:

[0014] When the audio amplitude of the direct audio signal of any audio information is higher than the minimum amplitude preset value, a confidence value is calculated for the dot product of the audio information.

[0015] As a further improvement of the present invention, the weight ratio of the audio amplitude of the direct audio signal, the spectral density, the audio amplitude of the echo signal, the audio amplitude of the reverberation signal and the audio amplitude of the background noise audio signal is 2:2:1:1:1.

[0016] As a further improvement of the present invention, the weight ratio of the audio amplitude of the direct audio signal, the average amplitude and the signal-to-noise ratio is 2:8:5.

[0017] As a further improvement of the present invention, the steps are also included:

[0018] Speech recognition is performed on the audio information picked up by the preferred microphone.

[0019] As a further improvement of the present invention, the steps are also included:

[0020] When the signal-to-noise ratio of any audio information is lower than the minimum signal-to-noise ratio preset value, the audio information corresponding to the signal-to-noise ratio is excluded.

[0021] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a smart home appliance, including:

[0022] Receiving module, receiving audio information

[0023] Memory, which stores computer programs;

[0024] The processor can implement the steps in the above-mentioned smart home appliance microphone sharing method when executing the computer program.

[0025] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a readable storage medium storing a computer program, which can implement the steps in the above-mentioned smart home appliance microphone sharing method when executed by a processor.

[0026] Compared with the prior art, the present invention has the following beneficial effects: by processing and calculating the audio information picked up by multiple microphones, the original amplitude, spectral density, processed average amplitude and signal-to-noise ratio of the audio information are calculated by weighted dot product to obtain the confidence value of each audio information, and the audio information of the microphone device with the highest confidence value is used. This method finds the audio information that best reflects the user's actual voice information among multiple microphones, so that multiple microphones around the user can work together, solving the problem that the microphones are far away from the user and the noise near some microphones is loud and further voice interaction is impossible. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a method for sharing a microphone of a smart home appliance according to an embodiment of the present invention;

[0028] Figure 2 Schematic diagram of a control module according to an embodiment of the present invention.

[0029] Figure 3 It is a schematic diagram of a scenario in which a microphone is selected according to an embodiment of the present invention.

[0030] Among them: 10, refrigerator; 11, memory; 12, processor; 13, receiving module; 14, communication bus; E1, smart TV; E2, smart air conditioner; E3, smart speaker; E4, smart light; P1, user A; P2, user B; P3, user C. DETAILED DESCRIPTION

[0031] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0032] An embodiment of the present invention provides a method for sharing a microphone of a smart home appliance, a smart home appliance and a readable storage medium. The method processes and analyzes audio signals of multiple microphones to select a microphone with higher information reliability, thereby facilitating intelligent selection of a suitable microphone for voice interaction with a user.

[0033] Figure 1 This is a method for sharing a microphone of a smart home appliance in one embodiment of the present application. Although the present application provides method operation steps as described in the following embodiment or flowchart, based on routine or no creative labor, the method has steps in which there is no necessary causal relationship logically, and the execution order of these steps is not limited to the execution order provided in the embodiment of the present application.

[0034] Specifically, the steps include:

[0035] Step 101: Acquire audio information picked up by multiple microphones;

[0036] Here, there are generally multiple electronic devices that support voice functions in the home, such as smart speakers, mobile phones, remote controls, and other smart devices that support voice interaction, such as smart refrigerators, all of which have microphones. These smart devices are distributed in various locations in the home, and the audio information received by the microphones of these devices is obtained, such as Figure 2 As shown, the audio information of 4 microphones is obtained.

[0037] Step 102: determining a corresponding frequency spectrum and energy spectrum according to each of the audio information, and determining an original amplitude corresponding to the frequency spectrum and a spectral density corresponding to the energy spectrum;

[0038] Here, the voice commands issued by the user are generally finite-length, non-periodic signals, so they can be analyzed in the form of energy spectrum. The density of the energy spectrum describes the distribution of the signal's energy with frequency. By calculating its spectral density, the distribution of the energy of the voice signal at each frequency point is reflected.

[0039] The original amplitude reflects the strength of the audio signal. The larger the amplitude, the higher the strength of the sound signal received by the audio signal at some frequencies. However, when the user is moving, for a voice command, different microphones may have higher amplitude points at different frequencies.

[0040] Step 103: Determine the human voice information corresponding to each of the audio information by using noise reduction processing and endpoint detection;

[0041] Step 104: Calculate the average amplitude and signal-to-noise ratio corresponding to each of the human voice information;

[0042] Here, after preprocessing the audio information, in order to make the audio clearer and easier to process, noise reduction and endpoint detection operations are required, and then the average amplitude and signal-to-noise ratio of the processed audio are calculated. The average amplitude reflects the overall strength of the voice received by the microphone during the entire voice command process. The signal-to-noise ratio is the ratio of voice information to background noise. The larger the ratio, the higher the reliability of the voice information.

[0043] In addition, when the audio information is judged as above, it is mainly divided into two steps, the preprocessing of the audio information, and the analysis of the information after the audio information noise reduction and endpoint detection. The noise reduction processing of the audio information is mainly used to remove the background noise, and the endpoint detection of the audio information is used to distinguish the user's voice command information from the ambient background sound. In the process of removing the background noise by the audio noise reduction, it is inevitable that the voice command itself will also be affected, and the background noise generally cannot be completely removed. The more the background noise is removed, the more the voice command part is lost. Similarly, in the process of endpoint detection, there is also the problem of being unable to completely distinguish, and the voice command itself is affected when distinguishing. Therefore, in order to avoid the problem of partial loss of voice commands in the audio information after the noise reduction and endpoint detection of the audio information, the audio information is first preprocessed, that is, the frequency spectrum and energy spectrum of the audio information are directly obtained, and the original audio information is judged.

[0044] Step 105: Calculate the dot product of the original amplitude, the spectral density, the average amplitude and the signal-to-noise ratio of each piece of audio information with their corresponding weight values ​​to obtain a confidence value of each piece of audio information;

[0045] Step 106: Set the microphone corresponding to the audio information with the highest confidence value as the preferred microphone.

[0046] When a user issues a voice command, there may be a pause in a sentence, and the content before and after the pause is semantically the content of one command, but the duration of the pause is generally different from the pause between two command information, that is, the pause time within a command is generally not long, but at the pause position, the amplitude of the audio will decrease, which will reduce the average amplitude of the entire audio information. If the microphones at other locations receive noise irrelevant to the voice command, causing the average amplitude to increase, then the average amplitude alone is not conducive to ensuring the reliability of the audio information received by the microphone. Therefore, the confidence value is calculated by weighting the information of the above multiple dimensions and then dot-producting. The confidence value reflects the reliability of the audio information. The microphone with the highest reliability for a voice command is determined when the intensity of the received user's voice command is high, the background noise is low, the overall energy intensity of the received voice is high during the user's movement, and the average amplitude is high.

[0047] When a user issues a voice command at any location, the audio information detected by microphones at different locations may be different. For example, some microphones are far away from the user and the sampling information is unclear. Some microphones have high ambient noise and cannot accurately identify the voice information they capture. Therefore, it is necessary to select the most reliable audio information from the audio information picked up by different microphones.

[0048] Furthermore, the audio information includes a direct audio signal, an echo signal, a reverberation signal and a background noise audio signal, and the energy spectrum is set to the energy spectrum of the direct audio signal, which affects the quality of the audio after endpoint detection.

[0049] Furthermore, the original amplitude includes the audio amplitude of the direct audio signal, the audio amplitude of the echo signal, the audio amplitude of the reverberation signal and the audio amplitude of the background noise audio signal.

[0050] The above four audio amplitudes all affect endpoint detection and the signal-to-noise ratio after noise reduction to varying degrees.

[0051] The original amplitude weighted value in step 105 is a value obtained by performing dot product calculation on the multiple audio amplitudes contained in the original amplitude after assigning respective weights to each audio amplitude, wherein weight 1 is the weight value corresponding to the spectral density, weight 2 is the weight value corresponding to the average amplitude, and weight 3 is the weight value corresponding to the signal-to-noise ratio.

[0052] Furthermore, the method further comprises the steps of:

[0053] When the audio amplitude of the direct audio signal of any audio information is higher than the minimum amplitude preset value, the confidence value is calculated for the dot product of the audio information. When the audio amplitude of the direct audio signal of any audio information is lower than the minimum amplitude preset value, the microphone corresponding to the audio information may be too far away from the user and has not received a voice command of appropriate intensity. Therefore, in order to reduce resource usage, the audio information of the microphone is not analyzed, and only the audio information of the microphone that is roughly judged to be closer to the user and receives audio information of higher intensity is processed.

[0054] Furthermore, a weight ratio of the audio amplitude of the direct audio signal, the spectral density, the audio amplitude of the echo signal, the audio amplitude of the reverberation signal, and the audio amplitude of the background noise audio signal is 2:2:1:1:1.

[0055] Furthermore, the ratio of the audio amplitude of the direct audio signal, the average amplitude and the weight of the signal-to-noise ratio is 2:8:5.

[0056] In this embodiment, confidence value = audio amplitude of direct audio signal*10%+spectral density*10%+audio amplitude of echo signal*5%+audio amplitude of reverberation signal*5%+audio amplitude of background noise frequency signal*5%+average amplitude*40%+signal-to-noise ratio*25%. The selection of the preferred microphone is a probability problem, and the weight distribution is also in the form of probability. Higher weights are assigned to indicators with greater influencing factors. Through experimental simulation and the preset of some empirical values, the reliability of the confidence value is higher under such weight configuration.

[0057] After selecting a suitable microphone as the preferred microphone, the audio information picked up by the preferred microphone is voice recognized for voice interaction with the user. The audio information of the preferred microphone can be voice recognized and the voice commands therein can be analyzed to perform more intelligent operations.

[0058] Additionally, the steps include:

[0059] When the signal-to-noise ratio of any audio information is lower than the preset minimum signal-to-noise ratio, the audio information corresponding to the signal-to-noise ratio is excluded. When the signal-to-noise ratio is too low, the quality of the detected audio information may be low, or the noise may be too loud, reflecting that the audio information cannot respond well to the voice command. In order to avoid making wrong feedback and minimize the waste of resources, the audio information content with too low signal-to-noise ratio is not further judged and processed, and only the audio with a signal-to-noise ratio higher than the preset minimum signal-to-noise ratio is analyzed.

[0060] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a smart home appliance, the module diagram of which is shown in FIG. Figure 2 As shown, including:

[0061] A receiving module 13 receives audio information;

[0062] A memory 11 for storing computer programs;

[0063] The processor 12 can implement any one of the steps in the above-mentioned method for sharing the microphone of a smart home appliance when executing the computer program, that is, implement the steps in any one of the technical solutions in the above-mentioned method for sharing the microphone of a smart home appliance.

[0064] In addition, the smart home appliance may further include a communication bus 14, which is used to establish a connection between the receiving module 13, the processor 12 and the memory 11. The communication bus 14 may include a path to transmit information between the above-mentioned receiving module 13, the processor 12 and the memory 11.

[0065] The receiving module 13 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc. The memory 11 can be independent and connected to the processor 12 through the communication bus 14. The memory 11 can also be integrated with the processor 12. The processor 12 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0066] In this embodiment, the smart home appliance can be set as a refrigerator 10, and the above-mentioned receiving module 13, processor 12, memory 11 and communication bus 14 are all integrated on the refrigerator 10. The refrigerator 10 receives audio information collected by microphones of other devices through the receiving module 13, and analyzes and processes these audio information using the above-mentioned microphone sharing method. In other words, the receiving module 13 on the refrigerator 10 receives audio information from microphones of various devices, processes these audio information according to the above-mentioned shared microphone control method, and determines the most reliable microphone device to perform other intelligent operations.

[0067] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by the processor 12, it can implement any one of the steps in the above-mentioned smart home appliance microphone sharing method, that is, implement the steps in any one of the technical solutions in the above-mentioned smart home appliance microphone sharing method.

[0068] In the above description, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system implementation described above is only schematic, for example, the division of device modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0069] For example, the processor 12 of the present application may be concentrated in one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0070] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for a computer system (which can be a personal computer, a server, or a network system, etc.) or a processor 12 (processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory 11 (Read-Only Memory, ROM), random access memory 11 (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0071] The following is an example of a specific scenario. Figure 3 As shown, the smart refrigerator 10, smart TV E1, smart speaker E3, smart lamp E4, and smart air conditioner E2 in the user's home all have built-in microphones. The smart home appliance for the above-mentioned method of processing the sharing of smart home appliance microphones is set to the smart refrigerator 10, that is, the processor 12 in the smart refrigerator 10 is used to analyze and process the audio information collected by each smart device. When the user is at any location in the home, for example, at different locations of user A P1, user B P2, and user C P3, there are differences in the processing results of the audio information of different smart devices.

[0072] When a user issues a voice command at any location in the home, for example, an instruction is issued at the location of user B P2, assuming that the sound wave of the voice command is received by the microphones of the smart TV E1, the smart speaker E3 and the smart air conditioner E2, and the microphones convert the sound wave signals into electrical signals. The three devices transmit the electrical signals of the audio information to the processor 12 of the refrigerator 10 in a wired or wireless form. The processor 12 processes the audio information transmitted by the above microphones according to the above-mentioned sharing method of the smart home appliance microphone, determines the corresponding frequency spectrum and energy spectrum, and calculates the original amplitude, spectral density, average amplitude and signal-to-noise ratio of the audio information, calculates the confidence value, and determines that the microphone with the highest confidence value is the preferred microphone. Assuming that the confidence value of the audio information output by the microphone of the smart TV E1 is the highest, the microphone of the smart TV E1 can better reflect the human voice command information of user B P2, recognize the audio information of the microphone of the smart TV E1, and perform further human-computer interaction actions.

[0073] Compared with the prior art, this embodiment has the following beneficial effects:

[0074] By processing and calculating the audio information picked up by multiple microphones, the confidence value of each audio information is obtained by weighted dot product calculation of the original amplitude, spectral density, processed average amplitude and signal-to-noise ratio of the audio information, and the audio information of the microphone device with the highest confidence value is used. This method finds the audio information that best reflects the user's actual voice information among multiple microphones, so that multiple microphones around the user can work together, solving the problem that the microphones are far away from the user and the noise near some microphones is loud and further voice interaction cannot be achieved.

[0075] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0076] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for sharing a microphone of a smart home appliance, characterized in that: The steps include: Get the audio information picked up by multiple microphones; Determine a corresponding spectrum according to each of the audio information, and determine an original amplitude corresponding to the spectrum, wherein the original amplitude includes an audio amplitude of a direct audio signal, an audio amplitude of an echo signal, an audio amplitude of a reverberation signal, and an audio amplitude of a background noise audio signal; Determine a corresponding energy spectrum according to each of the audio information, and determine a spectral density corresponding to the energy spectrum, wherein the energy spectrum is an energy spectrum of a direct audio signal of the audio information; Determine the human voice information corresponding to each of the audio information by using noise reduction processing and endpoint detection; Calculate the average amplitude and signal-to-noise ratio corresponding to each of the human voice information; Calculate the dot product of the audio amplitude of the direct audio signal, the audio amplitude of the echo signal, the audio amplitude of the reverberation signal, the audio amplitude of the background noise audio signal, the spectral density, the average amplitude and the signal-to-noise ratio of each of the audio information with their corresponding weight values ​​to obtain a confidence value for each of the audio information; Setting the microphone corresponding to the audio information with the highest confidence value as the preferred microphone; Also includes the steps: When the audio amplitude of the direct audio signal of any audio information is lower than the minimum amplitude preset value, the audio information of the microphone is not analyzed.

2. The method for sharing a microphone of a smart home appliance according to claim 1, characterized in that: The audio information includes an echo signal, a reverberation signal and a background noise audio signal.

3. The method for sharing a microphone of a smart home appliance according to claim 2, characterized in that: Also includes the steps: When the audio amplitude of the direct audio signal of any audio information is higher than the minimum amplitude preset value, a confidence value is calculated for the dot product of the audio information.

4. The method for sharing a microphone of a smart home appliance according to claim 2, characterized in that: The weight ratio of the audio amplitude of the direct audio signal, the spectral density, the audio amplitude of the echo signal, the audio amplitude of the reverberation signal, and the audio amplitude of the background noise audio signal is 2:2:1:1:

1.

5. The method for sharing a microphone of a smart home appliance according to claim 4, characterized in that: The weight ratio of the audio amplitude of the direct audio signal, the average amplitude and the signal-to-noise ratio is 2:8:

5.

6. The method for sharing a microphone of a smart home appliance according to claim 1, characterized in that: Also includes the steps: Perform speech recognition on the audio information picked up by the preferred microphone.

7. The method for sharing a microphone of a smart home appliance according to claim 1, characterized in that: Also includes the steps: When the signal-to-noise ratio of any audio information is lower than the minimum signal-to-noise ratio preset value, the audio information corresponding to the signal-to-noise ratio is excluded.

8. A smart home appliance, characterized in that: include: Receiving module, receiving audio information Memory, which stores computer programs; The processor, when executing the computer program, can implement the steps in the smart home appliance microphone sharing method described in any one of claims 1 to 7.

9. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the smart home appliance microphone sharing method described in any one of claims 1 to 7 can be implemented.

Images