Equipment control method and device

Abstract

The present application provides a device control method and apparatus, which is applied to a first device having at least one microphone, including: collecting an audio signal through a microphone; determining the signal type and audio characteristics of the audio signal; the signal type at least including wind noise; judging whether the signal type and audio characteristics meet a preset condition; when the signal type and audio characteristics meet the preset condition, triggering a control command corresponding to the preset condition, the control command is used to control a second device. The implementation of the embodiments of the present application can realize the use of a microphone to control a device without the user making a sound, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of electronic equipment control, and in particular to a method and apparatus based on equipment control. Background Art

[0002] With the development of smart phones, there are more and more portable audio devices, and many portable devices have built-in microphones. For example, mobile phones, tablet computers, headphones and other devices usually have microphones, which are mainly used to collect user voices for calls, recordings, etc.

[0003] In order to expand the application of microphones, there is a solution that can use microphones to realize voice control of devices. Users speak to the microphone to issue voice commands, and the microphone collects the user's voice commands, uses voice recognition technology to recognize the semantics, forms control commands, and then completes the control of the device according to the control commands.

[0004] However, in some occasions where silence is required, users are not suitable for speaking or are unwilling to speak. In such occasions, voice control cannot be used to control the device, causing trouble to the user. Summary of the invention

[0005] The present application provides a device control method and apparatus, which can realize controlling the device using a microphone without the user making any sound, thereby solving the user's pain points and improving the user experience.

[0006] In a first aspect, the present application provides a device control method, which is applied to a first device having at least one microphone, and includes: collecting an audio signal through the at least one microphone; determining the signal type and audio characteristics of the audio signal; the signal type includes at least wind noise; judging whether the signal type and the audio characteristics meet preset conditions; when the signal type and the audio characteristics meet the preset conditions, triggering a control command corresponding to the preset conditions.

[0007] The audio signal may be caused by a silent operation of the user, which is an operation triggered by the user's hand and not perceived by the human ear. For example, the minimum sound intensity that the human ear can hear is 1 decibel (db), so the sound caused by the silent operation of the user may be lower than 1 decibel. The silent operation may be, for example, at least one of rubbing at least one microphone, clicking at least one microphone, fanning near at least one microphone, and blowing near at least one microphone. The audio signal caused by the silent operation may be a wind noise signal or a signal similar to a wind noise signal.

[0008] The signal type may be the type of audio signal caused by wind noise. For example, in a specific classification, the signal type may be divided into: a signal type caused by rubbing the microphone, a signal type caused by clicking the microphone, a signal type caused by blowing air toward the microphone, and a signal type caused by blowing air toward the microphone. The signal type directly reflects the action type of the user's silent operation, and the action type may be divided into: rubbing the microphone, clicking the microphone, blowing air toward the microphone, or blowing air toward the microphone.

[0009] The audio feature may be a target feature in an audio signal, and the audio feature reflects the action characteristics of the silent operation, such as action frequency, action strength, action rhythm, action sequence, etc.

[0010] There is a mapping relationship between the combination of signal type and audio feature and the control command. Therefore, when the signal type and the audio feature meet the preset conditions (for example, the combination of signal type and audio feature meets the preset combination), the control command corresponding to the preset condition (for example, the preset combination) can be triggered.

[0011] It can be seen that in the embodiment of the present application, when the microphone collects the audio signal (wind noise signal) caused by the silent operation of the user rubbing or clicking the microphone or the silent operation of fanning or blowing near the microphone, the first device can identify the signal type and audio characteristics of the audio signal, thereby realizing the recognition of the silent operation, and then can realize the control of the second device according to the control command corresponding to the combination of the signal type and the audio characteristics. In this way, the second device can be controlled even when the user does not make a sound, which expands the user's control form over the device. The input method of silent operation is relatively simple, which improves the concealment of operations such as the second device making an emergency call. In addition, since the first device used in this application itself has a microphone, the control function can be realized by reusing the microphone on the device, without adding additional sensor equipment, and the cost is low, and the device power consumption is low. Therefore, the implementation of this application can greatly enhance the user experience.

[0012] Based on the first aspect, in a possible embodiment, the first device is communicatively connected to a second device, and the control command is used to control the second device.

[0013] For example, the first device may be a wearable device with a microphone such as headphones, smart glasses, smart watches, and smart bracelets, and the second device may be a mobile terminal with a controlled demand such as a smart phone, a tablet computer, and a laptop computer. When the user does not make a sound, the first device may use the method of the present application to control the second device, greatly improving the user experience.

[0014] Based on the first aspect, in a possible embodiment, the control command is used to control the first device. In such a solution, the first device can achieve self-control based on the method described in this application. For example, taking the first device as a smartphone, the smartphone may include one or more microphones, such as a top microphone disposed at the top of the smartphone and a bottom microphone disposed at the bottom of the smartphone. In the case where the user does not make a sound, the user can silently operate at least one of the two microphones as needed to trigger control of the smartphone, thereby expanding the user's means of controlling the smartphone and improving the user's experience.

[0015] Based on the first aspect, in a possible embodiment, the determining the signal type of the audio signal includes: obtaining the signal type by detecting one or more of the time domain characteristics and spectrum characteristics of the audio signal. The time domain characteristics represent the time domain pulse signal of the audio signal caused by the silent operation, for example, it can be reflected by the relationship between the change of amplitude and time; the spectrum characteristics represent the frequency spectrum density of the audio signal caused by the silent operation, for example, it can be reflected by the relationship between the change of amplitude and frequency. The signal type can be quickly identified by an algorithm, and the cost is low.

[0016] For example, different silent operations cause different time domain features of audio signals, and the time domain features represent the time domain pulse signals of audio signals caused by silent operations. Then, after the audio signal is collected, the signal type of the current signal can be determined by a detection algorithm based on the time domain features of the audio signal.

[0017] For another example, the audio signal caused by silent operation may be a wind noise signal or a signal similar to wind noise. The first device may use a wind noise detection algorithm to detect whether the audio signal collected by the microphone contains a signal with wind noise characteristics. For example, a method based on digital signal processing is used to calculate the power spectrum density of the spectrum of the collected audio signal, and the characteristics of the power spectrum density are used to identify whether the audio signal has a signal with wind noise characteristics. If so, it means that the audio signal is caused by silent operation, otherwise, it means that the audio signal is not caused by silent operation (for example, it may be the sound of the user speaking, background environmental noise, etc.). Then, the signal types corresponding to different silent operations can be distinguished by a detection algorithm. For example, for an audio signal caused by rubbing or clicking the sound collecting part of the microphone, since it is applied to the sound collecting part of the microphone at a close distance, the generated audio signal has greater energy than the audio signal caused by blowing wind / air to the microphone. The first device can distinguish the signal type of the audio signal (noise signal) caused by silent operation by the energy of the signal. If the energy is less than the set threshold, it is considered to be the signal type caused by the action of fanning / blowing near the microphone, otherwise it is considered to be the signal type caused by the action of rubbing or clicking the sound collecting part of the microphone. Thus, the first device can distinguish different signal types of silent operation.

[0018] Based on the first aspect, in a possible embodiment, determining the signal type of the audio signal includes: obtaining the signal type according to the audio signal and a neural network model; wherein the neural network model characterizes a mapping relationship between the audio signal and the signal type. The accuracy and recognition efficiency of the signal type can be improved by machine learning methods.

[0019] For example, a machine learning method can be used in advance to train a model using a large amount of training data. The training data includes audio signal data caused by silent operations (such as audio signals caused by silent operations such as rubbing the microphone, clicking the microphone, blowing air into the microphone, etc.) and signal types, that is, the model can characterize the mapping relationship between audio signals and signal types. The first device acquires and saves the model. When the first device collects an audio signal, the audio signal can be input into the model to obtain the signal type output by the model.

[0020] The machine learning model can specifically be one of the following models: Neural Network (NN) model, Deep Neural Network (DNN) model, Factorization-machine supported Neural Networks (FNN) model, Convolutional Neural Networks (CNN) model, Inner Product-based Neural Network (IPNN) model, Outer Product-based Neural Network (OPNN) model, Neural Factorization Machines (NFM) model, etc.

[0021] Based on the first aspect, in a possible embodiment, determining the signal type and audio features of the audio signal includes: obtaining the audio features by extracting features from the audio signal.

[0022] Among them, the audio features include, for example: one or more of the frequency, energy, duration, sound length characteristics, sound intensity characteristics, and the order of collecting different signal types of wind noise pulses in the audio signal. The frequency of wind noise pulses reflects the number of pulse excitations in the audio signal, the speed of pulse excitation, etc. The energy of the pulse reflects the energy size of the pulse excitation in the audio signal. The duration of the pulse reflects the duration of the pulse in the audio signal. The order of collecting different signal types can be the order in which a single microphone collects audio signals caused by silent operations of different types (such as rubbing the microphone, clicking the microphone, fanning the microphone, or blowing air into the microphone), or the order in which different microphones (such as two microphones) collect audio signals caused by the same or different types of silent operations.

[0023] The audio features reflect the action characteristics of silent operation. That is to say, the action characteristics of silent operation (such as one or more of action frequency, action strength, action rhythm, action sequence, etc.) can be identified based on the audio features (such as one or more of the frequency, energy, duration, sound length characteristics, sound intensity characteristics, and the order of collecting different signal types of pulses in the audio signal).

[0024] For example, the audio signal collected by the microphone of the first device may include audio signals generated by multiple action types of silent operation. The audio signal can be segmented by voice activation detection technology, and each segment represents an action type. By detecting the duration of the audio signal corresponding to the action to be identified in a segment, and the interval of the start time of the audio signal corresponding to the action to be identified in different segments, the speed of actions of different action types can be distinguished. By counting the action types in different segments, the action frequency of different action types is obtained. By counting the action types in different segments and the frequency of each action occurring continuously over a period of time, the action rhythm or action sequence of the silent operation can be further obtained. Thereby, the first device obtains the action features corresponding to the audio signal of the silent operation.

[0025] Based on the first aspect, in a possible embodiment, the preset condition is one or more preset combinations including signal type and audio features; the judging whether the signal type and the audio features meet the preset conditions includes: judging whether a combination formed by one or more of the signal type and the frequency, energy, duration, sound length characteristics, sound intensity characteristics, and the order of collecting different signal types of the wind noise pulse meets the preset combination.

[0026] Each preset combination has a mapping relationship with at least one control instruction.

[0027] When the signal type and the audio feature meet the preset condition, triggering the control command corresponding to the preset condition includes: when the combination of the signal type and the audio feature meets the target preset combination in the preset combination, triggering the control command corresponding to the target preset combination.

[0028] In the embodiment of the present application, the action type and the signal type have a corresponding relationship, and the action feature and the signal feature have a corresponding relationship. Therefore, the combination of the signal type and the audio feature (preset combination) can be embodied by the combination of the action type and the action feature (preset combination), then the mapping relationship between the combination of the action type and the action feature and the control command can be stored in advance, so that the corresponding control command can be obtained according to the mapping relationship. Therefore, the implementation of the present application can quickly determine the control command corresponding to the combination of the signal type and the audio feature, reduce the control delay, and improve the user experience.

[0029] Based on the first aspect, in a possible embodiment, the first device is, for example, a headset (such as a wireless headset), and the second device is, for example, a smart device; the first device and the second device are in communication connection. The control command is used to perform at least one of the following controls on the second device: control the second device to make a call, answer / hang up a call, send a message, play / pause music, play / pause a video, adjust a track, adjust the volume, lock / unlock the screen, or turn on / off a specified function mode. The specified function mode can be, for example, one or more functions of a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a listening mode (HearThrough), and the like.

[0030] Based on the first aspect, in a possible embodiment, the first device is, for example, a smart device; the control command is used to perform at least one of the following controls on the first device: controlling the first device to make a call, answer / hang up a call, send a message, play / pause music, play / pause a video, adjust a track, adjust the volume, lock / unlock the screen, or turn on / off a specified function mode. The specified function mode may be, for example, one or more of a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a hearing mode (HearThrough), and the like.

[0031] Based on the first aspect, in a possible embodiment, the control command corresponding to the signal type and the audio feature can also be determined by the following method: determining the action type corresponding to the signal type, the action type indicating the operation type of the user action that causes the noise signal; determining the action feature corresponding to the audio feature, the action feature including one or more of the action frequency, action strength, action rhythm, and action sequence of the user action; determining the control command corresponding to the action type and the action feature based on a mapping relationship between the action type and action feature preset by the user and the control command (for example, a mapping table).

[0032] Based on the first aspect, in a possible embodiment, the action type includes: blowing air toward the at least one microphone fan;

[0033] The action characteristics include at least one of the following: the number of times the wind is blown to the at least one microphone is one or more times, the number of times the wind is blown to the at least one microphone is greater than or equal to a first number threshold, the duration of the wind being blown to the at least one microphone is greater than or equal to a first duration threshold, the intensity of the wind being blown to the at least one microphone is greater than or equal to a first intensity threshold, and the interval of the wind being blown to the at least one microphone is greater than or equal to a first duration interval.

[0034] Based on the first aspect, in a possible embodiment, the action type includes: rubbing the at least one microphone;

[0035] The action characteristics include at least one of the following: the number of times the at least one microphone is rubbed is one or more times, the number of times the at least one microphone is rubbed is greater than or equal to a second number threshold, the duration of rubbing the at least one microphone is greater than or equal to a second duration threshold, the intensity of rubbing the at least one microphone is greater than or equal to a second intensity threshold, and the interval of rubbing the at least one microphone is greater than or equal to a second duration interval.

[0036] Based on the first aspect, in a possible embodiment, the action type includes: blowing toward the at least one microphone;

[0037] The action characteristics include at least one of the following: the number of times of blowing into the at least one microphone is one or more times, the number of times of blowing into the at least one microphone is greater than or equal to a third number threshold, the duration of blowing into the at least one microphone is greater than or equal to a third duration threshold, the intensity of blowing into the at least one microphone is greater than or equal to a third intensity threshold, and the interval of blowing into the at least one microphone is greater than or equal to a third duration interval.

[0038] Based on the first aspect, in a possible embodiment, the action type includes: clicking the at least one microphone;

[0039] The action features include at least one of the following: the number of times the at least one microphone is clicked is one or more times, the number of times the at least one microphone is clicked is greater than or equal to a fourth number threshold, the duration of clicking the at least one microphone is greater than or equal to a fourth duration threshold, the intensity of clicking the at least one microphone is greater than or equal to a fourth intensity threshold, and the interval of clicking the at least one microphone is greater than or equal to a fourth duration interval.

[0040] Based on the first aspect, in a possible embodiment, when other sensors besides a microphone are provided in the first device, the other sensors can also be used to assist the first device in detecting whether there is a noise signal corresponding to the silent operation to be identified in the audio signal collected by the microphone, thereby improving the accuracy and reliability of recognition, further enhancing the user experience, and saving power consumption caused by misidentification.

[0041] Based on the first aspect, in a possible embodiment, the first device can also use the information provided by the second device or other terminal device or server to control the opening or closing of the function of the device control method provided in this application, thereby improving the reliability and accuracy of the device and improving the user experience.

[0042] In a second aspect, the present application provides a device for device control, which is applied to a first device having at least one microphone, and includes: a collection module, which is used to collect audio signals through the at least one microphone; a signal processing module, which is used to determine the signal type and audio characteristics of the audio signal; the signal type at least includes wind noise; a control module, which is used to determine whether the signal type and the audio characteristics meet the preset conditions; when the signal type and the audio characteristics meet the preset conditions, triggering the control command corresponding to the preset conditions. The functional modules of the device are specifically used to implement the method described in the first aspect.

[0043] It can be seen that in the embodiment of the present application, when the microphone collects the audio signal (wind noise signal) caused by the silent operation of the user rubbing or clicking the microphone or the silent operation of fanning or blowing near the microphone, the first device can identify the signal type and audio characteristics of the audio signal, thereby realizing the recognition of the silent operation, and then can realize the control of the second device according to the control command corresponding to the combination of the signal type and the audio characteristics. In this way, the second device can be controlled even when the user does not make a sound, which expands the user's control form over the device. The input method of silent operation is relatively simple, which improves the concealment of operations such as the second device making an emergency call. In addition, since the first device used in this application itself has a microphone, the control function can be realized by reusing the microphone on the device, without adding additional sensor equipment, and the cost is low, and the device power consumption is low. Therefore, the implementation of this application can greatly enhance the user experience.

[0044] Based on the second aspect, in a possible embodiment, the first device is communicatively connected to a second device, and the control command is used to control the second device.

[0045] Based on the second aspect, in a possible embodiment, the control command is used to control the first device.

[0046] Based on the second aspect, in a possible embodiment, the signal processing module is used to: obtain the signal type by detecting one or more of the time domain characteristics and spectrum characteristics of the audio signal; and obtain the audio characteristics by performing feature extraction on the audio signal.

[0047] Based on the second aspect, in a possible embodiment, the signal processing module is used to: obtain the signal type based on the audio signal and the neural network model; wherein the neural network model characterizes the mapping relationship between the audio signal and the signal type; and obtain the audio feature by performing feature extraction on the audio signal.

[0048] Based on the second aspect, in a possible embodiment, the audio features include: one or more of the frequency, energy, duration, sound length features, sound intensity features, and the order of collecting different signal types of wind noise pulses in the audio signal.

[0049] Based on the second aspect, in a possible embodiment, the preset condition is at least one preset combination including signal type and audio characteristics; the control module is specifically used to determine whether the combination formed by the signal type and one or more of the frequency, energy, duration, sound length characteristics, sound intensity characteristics, and the order of collecting different signal types of the wind noise pulse meets the preset combination.

[0050] Based on the second aspect, in a possible embodiment, each preset combination has a mapping relationship with at least one control instruction;

[0051] The control module is specifically configured to trigger a control command corresponding to the target preset combination when the combination of the signal type and the audio feature matches a target preset combination in the preset combinations.

[0052] Based on the second aspect, in a possible embodiment, the first device is, for example, a headset (such as a wireless headset), and the second device is, for example, a smart device; the first device and the second device are in communication connection. The control command is used to perform at least one of the following controls on the second device: control the second device to make a call, answer / hang up a call, send a message, play / pause music, play / pause a video, adjust a track, adjust the volume, lock / unlock the screen, or turn on / off a specified function mode. The specified function mode can be, for example, one or more functions of a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a listening mode (HearThrough), and the like.

[0053] Based on the second aspect, in a possible embodiment, the first device is, for example, a smart device; the control command is used to perform at least one of the following controls on the first device: control the first device to make a call, answer / hang up a call, send a message, play / pause music, play / pause a video, adjust a track, adjust the volume, lock / unlock the screen, or turn on / off a specified function mode. The specified function mode may be, for example, one or more of a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a hearing mode (HearThrough), and the like.

[0054] In a third aspect, an embodiment of the present application provides a device, which is a first device, comprising: at least one microphone; one or more processors; a memory; and one or more computer programs; wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, and when the instructions are executed by the first device, the first device performs the method described in the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a chip, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method in the first aspect or any possible implementation of the first aspect.

[0056] Optionally, as an implementation, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the first aspect or any possible implementation of the first aspect.

[0057] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0058] In the sixth aspect, an embodiment of the present invention provides a computer program product, which may be a software installation package. The computer program product includes program instructions. When the computer program product is executed by an electronic device, the processor of the electronic device executes the method in any embodiment of the first aspect.

[0059] It can be seen that in the embodiment of the present application, when the microphone collects the audio signal caused by the silent operation of the user rubbing or clicking the microphone or the silent operation of fanning or blowing near the microphone, the first device can identify the signal type and audio characteristics of the audio signal, and further determine the action type and action characteristics corresponding to the signal type and audio characteristics, thereby realizing the recognition of the silent operation, and then the control of the second device can be realized according to the control command corresponding to the combination of the action type and the action characteristics. In this way, the second device can be controlled even when the user does not make a sound, which expands the user's control form of the device. The input method of silent operation is relatively simple, which improves the concealment of the second device's operations such as making emergency calls. In addition, since the first device used in this application itself has a microphone, the control function can be realized by reusing the microphone on the device, without adding additional sensor equipment, and the cost is low, and the device power consumption is low. Therefore, the implementation of this application can greatly enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0061] Figure 2 An exemplary schematic diagram of a wireless communication headset provided in an embodiment of the present application;

[0062] Figure 3 An exemplary schematic diagram of an in-ear headset for wireless communication provided in an embodiment of the present application;

[0063] Figure 4 An exemplary schematic diagram of a neck-hanging headset for wireless communication provided in an embodiment of the present application;

[0064] Figure 5 An exemplary schematic diagram of a smart phone provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of an exemplary device structure provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of a flow chart of a device control method provided in an embodiment of the present application;

[0067] Figure 8 A schematic diagram of a silent operation scenario of a friction microphone provided in an embodiment of the present application;

[0068] Fig. 9 A schematic diagram of a silent operation scenario of clicking and wiping a microphone provided in an embodiment of the present application;

[0069] Fig.10 A schematic diagram of a silent operation scenario of blowing air toward a microphone provided in an embodiment of the present application;

[0070] Fig.11 A schematic diagram of a silent operation scenario of blowing air into a microphone provided in an embodiment of the present application;

[0071] Fig.12 A schematic diagram of a scenario implemented by the method of the present application provided in an embodiment of the present application;

[0072] Fig.13 A schematic diagram of a scenario implemented by the method of the present application provided in an embodiment of the present application;

[0073] Fig.14 A schematic diagram of a flow chart of a device control method provided in an embodiment of the present application;

[0074] Fig.15A schematic diagram of a time domain pulse signal of a friction microphone provided in an embodiment of the present application;

[0075] Fig.16 A schematic diagram of a time domain pulse signal of a fan blowing toward a microphone provided in an embodiment of the present application;

[0076] Fig.17 A schematic diagram of a time domain pulse signal of a human voice involved in an embodiment of the present application;

[0077] Fig.18 A schematic diagram of a flow chart of a device control method provided in an embodiment of the present application;

[0078] Fig.19 A schematic diagram of a user interface provided in an embodiment of the present application;

[0079] Fig. 20 A schematic diagram of the structure of a first device provided in an embodiment of the present application, and a schematic diagram of the structure of a system consisting of the first device and the second device. DETAILED DESCRIPTION

[0080] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. The terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to limit a specific order. In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0081] The singular forms of "a", "said" and "the" used in the embodiments of the present application or the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. As used herein, the singular forms of "one", "some" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "include", "have", "include" and / or "contain" specify the presence of stated features, integers, steps, operations, elements, and / or components when used in this article, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0082] It should be noted that the terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0083] The present application provides a device control method based on a microphone, which can be applied to a device with a microphone. The microphone referred to in this article is a device for collecting sound signals. The microphone may also be called a microphone, an earphone, a pickup, a receiver, a microphone, a sound sensor, a sound sensitive sensor, an audio collection device or other appropriate terms. This article mainly describes the technical solution by taking a microphone as an example.

[0084] Herein, a device to which the device control method described in the present application is applied may also be referred to as a first device, and a device controlled by the first device may be referred to as a second device.

[0085] In some embodiments, the first device and the second device are different devices, that is, the first device can control the second device based on the method described in the present application. The first device and the second device can communicate with each other, for example, the first device can control the second device through communication methods such as wireless fidelity (Wifi) communication, Bluetooth communication, infrared communication, or cellular 2 / 3 / 4 / 5 generation (2G / 3G / 4G / 5G) communication.

[0086] Among them, at least one of the first device and the second device may also be referred to as user equipment (UE), wearable device, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, terminal device, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handset, user agent, mobile client, client, or other suitable terminology.

[0087] For example, the first device may be a wearable device with a microphone such as headphones, smart glasses, smart watches, and smart bracelets, or a mobile terminal with a microphone such as a smart phone, a tablet computer, and a laptop computer, or a smart home device with a microphone such as an audio device, a smart TV, a smart air conditioner, and a smart refrigerator, or a vehicle-mounted device with a microphone such as a motorcycle device and a car device. The embodiments of the present application do not impose any special restrictions on the specific form of the first device.

[0088] For example, the second device may be a mobile terminal with a control requirement, such as a smart phone, a tablet computer, or a laptop computer, or a smart home device with a control requirement, such as an audio device, a smart TV, a smart air conditioner, or a smart refrigerator, or a vehicle-mounted device with a control requirement, such as a motorcycle device or a car device. The embodiment of the present application does not impose any special restrictions on the specific form of the second device.

[0089] To facilitate the understanding of the solution, this article will mainly describe the technical solution by taking the first device as a headset and the second device as a mobile terminal (such as a mobile phone) as an example.

[0090] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is provided for an embodiment of the present application, wherein the system architecture includes a mobile terminal (the mobile terminal in the diagram takes a smart phone as an example) and an earphone with a microphone, and a communication connection can be established between the earphone and the mobile terminal.

[0091] From the perspective of the headset's communication method, the headset to which this application is applied can be a wireless headset or a wired headset. A wireless headset is a headset that can be wirelessly connected to a mobile terminal. According to the electromagnetic wave frequency used by the wireless headset, they can be further divided into: infrared wireless headset, meter wave wireless headset (such as FM headset), decimeter wave wireless headset (such as Bluetooth headset), etc. A wired headset is a headset that can be connected to a mobile terminal through a wire (such as a cable). According to the shape of the cable, it can be divided into cylindrical cable headset, noodle cable headset, etc.

[0092] From the perspective of how the headphones are worn, the headphones to which this application is applied can be in-ear headphones, semi-in-ear headphones, ear-hook headphones, neck-hook headphones, head-mounted headphones (on-ear headphones, over-ear headphones), bone conduction headphones, and the like.

[0093] From the perspective of the structure and function of the earphones, the earphones to which the present application is applied may be closed earphones, open earphones, semi-open earphones, semi-open earphones, and the like.

[0094] From the perspective of the noise reduction method of the earphone, the earphone to which the present application is applied may be an earphone with active noise reduction (ANC) function, an earphone with passive noise reduction function, or an earphone without noise reduction function.

[0095] In this document, a wireless headset with a microphone (such as a Bluetooth headset) is mainly used as an example to describe the solution. The number of microphones can be one or more.

[0096] For example, in Figure 2A headset for wireless communication (such as a Bluetooth headset) is shown. The headset includes a microphone. The microphone can be extended out of an earpiece device of the headset to facilitate a user to operate the microphone as needed.

[0097] For example, in Figure 3 An in-ear headset for wireless communication (such as a Bluetooth headset) is shown, and the microphone of the in-ear headset can be built into the earphone receiver device of the in-ear headset, so that the in-ear headset is more portable, light and easy to carry. Each in-ear headset includes a microphone, so it can be understood that a pair of in-ear headsets (i.e., an in-ear headset for the left ear and an in-ear headset for the right ear) includes two microphones, and the microphone of the in-ear headset for the left ear can be called a left microphone, and the microphone of the in-ear headset for the right ear can be called a right microphone.

[0098] For example, in Figure 4 A wireless communication neck-hanging headset is shown, which includes two microphones, a left microphone connected to the left earphone and a right microphone connected to the right earphone. A user can operate at least one of the two microphones as needed, thereby expanding the user's operation methods for the microphones.

[0099] It should be noted that in some other solutions of the present application, the first device and the second device may also be the same device, that is, in such a solution, the first device can achieve self-control based on the method described in the present application. Figure 5 As shown, Figure 5 Taking the first device as a smartphone as an example, the smartphone may include one or more microphones, for example, a top microphone disposed at the top of the smartphone and a bottom microphone disposed at the bottom of the smartphone. The user may operate at least one of the two microphones as required. The specific implementation process of these solutions can refer to the implementation method of the solution when the first device and the second device are different devices, and will not be described in detail herein.

[0100] See also Figure 6 , Figure 6 1 is a schematic diagram of an exemplary device 100 provided in an embodiment of the present application. In some embodiments, the device 100 may be the first device described in an embodiment of the present application. Figure 6As shown, the device 100 includes one or more processors 110, one or more memories 120, a communication interface 130, an audio acquisition circuit and an audio playback circuit. The audio acquisition circuit may further include a microphone 140 and an analog-to-digital converter (ADC) 150. The audio playback circuit may further include a speaker 160 and a digital-to-analog converter (DAC). The above components may communicate on one or more communication buses. They are described as follows:

[0101] The processor 110 is the control center of the device 100. The processor may also be called a control unit, a controller, a microcontroller or other appropriate terms. The processor 110 connects various components of the device 100 using various interfaces and lines. In a possible embodiment, the processor 110 may also include one or more processing cores.

[0102] The memory 120 may be coupled to the processor 110, or connected to the processor 110 via a bus, and is used to store various software programs and / or multiple sets of instructions and data. In a specific implementation, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 120 may also store one or more computer programs, and the one or more computer programs include program instructions for the method described in the present application. The memory 120 may also store a communication program that can be used to communicate with at least one second device. The memory 120 may also store execution instructions such as Fig. 20 Relevant data / code of various functional modules of the first device in the embodiment.

[0103] In a specific embodiment of the present application, the processor 110 may be used to call program instructions in the memory 120 to execute the following steps: Figure 7 , Fig.14 ,or Fig.18 Functions of the first device in the embodiment Alternatively, the processor 110 may control the second device by running various functional modules stored in the memory 120 and calling data (eg, collected audio signals) stored in the memory 120 .

[0104] The communication interface 130 is used to communicate with the second device, and the communication mode can be a wired mode or a wireless mode. When the communication mode is wired communication, the communication interface 130 can be connected to the second device through a cable. When the communication mode is wireless communication, the communication interface 130 is used to receive and send radio frequency signals, and the wireless communication mode supported by it can be, for example, Bluetooth communication, wireless fidelity (wireless-fidelity, Wifi) communication, infrared communication, or cellular 2 / 3 / 4 / 5 generation (2 / 3 / 4 / 5 generation, 2G / 3G / 4G / 5G) communication and other communication modes. In a specific implementation, the communication interface 130 may include but is not limited to: an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chip, a SIM card, and a storage medium. In some embodiments, the communication interface 130 can be implemented on a separate chip.

[0105] The microphone 140 may be used to collect audio signals (or sound signals, which are analog signals), and the analog-to-digital converter 150 may be used to convert the analog signals collected by the microphone 140 into digital signals, and send the digital signals to the processor 110 for processing.

[0106] In an embodiment of the present application, the microphone 140 can collect audio signals, and the audio signals can be not recognized by the human ear in a real-life environment. For example, the user performs silent operations around the microphone 140, and the silent operation refers to an operation triggered by the user's hand and not perceived by the human ear. For example, the user fans the air near the sound collection part of the microphone 140, or touches / rubs / lightly clicks the sound collection part of the microphone 140. The sound collection part of the microphone 140 is the part of the microphone 140 used to collect audio signals. The microphone 140 can collect audio signals caused by the user's silent operation, and finally transmit them to the processor 110 through the analog-to-digital converter 150. The processor 110 can identify the signal collected by the microphone 140, and according to the characteristics of the identification signal, and then according to the pre-set rules, send a control command to the second device through the communication interface 130 to complete the control of the second device. For example, control the second device to perform at least one of making a call, answering / hanging up a call, sending a message, playing / pausing music, playing / pausing a video, adjusting tracks, adjusting volume, locking / unlocking the screen, and turning on / off a specified function mode.

[0107] The communication interface 130 can also be used to receive data (the data is a digital signal) from the second device and transmit it to the processor 110 for processing. The processor 110 transmits the processed data to the digital-to-analog converter 170. The digital-to-analog converter 170 can convert the received data into an analog signal, which is then transmitted to the speaker 160. The speaker is used to play according to the analog signal so that the user can hear the played sound. For example, the analog signal can be a music or voice signal.

[0108] Those skilled in the art will appreciate that the device 100 is only an example provided by the embodiment of the present application. In a specific implementation of the present application, the device 100 may have more or fewer components than those shown, may combine two or more components, or may have different configurations of components.

[0109] For example, in one implementation, when the device 100 is an active noise reduction headset, it also includes a noise reduction processing circuit (ANC circuit), and the noise reduction processing circuit (not shown) is used to implement the active noise reduction function of the device 100. The processor 110 and the noise reduction processing circuit can be integrated on one processor chip, or can be implemented on two independent processor chips.

[0110] It should be noted that, in an optional case, the above-mentioned components of the device 100 can also be coupled together.

[0111] It should be understood that in the various embodiments of the present application, the term "coupling" refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, through various interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but it does not exclude that they may be mechanical interfaces or other forms of interfaces. The embodiments of the present application are not limited to this.

[0112] Based on the above description, some device control methods provided in the embodiments of the present application are given below.

[0113] For the sake of convenience, the various method embodiments described below are expressed as a combination of a series of action steps, but those skilled in the art should be aware that the specific implementation of the technical solution of the present application is not limited by the order of the described series of action steps.

[0114] See also Figure 7 , Figure 7 1 is a flow chart of a device control method provided by an embodiment of the present application. In some implementations, the method can be applied to a first device having at least one microphone. The method includes but is not limited to the following steps:

[0115] S101. Detecting a silent operation of a user around (or near) at least one microphone of a first device.

[0116] In this application, the so-called "silent operation" refers to an operation that is triggered by the user's hand and is not perceived by the human ear. For example, the minimum sound intensity that the human ear can hear is 1 decibel (db), so the sound caused by the user's operation can be lower than 1 decibel.

[0117] In some specific embodiments, the user may trigger the silent operation by touching the first device. For example, the user may trigger the silent operation by using a finger on the sound collecting part of at least one microphone of the first device, where the sound collecting part of the microphone is the part of the microphone used to collect audio signals. The silent operation may include, for example, at least one of the following: rubbing the microphone, clicking (or touching, bumping, hitting, or knocking) the microphone.

[0118] Taking the silent operation of the in-ear headset by the user as an example, the in-ear headset may include at least one microphone. Figure 8 and Fig. 9 , Figure 8 and Fig. 9 The schematic diagram exemplarily shows several scenarios of silent operations of the in-ear headphones by the user.

[0119] in, Figure 8 The silent operation shown is rubbing the microphone, and the user can use a finger to rub the sound collecting part of the microphone, for example, the user rubs on the sound collecting part (sound-transmitting hole) of the microphone, or the user's finger touches the microphone and moves along the microphone for a short distance, or the user's finger moves back and forth to rub the microphone, or the user's finger makes a circle or a curve on the microphone, etc. In this way, the first device can detect the silent operation triggered by the user. Figure 8 (1) in FIG. 1 shows a schematic diagram of a scenario in which a user rubs the microphone along the longitudinal direction of the sound collecting part of the microphone. Figure 8 (2) in FIG. 1 shows a schematic diagram of a scenario in which a user rubs the microphone in the lateral direction of the sound collecting part of the microphone. The arrow in the diagram represents a rubbing operation. Figure 8 (1) and (2) in the figure take three times of friction as examples, and the direction of each friction can be the same or different. In the embodiment of the present application, the so-called "friction microphone" can be a combination of one or more of the following implementation forms:

[0120] The at least one microphone is rubbed once. For example, each time the sound collecting part of the microphone is rubbed once, the first device can detect the operation.

[0121] The at least one microphone is rubbed a number of times greater than or equal to a second number threshold, and the direction of each rubbing can be the same or different, and the second number threshold represents the number of times that triggers the first device to detect the silent operation. For example, in one implementation, the second number threshold is 2 times, and when the number of times the sound collecting part of the microphone is rubbed is equal to 2 times, the first device can detect the operation. For another example, in an implementation, the second number threshold is 2 times, and when the number of times the sound collecting part of the microphone is rubbed exceeds 2 times, the first device can detect the operation.

[0122] The at least one microphone is continuously rubbed for a time period greater than or equal to a second time threshold, and the direction of each rubbing can be the same or different, and the second time threshold represents the operation time that triggers the first device to detect the silent operation. For example, in one implementation, the second time threshold is 2 seconds, and whenever the time period of rubbing the sound-collecting part of the microphone is equal to 2 seconds, the first device can detect the operation. For another example, in an implementation, the second time threshold is 2 seconds, and when the time period of rubbing the sound-collecting part of the microphone exceeds 2 seconds, the first device can detect the operation.

[0123] The at least one microphone is rubbed with an intensity greater than or equal to a second intensity threshold. The position of each rubbing may be the same or different, and the second intensity threshold represents the friction intensity value that triggers the first device to detect the friction action of rubbing the at least one microphone to control the second device, and the click intensity value may be characterized by the sound intensity characteristics or amplitude or energy of the audio signal caused by the friction action. For example, in one implementation, when the friction intensity value of rubbing the at least one microphone is equal to the second intensity threshold, the first device can detect the operation. For another example, in an implementation, when the friction intensity value of rubbing the at least one microphone exceeds the second intensity threshold, the first device can detect the operation.

[0124] Rub the at least one microphone at intervals greater than or equal to the second time interval. The position of each rubbing may be the same or different, and the second time interval represents the time interval that triggers the first device to detect the rubbing of the at least one microphone to control the rubbing action of the second device. For example, in one implementation, the second time interval is 1 second, and when the user rubs the at least one microphone at intervals of 1 second, the first device can detect the operation. For another example, in an implementation, the second time interval is 1 second, and when the user rubs the at least one microphone at intervals of more than 1 second, the first device can detect the operation.

[0125] It should be noted that the above example is described using the user's finger rubbing the microphone as an example. It should be understood that in a possible implementation, other parts of the human body (such as the palm) or other tools (such as gloves, pens, etc.) can also be used to rub the microphone to achieve the silent operation.

[0126] It should also be noted that the above examples are only used to explain the embodiments of the present application. The above second number threshold, second duration threshold, second intensity threshold, and second duration interval may also be other values, which are not limited in the present application.

[0127] Fig. 9 The silent operation shown is clicking the microphone, that is, the user can use a finger to touch, touch, gently bump, or gently tap the sound-collecting part of the microphone. In this way, the first device can detect the silent operation triggered by the user. The circle in the figure represents a click operation (one click is taken as an example in the figure). In the embodiment of the present application, the so-called "clicking the microphone" can specifically be a combination of one or more of the following implementation forms:

[0128] Single click on the at least one microphone. For example, each time the sound collecting part of the microphone is clicked once, the first device can detect the operation.

[0129] The at least one microphone is clicked a number of times greater than or equal to a fourth number threshold, and the position of each click may be the same or different, and the fourth number threshold represents the number of times that triggers the first device to detect the silent operation. For example, in one implementation, the fourth number threshold is 3 times, and whenever the number of times the sound collecting part of the microphone is clicked is equal to 3 times, the first device can detect the operation. For another example, in an implementation, the fourth number threshold is 3 times, and when the number of times the sound collecting part of the microphone is clicked exceeds 3 times, the first device can detect the operation.

[0130] Continuously click the at least one microphone for a time period greater than or equal to a fourth time threshold, and the position of each click may be the same or different, and the fourth time threshold represents the operation time that triggers the first device to detect the silent operation. For example, in one implementation, the fourth time threshold is 2 seconds, and each time the duration of clicking the sound-collecting part of the microphone is equal to 2 seconds, the first device can detect the operation. For another example, in an implementation, the fourth time threshold is 2 seconds, and each time the duration of clicking the sound-collecting part of the microphone exceeds 2 seconds, the first device can detect the operation.

[0131] Click the at least one microphone with an intensity greater than or equal to a fourth intensity threshold. The position of each click may be the same or different, and the fourth intensity threshold represents the click intensity value that triggers the first device to detect the click action of clicking the at least one microphone to control the second device. The click intensity value may be characterized by the sound intensity characteristics, amplitude or energy of the audio signal caused by the click action. For example, in one implementation, when the click intensity value of clicking the at least one microphone is equal to the fourth intensity threshold, the first device can detect the operation. For another example, in an implementation, when the click intensity value of clicking the at least one microphone exceeds the fourth intensity threshold, the first device can detect the operation.

[0132] Click the at least one microphone at intervals greater than or equal to a fourth time interval. The location of each click may be the same or different, and the fourth time interval represents the time interval between two clicks that trigger the first device to detect the silent operation (i.e., click). For example, in one implementation, the fourth time interval is 1 second, and when the user clicks the at least one microphone at intervals of 1 second, the first device can detect the operation. For another example, in an implementation, the fourth time interval is 1 second, and when the user clicks the at least one microphone at intervals of more than 1 second, the first device can detect the operation.

[0133] It should be noted that the above example is described using the user's finger clicking the microphone as an example. It should be understood that in a possible implementation, other parts of the human body (such as knuckles) or other tools (such as pens, branches, etc.) can also be used to click the microphone to achieve the silent operation.

[0134] It should also be noted that the above examples are only used to explain the embodiments of the present application, and the above fourth number threshold, fourth duration threshold, fourth intensity threshold, and fourth duration interval may also be other values, which are not limited in the present application.

[0135] In some other specific embodiments, the user may trigger the silent operation without touching the first device. For example, the user may trigger the silent operation at a relatively close distance from the first device (i.e., near the first device), and the silent operation may include at least one of the following: fanning near at least one microphone of the first device, or blowing toward at least one microphone of the first device.

[0136] Taking the silent operation of the in-ear headset by the user as an example, the in-ear headset may include at least one microphone. Fig.10 and Fig.11 , Fig.10 and Fig.11 The schematic diagrams exemplarily show several scenarios of silent operations of the in-ear headphones by the user.

[0137] Fig.10 The silent operation shown is to blow air at the microphone, that is, the user can use the palm of the hand to fan the sound collecting part of the microphone near the microphone. In this way, the first device can detect the silent operation triggered by the user. The arrow in the figure represents a fanning operation (three fanning operations are taken as an example in the figure). In the embodiment of the present application, the so-called "blowing air at the microphone" can be specifically a combination of one or more of the following implementation forms:

[0138] The at least one microphone is blown once. For example, each time the sound collecting part of the microphone is blown once, the first device can detect the operation.

[0139] The at least one microphone is blown with a number of times greater than or equal to a first count threshold. The position of each fanning can be the same or different, and the first count threshold represents the number of times that triggers the first device to detect the silent operation. For example, in one implementation, the first count threshold is 3 times, and whenever the number of times the sound collecting part of the microphone is fanned is equal to 3 times, the first device can detect the operation. For another example, in an implementation, the first count threshold is 3 times, and when the number of times the sound collecting part of the microphone is fanned exceeds 3 times, the first device can detect the operation.

[0140] The at least one microphone is continuously blown for a duration greater than or equal to a first duration threshold. The position of each fanning may be the same or different, and the first duration threshold represents the duration of the operation that triggers the first device to detect the silent operation. For example, in one implementation, the first duration threshold is 2 seconds, and whenever the duration of fanning the sound collecting part of the microphone is equal to 2 seconds, the first device can detect the operation. For another example, in an implementation, the first duration threshold is 2 seconds, and whenever the duration of fanning the sound collecting part of the microphone exceeds 2 seconds, the first device can detect the operation.

[0141] The at least one microphone is blown with an intensity greater than or equal to a first intensity threshold. The position of each fanning may be the same or different, and the first intensity threshold represents the fan intensity value that triggers the first device to detect the fanning of the microphone to control the fanning action of the second device, and the fan intensity value may be characterized by the sound intensity characteristics or amplitude or energy of the audio signal caused by the fanning, for example. For example, in one implementation, when the fan intensity value of fanning the sound collecting part of the microphone is equal to the first intensity threshold, the first device can detect the operation. For another example, in an implementation, when the fan intensity value of fanning the sound collecting part of the microphone exceeds the first intensity threshold, the first device can detect the operation.

[0142] Fan the at least one microphone at intervals greater than or equal to a first time interval. The position of each fanning may be the same or different, and the first time interval represents the time interval between two fannings that trigger the first device to detect the silent operation (i.e., fanning). For example, in one implementation, the first time interval is 1 second, and when the user fans the sound collecting part of the microphone at intervals of 1 second, the first device can detect the operation. For another example, in an implementation, the first time interval is 1 second, and when the user fans the sound collecting part of the microphone at intervals of more than 1 second, the first device can detect the operation.

[0143] It should be noted that the above examples are described using the user's palm fanning as an example. It should be understood that in a possible implementation, other tools (such as paper, a small fan, etc.) can also be used to fan the microphone to achieve the silent operation.

[0144] It should also be noted that the above examples are only used to explain the embodiments of the present application. The above first number threshold, first duration threshold, first intensity threshold, and first duration interval may also be other values, which are not limited in the present application.

[0145] Fig.11 The silent operation shown is blowing towards the microphone, that is, the user can blow / blow air / exhale / exhale the sound-collecting part of the microphone near the microphone with his mouth. In this way, the first device can detect the silent operation triggered by the user. In the embodiment of the present application, the so-called "blowing towards the microphone" can specifically be a combination of one or more of the following implementation forms:

[0146] Blowing air into the at least one microphone once. For example, each time blowing air into the sound collecting part of the microphone, the first device can detect the operation.

[0147] Blow into the at least one microphone for a number of times greater than or equal to a third number threshold. The position of each blowing can be the same or different, and the third number threshold represents the number of times that triggers the first device to detect the silent operation. For example, in one implementation, the third number threshold is 2 times, and whenever the number of times blowing into the sound-collecting part of the microphone is equal to 2 times, the first device can detect the operation. For another example, in an implementation, the third number threshold is 2 times, and when the number of times blowing into the sound-collecting part of the microphone exceeds 2 times, the first device can detect the operation.

[0148] Blow continuously into the at least one microphone for a time period greater than or equal to a third time threshold. The position of each blowing may be the same or different, and the third time threshold represents the operation time for triggering the first device to detect the silent operation. For example, in one implementation, the third time threshold is 1 second, and when the time of blowing into the sound collecting part of the microphone is equal to 1 second, the first device can detect the operation. For another example, in an implementation, the third time threshold is 1 second, and when the time of blowing into the sound collecting part of the microphone exceeds 1 second, the first device can detect the operation.

[0149] Blow the at least one microphone at an intensity greater than or equal to a third intensity threshold. The position of each fanning may be the same or different, and the third intensity threshold represents the blowing intensity value that triggers the first device to detect blowing into the microphone to control the blowing action of the second device, and the blowing intensity value may be characterized by the sound intensity characteristics or amplitude or energy of the audio signal caused by blowing. For example, in one implementation, when the blowing intensity value of blowing into the sound collecting part of the microphone is equal to the third intensity threshold, the first device can detect the operation. For another example, in an implementation, when the blowing intensity value of fanning the sound collecting part of the microphone exceeds the third intensity threshold, the first device can detect the operation.

[0150] Blow into the at least one microphone at intervals greater than or equal to the third time interval. The position of each blow may be the same or different, and the third time interval represents the time interval between two blows that trigger the first device to detect the silent operation (i.e., blowing). For example, in one implementation, the third time interval is 1 second, and when the user blows into the sound-collecting part of the microphone at intervals of 1 second, the first device can detect the operation. For another example, in an implementation, the third time interval is 1 second, and when the user blows into the sound-collecting part of the microphone at intervals of more than 1 second, the first device can detect the operation.

[0151] It should be noted that the above examples are only used to explain the embodiments of the present application. The above third number threshold, third duration threshold, third intensity threshold, and third duration interval may also be other values, and the present application does not limit them.

[0152] It should also be noted that the values ​​of the second number threshold, the first number threshold, the third number threshold, and the fourth number threshold can be different or the same, which is not limited in this application; the values ​​of the second duration threshold, the first duration threshold, the third duration threshold, and the fourth duration threshold can be different or the same, which is not limited in this application. The values ​​of the second intensity threshold, the first intensity threshold, the third intensity threshold, and the fourth intensity threshold can be different or the same, which is not limited in this application; the values ​​of the second duration interval, the first duration interval, the third duration interval, and the fourth duration interval can be different or the same, which is not limited in this application.

[0153] It should also be noted that, in addition to the silent operation exemplified above, the present application may also modify the silent operation, or integrate two or more of the silent operations. For example, when the first device includes more than two microphones, the silent operation may be formed by combining the operation sequence of different microphones. In addition, based on the technical idea of ​​the present application, other implementation forms of silent operation may also be derived.

[0154] S102: In response to the silent operation, control the second device.

[0155] Specifically, in response to the silent operation detected in S101, the first device generates a corresponding control command (or control command), which is used to instruct the second device to perform a certain function. The first device sends the control command to the second device, and the second device can perform related functional operations according to the control command.

[0156] For example, taking the second device as a smart phone, the first device can control the second device to perform at least one of making a call, answering / hanging up a call, sending a message, playing / pausing music, playing / pausing a video, adjusting a track, adjusting the volume, locking / unlocking the screen, and turning on / off a specified function mode through a control command. The specified function mode may be, for example, a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a listening mode (HearThrough), etc., which is not limited in this application.

[0157] In order to better understand the solution of this application, Fig.12 and Fig.13 The illustrated scenario is described as an example.

[0158] See also Fig.12In a possible application scenario, taking the first device as an in-ear headset and the second device as a smartphone as an example, the in-ear headset may include at least one microphone, and the headset establishes a communication connection with the smartphone, and the user can listen to music in the smartphone through the in-ear headset, such as Fig.12 As shown in (1) in the figure, the music playing interface is displayed on the smartphone, and the speaker of the headset will play music. When the user wants to increase the volume of the headset, since it may be inconvenient or unpleasant for the user to operate directly on the smartphone, the user can choose to rub the sound collecting part of the microphone of the in-ear headset with his fingers, as shown in FIG. Fig.12 (2) in FIG. 1 shows the sound collecting part of the friction microphone. In this way, after the in-ear headset detects the silent operation of the user, it triggers the in-ear headset to generate a control command of "increase volume", and controls the smartphone to increase the volume based on the control command, such as Fig.12 In (3), an interface for increasing the volume of music playback is presented. For example, in a specific implementation, the music playback volume can be designed to increase as the time of rubbing the microphone increases, or it can be designed to increase the volume by a preset level each time the microphone is rubbed, and this application does not limit this. In this way, the purpose of using the microphone to control the smartphone to adjust the music playback volume is achieved without the user making any sound, which is easy to operate and improves the user experience.

[0159] See also Fig.13 In another possible application scenario, taking the first device as an in-ear headset and the second device as a smartphone as an example, the in-ear headset may include at least one microphone, and the headset establishes a communication connection with the smartphone, for example, a user can listen to music in the smartphone through the headset (not shown). Fig.12 In (1), the smartphone receives an incoming call and displays the incoming call interface as shown in the figure. The user may not be convenient or do not like to operate directly on the smartphone, so the user can choose to use the palm of his hand to blow the microphone fan of the in-ear headset, such as Fig.12 (2) in FIG. 1 shows blowing air to the microphone twice. In this way, after the in-ear headset detects the silent operation of the user, it triggers the in-ear headset to generate a control command of "answering the call", and controls the smartphone to answer the call based on the control command, such as Fig.12 The call interface is presented in (3). Subsequently, the user can continue to use the microphone to make a voice call. In this way, the purpose of using the microphone to control the smartphone to make a call is achieved without the user making any sound, which is easy to operate and improves the user experience.

[0160] It can be seen that in the embodiment of the present application, the first device can control the second device by detecting the silent operation of the user rubbing or clicking the microphone or the silent operation of fanning or blowing near the microphone. In this way, the second device can be controlled even when the user does not make a sound, which expands the user's control over the device. The silent operation input method is relatively simple, which improves the concealment of operations such as making emergency calls on the second device. In addition, since the first device used in this application itself has a microphone, the control function can be achieved by reusing the microphone on the device, without adding additional sensor equipment, and the cost is low, and the device power consumption is low. Therefore, the implementation of this application can greatly enhance the user experience.

[0161] See also Fig.14 , Fig.14 This is a specific flow chart of a device control method provided by an embodiment of the present application. In some implementations, the method can be applied to a first device having at least one microphone. The method includes but is not limited to the following steps:

[0162] S201. Collect audio signals through at least one microphone.

[0163] Herein, the audio signal may include a wind noise signal caused by a silent operation of a user.

[0164] In a specific embodiment of the present application, when the user performs silent operations on the at least one microphone such as rubbing the microphone, clicking the microphone, blowing air toward the microphone, etc., the audio signal collected by the first device may be a wind noise signal caused by the user's silent operation, that is, the audio signal may be a signal with similar wind noise characteristics.

[0165] For example, the user may rub or click the sound collecting part of the microphone with a finger, thereby causing the sound collecting part to vibrate and / or generate airflow disturbance, thereby driving the microphone diaphragm to vibrate and generate an audio signal.

[0166] For another example, a user may fan the microphone with his palm to generate air flow, which in turn drives the microphone diaphragm to vibrate and generate an audio signal.

[0167] For another example, the user blows air near the microphone with his mouth, generating airflow, which in turn drives the microphone diaphragm to vibrate and generate an audio signal.

[0168] S202: Determine whether there is a control command corresponding to the audio signal. If there is a control command corresponding to the audio signal, then continue to execute S204; if there is no control command corresponding to the audio signal, then end this process.

[0169] For example, the control command includes at least one of the following control commands: making a call, answering / hanging up a call, sending a message, playing / pausing music, playing / pausing a video, adjusting a track, adjusting the volume, locking / unlocking the screen, and turning on / off a specified function mode. The specified function mode may be, for example, a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a listening mode (HearThrough), etc., which is not limited in this application.

[0170] In another possible implementation of the present application, when a machine learning method is used for model training, the training data used includes audio signal data caused by silent operations (for example, audio signals caused by silent operations such as rubbing a microphone, clicking a microphone, blowing air into a microphone, etc.) and control tags (used to indicate control commands corresponding to the audio signal data), that is, the model can characterize the mapping relationship between audio signals and control commands. The first device obtains and saves the model. When the first device collects an audio signal, the audio signal can be input into the model to obtain the control command output by the model.

[0171] In another possible implementation of the present application, a mapping relationship between a combination of a signal type and audio features of an audio signal caused by a user's silent operation and a control command may be preconfigured, and then the control command corresponding to the audio signal may be determined based on the mapping relationship.

[0172] Among them, the signal type may include wind noise, that is, the signal type may be the type of audio signal caused by wind noise. For example, in a specific classification, the signal type may be divided into: a signal type caused by rubbing the microphone, for example, the user rubs the sound collecting part of the microphone with a finger, causing the vibration of the sound collecting part and / or generating air flow disturbance, thereby driving the microphone diaphragm to vibrate and generate the audio signal; a signal type caused by clicking the microphone, for example, the user clicks the sound collecting part of the microphone with a finger, causing the vibration of the sound collecting part and / or generating air flow disturbance, thereby driving the microphone diaphragm to vibrate and generate the audio signal; a signal type caused by fanning the microphone, for example, the user fanning the microphone with the palm of his hand to generate air flow movement, thereby driving the microphone diaphragm to vibrate and generate the audio signal; a signal type caused by blowing air into the microphone, for example, the user blows air near the microphone with his mouth, generating air flow movement, thereby driving the microphone diaphragm to vibrate and generate the audio signal. Of course, in other embodiments, there may be more or fewer classifications than the examples, or signal types caused by other silent operations may be included, which are not limited here.

[0173] The audio features include, but are not limited to, one or more of: the frequency of pulse signal triggering, the duration of the pulse signal, the energy of the pulse signal, the order of collecting different signal types, the sound length feature, the sound intensity feature, etc. The audio features of the audio signal can reflect the action features of the user's silent operation, and the action features of the silent operation include, for example, one or more of the action frequency, action strength, action rhythm, and action sequence. Among them, the action frequency indicates the number of times the action in the silent operation is implemented, and the action frequency reflects the number of actions, the speed of the action, etc. The action strength indicates the implementation intensity of the action in the silent operation. The action strength indicates the implementation intensity of the action in the silent operation. The action rhythm can indicate the time interval between two actions, such as the time interval between two fanning actions; the action rhythm can also be formed by combining the action frequency and the action strength. The action sequence indicates the order of implementation of actions of different action types when there are multiple action types of silent operation, and / or, when the number of microphones is two or more, the order of implementation of silent operations for different microphones.

[0174] In another possible implementation of the present application, a machine learning method can be used in advance to obtain a model by using a large amount of training data for model training. The training data includes audio signal data caused by silent operations (such as audio signals caused by silent operations such as rubbing the microphone, clicking the microphone, blowing air to the microphone, or blowing air to the microphone) and signal type labels (used to indicate the signal type of the corresponding silent operation), that is, the model can characterize the mapping relationship between audio signals and signal types. The first device acquires and saves the model. When the first device collects an audio signal, the audio signal can be input into the model to obtain the signal type output by the model. In addition, the first device can also extract features of the audio signal of the signal type through an algorithm to obtain audio features. Then, according to the mapping relationship between the combination of preset signal types and audio features and the control command, the control command corresponding to the audio signal can be determined.

[0175] In a specific implementation, the model involved in the embodiments of this article can be one of the following models: Neural Network (NN) model, Deep Neural Network (DNN) model, Factorization-machine supported Neural Networks (FNN) model, Convolutional Neural Networks (CNN) model, Inner Product-based Neural Network (IPNN) model, Outer Product-based Neural Network (OPNN) model, Neural Factorization Machines (NFM) model, etc.

[0176] S203: Control the second device according to the control command.

[0177] In a specific embodiment, the control command is used to instruct the second device to perform a certain function. The first device sends the control command to the second device, and the second device can perform related functional operations according to the control command. For example, taking the second device as a smart phone, the first device can control the second device through a control command to perform at least one of making a call, answering / hanging up a call, sending a message, playing / pausing music, playing / pausing a video, adjusting a track, adjusting the volume, locking / unlocking the screen, and turning on / off a specified functional mode. Among them, the specified functional mode can be, for example, a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a listening mode (HearThrough), etc., which is not limited in this application.

[0178] It should be noted that, in a possible embodiment, the control command may also be used to instruct the first device to perform a certain function by itself. For example, in a scenario where the first device is a mobile phone, the first device can perform at least one of making a call, answering / hanging up a call, sending a message, playing / pausing music, playing / pausing a video, adjusting a track, adjusting the volume, locking / unlocking the screen, and turning on / off a specified function mode according to the control command. Among them, the specified function mode can be, for example, a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a listening mode (HearThrough), etc., which is not limited in this application.

[0179] It can be seen that in the embodiment of the present application, the first device collects the audio signal caused by the silent operation of the user rubbing or clicking the microphone or the silent operation of fanning or blowing near the microphone through the microphone. The control command required to trigger the silent operation can be determined through the audio signal, and then the control of the second device can be realized according to the control command corresponding to the silent operation. In this way, the second device can be controlled even when the user does not make a sound, which expands the user's control form over the device. The input method of silent operation is relatively simple, which improves the concealment of operations such as making emergency calls by the second device. In addition, since the first device used in this application itself has a microphone, the control function can be realized by reusing the microphone on the device, without adding additional sensor equipment, and the cost is low, and the device power consumption is low. Therefore, the implementation of this application can greatly enhance the user experience.

[0180] See also Fig.18 , Fig.18 This is a specific flow chart of another device control method provided by an embodiment of the present application. In some implementations, the method can be applied to a first device having at least one microphone. The method includes but is not limited to the following steps:

[0181] S301. The first device locally stores in advance the mapping relationship between the action type and action characteristics of the silent operation and the control command.

[0182] That is to say, when the action type and the action feature meet the preset condition (ie, the preset combination), there is a mapping relationship between the preset condition (ie, the preset combination) and the control command.

[0183] The action type of the silent operation indicates the operation type of the action in the silent operation, for example, the action type may be rubbing the microphone, clicking the microphone, fanning the microphone, or blowing air toward the microphone, etc.

[0184] The action features of the silent operation include, for example, one or more of action frequency, action strength, action rhythm, and action sequence. Among them: the action type indicates the implementation form of the action in the silent operation; the action frequency indicates the implementation frequency of the action in the silent operation, and the action frequency reflects the number of actions, the speed of the action, etc.; the action strength indicates the implementation intensity of the action in the silent operation; the action rhythm can indicate the time interval between two actions, or it can be formed by combining the action frequency and the action strength; the action sequence indicates the sequential implementation order of actions of different action types when there are multiple action types of the silent operation, and / or, when there are two or more microphones, the sequential implementation order of the silent operation for different microphones.

[0185] The control command is used to control the second device, and the control command includes at least one of the following commands: controlling the second device to make a call, answer / hang up a call, send a message, play / pause music, play / pause a video, adjust a track, adjust the volume, lock / unlock the screen, and turn on / off a specified function mode. The specified function mode may be, for example, a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a hearing mode (HearThrough), and the like.

[0186] For example, in a possible implementation scenario, taking the first device as a headset and the second device as a smart phone, when the first device includes only a single microphone, or only a certain microphone in the first device is configured for silent operation, the mapping relationship information between the combination of configurable action types and action features and the control command is shown in the following Table 1. The mapping relationship information may be obtained by the first device in advance from other devices (such as the second device or other terminal devices) or a server, or may be a factory default setting.

[0187] Table 1

[0188] Preset combinations of action types and action features Control Commands Fan once near the microphone Toggle Start / Stop Click the microphone once Volume Down The duration of rubbing the microphone How much to reduce the volume How many times you rub the microphone How much to increase the volume Click the microphone twice Answer Fan twice near the microphone hang up Click the microphone once, then rub the microphone Make an emergency call … …

[0189] It should be noted that Table 1 is only used to explain the solution of the present application and does not constitute a limitation to the present application.

[0190] For another example, in a possible implementation scenario, taking the first device as a headset and the second device as a smart phone, when the first device includes two microphones (including a left microphone or a right microphone), or only two microphones in the first device are configured for silent operation, the mapping relationship information between the combination of configurable action types and action features and the control command is shown in the following Table 2. Among them, the mapping relationship information can be obtained by the first device in advance from other devices (such as the second device or other terminal devices) or a server, or can be factory default settings.

[0191] Table 2

[0192]

[0193] It should be noted that Table 2 is only used to explain the solution of the present application and does not constitute a limitation to the present application.

[0194] For another example, in a possible implementation scenario, the first device is a headset and the second device is a smartphone. The first device includes two microphones (including a left microphone or a right microphone), or only two microphones in the first device are configured for silent operation. The user can customize the mapping relationship rules through the smartphone. For example, the headset is connected to the smartphone, and the smartphone displays a user interface (UI) for customizing the mapping relationship rules, such as Fig.19 As shown. The user can reset, add, delete, or modify some rules of the mapping relationship between the combination of action type and action feature and control command through the UI interface of the smartphone. After the user completes the setting, the smartphone can send the mapping relationship information to the headset for storage. In this embodiment, the user can customize the rules through the UI interface, making the operation more flexible and convenient, more in line with the user's personal habits, and improving the user experience.

[0195] It should be noted that in the embodiment of the present application, the action type and the signal type have a corresponding relationship, and the action feature and the signal feature have a corresponding relationship. Therefore, in the above mapping relationship, the combination of the action type and the action feature (preset combination) directly reflects the combination of the signal type and the audio feature (preset combination), and the mapping relationship between the combination of the action type and the action feature and the control command directly reflects the mapping relationship between the combination of the signal type and the audio feature and the control command.

[0196] It should be noted that Fig.19 The embodiments are only used to explain the scheme of the present application and do not constitute a limitation on the present application.

[0197] S302: Collect audio signals through at least one microphone.

[0198] The audio signal may include a wind signal (or noise signal) caused by a silent operation of the user.

[0199] In a specific embodiment of the present application, when the user performs silent operations on the at least one microphone such as rubbing the microphone, clicking the microphone, blowing air toward the microphone, etc., the audio signal collected by the first device may be a wind noise signal caused by the user's silent operation, that is, the audio signal may be a signal with similar wind noise characteristics.

[0200] For example, the user may rub or click the sound collecting part of the microphone with a finger, thereby causing the sound collecting part to vibrate and / or generate airflow disturbance, thereby driving the microphone diaphragm to vibrate and generate an audio signal.

[0201] For another example, a user may fan the microphone with his palm to generate air flow, which in turn drives the microphone diaphragm to vibrate and generate an audio signal.

[0202] For another example, the user blows air near the microphone with his mouth, generating airflow, which in turn drives the microphone diaphragm to vibrate and generate an audio signal.

[0203] S303: Determine the signal type and audio features of the audio signal.

[0204] Among them, the signal type may include wind noise, that is, the signal type may be the type of audio signal caused by wind noise. For example, in a specific classification, the signal type may be divided into: a signal type caused by rubbing the microphone, a signal type caused by clicking the microphone, a signal type caused by blowing wind toward the microphone, and a signal type caused by blowing air toward the microphone. Of course, in other embodiments, there may be more or fewer classifications than the examples, or signal types caused by other silent operations may be included, which is not limited here.

[0205] In a specific embodiment, the first device can obtain the signal type of the collected audio signal by detecting the audio signal. Specifically, the signal type can be obtained by detecting one or more of the time domain characteristics and spectrum characteristics of the audio signal. The time domain characteristics represent the time domain pulse signal of the audio signal caused by the silent operation, for example, it can be reflected by the relationship between the change of amplitude and time; the spectrum characteristics represent the frequency spectrum density of the audio signal caused by the silent operation, for example, it can be reflected by the relationship between the change of amplitude and frequency.

[0206] In the specific embodiment of the present application, many different methods can be used to determine the signal type corresponding to the audio signal according to the audio signal. The traditional pattern recognition method based on audio signal extraction can be used, and the signal type can also be determined by a method based on a neural network or a deep learning method. For example, the collected audio signal is caused by the action of fanning the microphone, and the type of this signal can be called a wind noise signal. Whether the traditional pattern recognition method is used, or the method based on a neural network or a deep learning method is used, a large number of wind noise signals caused by the action of fanning the microphone must first be collected. The different distances of the fanning action from the microphone, the different strengths of the fanning action, the frequency of the fanning action, etc. will affect the time-frequency characteristics of the collected audio signal. If the traditional pattern recognition method is used, a large number of wind noise signals caused by the action of fanning the microphone must first be collected to form a training data set, and the wind noise signals in the training data set must be feature extracted to form a feature vector. According to the feature vector training of the training data set, the decision threshold corresponding to each feature is obtained to determine whether the signal type belongs to a wind noise signal. In the process of signal type judgment, feature extraction is performed on the collected audio signal to obtain a feature vector of the audio signal, and based on the feature vector and the judgment threshold corresponding to each feature, it is judged whether the collected audio signal is a wind noise signal. If a method based on neural network or deep learning is used, it is also necessary to collect a large number of wind noise signals caused by the action of blowing wind to the microphone to form a training data set, and train a network for judging whether the input signal is a wind noise signal based on the training data set. In the process of signal type judgment, the collected audio signal is used as the input of the network to obtain the judgment result of the signal type to determine whether the collected audio signal is a wind noise signal.

[0207] When the collected audio signal is caused by the action of rubbing the microphone, the type of this signal can be called a friction signal. A similar method to the above can be used for training and signal type judgment, and the extracted signal features and the judgment of whether it is a wind noise signal may be different.

[0208] In a possible implementation of the present application, the audio signal caused by silent operation may be a wind noise signal or a signal similar to wind noise, and the first device may use a wind noise detection algorithm to detect whether the audio signal collected by the microphone contains a signal with wind noise characteristics. For example, a method based on digital signal processing is used to calculate the power spectrum density of the frequency spectrum of the collected audio signal, and the characteristics of the power spectrum density are used to identify whether the audio signal has wind noise characteristics. If so, it means that the audio signal is caused by silent operation, otherwise, it means that the audio signal is not caused by silent operation (for example, it may be the sound of the user speaking, background environmental noise, etc.). Thereby, the first device is enabled to distinguish between audio signals of silent operation and audio signals of other operations.

[0209] Then, the signal types corresponding to different silent operations can be distinguished by a detection algorithm. For example, for the audio signal caused by rubbing or clicking the sound collecting part of the microphone, since it is applied to the sound collecting part of the microphone at a close distance, the generated audio signal has greater energy than the audio signal caused by fanning / blowing to the microphone. In order to distinguish the action type of the silent operation, the first device can further detect the energy of the signal with wind noise characteristics, so as to determine whether the signal is caused by the action of blowing near the microphone or by rubbing or clicking the sound collecting part of the microphone. For example, the first device can distinguish the signal type of the audio signal (noise signal) caused by the silent operation by the size of the signal energy. If the energy is less than the set threshold, it is considered to be the signal type caused by the action of fanning / blowing near the microphone, otherwise it is considered to be the signal type caused by the action of rubbing or clicking the sound collecting part of the microphone. Thereby, the first device distinguishes the signal types of different silent operations.

[0210] In another possible implementation of the present application, a machine learning method can be used in advance to obtain a model by using a large amount of training data for model training. The training data includes audio signal data caused by silent operations (such as audio signals caused by silent operations such as rubbing the microphone, clicking the microphone, blowing air into the microphone, etc.) and signal types, that is, the model can characterize the mapping relationship between audio signals and signal types. The first device obtains and saves the model. When the first device collects an audio signal, the audio signal can be input into the model to obtain the signal type output by the model.

[0211] The following is an example to explain the scheme. In one scenario, the user fanning his hand near the microphone of the first device generates airflow movement, which drives the microphone diaphragm to vibrate and generate an audio signal. The audio signal is similar to a wind noise signal, that is, the audio characteristics of the audio signal can reflect the characteristics of wind noise. In one implementation, the first device can use a wind noise detection algorithm to detect whether the audio signal collected by the microphone contains a signal with wind noise characteristics. For example, the first device uses a method based on digital signal processing to calculate the power spectrum density of the spectrum of the audio signal, and identifies whether the audio signal has a signal with wind noise characteristics through the characteristics of the power spectrum density. In another implementation, a deep learning method can be used in advance, using a large amount of training data, and a model is obtained through training. Among them, the training data is an audio signal generated by fanning near the microphone and a wind noise label (used to indicate whether the corresponding audio feature has wind noise characteristics). Then, the first device can use the model to identify whether the audio signal collected by the microphone is a signal with wind noise characteristics. If it is a signal with wind noise characteristics, the first device can also detect the energy of the audio signal with wind noise characteristics to determine which silent operation action caused the audio signal. For example, the first device can distinguish the type of audio signal by the energy of the signal. If the energy is less than a set threshold, it is caused by the action of fanning near the microphone; otherwise, it is caused by the action of rubbing the sound collecting part of the microphone. Then, in this example, if the energy is detected to be less than the set threshold, the signal type of the audio signal is determined to be the type caused by the action of fanning near the microphone.

[0212] The audio feature may be a target feature in the audio signal. In a specific embodiment, the first device may obtain the audio feature by performing feature extraction on the audio signal.

[0213] The audio features include, for example, one or more of the frequency, energy, duration, length, intensity, and order of collecting different signal types of pulses in the audio signal. The frequency of the pulses reflects the number of pulse excitations in the audio signal, the speed of the pulse excitation, etc. The energy of the pulse reflects the energy of the pulse excitation in the audio signal. The duration of the pulse reflects the duration of the pulse in the audio signal. The order of collecting different signal types can be the order in which a single microphone collects audio signals caused by different types of silent operations (such as rubbing the microphone, clicking the microphone, blowing air to the microphone, etc.), or the order in which different microphones (such as two microphones) collect audio signals caused by the same or different types of silent operations. The length feature represents the duration of the audio signal caused by the silent operation, and the intensity feature represents the energy of the audio signal caused by the silent operation.

[0214] The audio features reflect the action characteristics of silent operation. That is to say, the action characteristics of silent operation (such as one or more of action frequency, action strength, action rhythm, action sequence, etc.) can be identified based on the audio features (such as one or more of the frequency, energy, duration, sound length characteristics, sound intensity characteristics, and the order of collecting different signal types of pulses in the audio signal).

[0215] S304: Determine an action type corresponding to the signal type, and determine an action feature corresponding to the audio feature.

[0216] In the embodiment of the present application, the audio signal may be caused by the user rubbing the at least one microphone, clicking the microphone, blowing air at the microphone, or blowing air at the microphone, so the signal type of the audio signal and the action type have a one-to-one correspondence, and the action type of the silent operation can be determined according to the signal type. The action type indicates the operation type of the action in the silent operation, for example, the action corresponding to the action type may be rubbing the microphone, clicking the microphone, blowing air at the microphone, or blowing air at the microphone, etc.

[0217] Similarly, the audio features of the audio signal reflect the action features of the silent operation, that is, the audio features of the audio signal and the action features also have a corresponding relationship, so the action features of the silent operation can be determined based on the audio features. The action features of the silent operation include, for example, one or more of the action frequency, action strength, action rhythm, and action sequence. Among them:

[0218] The action frequency indicates the number of times the action in the silent operation is performed, and the action frequency reflects the number of actions, the speed of the actions, etc. For example, quickly rubbing the sound collecting part of the microphone with your hand once, twice, three times (i.e., the action frequency is once, twice, three times), etc.; for another example, quickly clicking the sound collecting part of the microphone with your hand once, twice, three times (i.e., the action frequency is once, twice, three times), etc.; for another example, fanning near the microphone once, twice, three times (i.e., the action frequency is once, twice, three times), etc.; for another example, blowing air near the microphone once, twice, three times (i.e., the action frequency is once, twice, three times), etc., this application does not specifically limit the action frequency.

[0219] The action intensity indicates the intensity of the action in the silent operation, such as gently rubbing the microphone, vigorously rubbing the microphone, gently touching the microphone, vigorously pressing the microphone, blowing slightly toward the microphone, blowing heavily toward the microphone, gently blowing toward the microphone, rapidly blowing toward the microphone, etc. The application does not specifically limit the action intensity.

[0220] The action rhythm can represent the time interval between two actions, such as the time interval between two fanning actions; the action rhythm can also be formed by combining the action frequency and action strength. For example, an action rhythm can be "slightly rub the microphone - rub the Nike fan hard - slightly rub the microphone - rub the Nike fan hard...", and for another example, an action rhythm can be "slightly fan the microphone - slightly fan the microphone - greatly fan the microphone - greatly fan the microphone...", and this application does not specifically limit the action rhythm.

[0221] The action sequence indicates the order in which the actions of different action types are implemented when there are multiple silent operation action types, and / or the order in which the silent operation for different microphones is implemented when there are two or more microphones. For example, one action sequence is to fan the microphone once, and then rub the sound collecting part of the microphone with your hand. For another example, when the microphone includes a left microphone and a right microphone, one action sequence is to rub the sound collecting part of the left microphone once, and then rub the sound collecting part of the right microphone once. The present application does not specifically limit the action sequence.

[0222] In this way, since there is a one-to-one correspondence between the action features of silent operation and the audio features, the action features based on the time axis can be combined to generate the audio features based on the time axis, and then determine the corresponding action features. In one example, the action frequency can be determined according to the frequency of pulses in the audio signal, the action intensity can be determined according to the energy of the pulses, the action rhythm can be determined according to the duration of the pulse signal, and the action sequence can be determined according to the order of collecting different signal types.

[0223] In a possible implementation of the present application, taking the time domain feature of the audio feature as an example, the time domain feature represents the time domain pulse signal of the audio signal caused by the silent operation. Fig.15 , Fig.16 and Fig.17 , Fig.15 Schematic diagram showing a time domain pulse signal (audio signal) caused by the silent operation of the "friction microphone fan wind", Fig.16 A schematic diagram showing a time domain pulse signal (audio signal) caused by a silent operation of "blowing the microphone fan", Fig.17The schematic diagram of the time domain pulse signal (audio signal) caused by the user speaking (i.e. not silent operation) is exemplarily shown. The first device can identify whether the current audio signal is caused by silent operation or other operations (such as speaking) based on the waveform characteristics of different pulse signals (such as the amplitude, frequency, energy and other parameters of the waveform). If it is identified that the current audio signal is caused by silent operation, the audio features in the audio signal can be further identified, such as the frequency of pulse signal triggering, the duration of the pulse signal, the energy of the pulse signal, etc.

[0224] For example, the first device identifies Fig.15 The pulse of the audio signal shown is caused by the action of the friction microphone. The frequency of pulse signal triggering is further identified (for example, the friction frequency is used to indicate the different time intervals between two friction actions, which also corresponds to the action frequency), the duration of the pulse signal (for example, the friction duration is used to indicate the total duration of multiple friction actions in the figure), and the energy of the pulse signal (for example, the amplitude is used to indicate the intensity of the friction action in the figure, which also corresponds to the action strength), that is, the action characteristics corresponding to the audio characteristics of the audio signal are identified.

[0225] For another example, the first device identifies Fig.16 The pulse of the audio signal shown is caused by the action of blowing air toward the microphone. The frequency of pulse signal triggering is further identified (for example, the fanning frequency is used to indicate the different time intervals between two fanning actions, which also corresponds to the action frequency), the duration of the pulse signal (for example, the total duration of multiple fanning actions is indicated by the fanning duration in the figure), and the energy of the pulse signal (for example, the amplitude is used to indicate the intensity of the fanning action, which also corresponds to the action strength) are identified, that is, the action characteristics corresponding to the audio characteristics of the audio signal are identified.

[0226] It should be noted that Fig.15 , Fig.16 and Fig.17 The audio signal shown is only used to explain the solution of the present application and does not constitute a limitation to the present application.

[0227] In a possible implementation of the present application, the audio signal collected by the microphone of the first device may include audio signals generated by multiple action types of silent operation. The audio signal can be segmented by voice activation detection technology, each segment represents an action type, and the speed of actions of different action types can be distinguished by detecting the duration of the audio signal corresponding to the action to be identified in a segment, and the interval of the start time of the audio signal corresponding to the action to be identified in different segments. By counting the action types in different segments, the action frequency of different action types is obtained. By counting the action types in different segments and the frequency of each action occurring continuously over a period of time, the action rhythm or action sequence of the silent operation can be further obtained. Thereby, the first device obtains the action features corresponding to the audio signal of the silent operation.

[0228] In another possible implementation of the present application, a machine learning method can be used in advance to obtain a model by using a large amount of training data for model training. The training data includes audio features and action feature labels of audio signals caused by silent operations (the labels are used to indicate the action features of the corresponding silent operations), that is, the model can characterize the mapping relationship between audio features and action features. The first device acquires and saves the model. When the first device collects an audio signal, the audio features are extracted and input into the model, thereby obtaining the action features output by the model.

[0229] In a specific implementation, the model involved in the embodiments of this article can be one of the following models: Neural Network (NN) model, Deep Neural Network (DNN) model, Factorization-machine supported Neural Networks (FNN) model, Convolutional Neural Networks (CNN) model, Inner Product-based Neural Network (IPNN) model, Outer Product-based Neural Network (OPNN) model, Neural Factorization Machines (NFM) model, etc.

[0230] S305: Determine whether there is a control command corresponding to the audio signal.

[0231] Through S304, the first device has determined the action type and action feature corresponding to the audio signal, then the first device can determine whether the combination of the action type and action feature meets the preset conditions, such as one or more preset combinations containing action types and action features. For example, by querying the mapping relationship between the preset combination (the combination formed by the action type and action feature) saved in S301 and the control command, it is determined whether the combination of the action type and action feature meets the target preset combination in the preset combination, that is, to determine whether the combination exists in the mapping table preset in S301. If so, it means that it meets the target preset combination. At this time, the combination has a corresponding control command in the mapping table, and S306 can be executed later. Otherwise, it means that it does not meet the preset combination, and the combination does not have a corresponding control command, then this process can be terminated.

[0232] For example, the mapping table preset by the first device is as follows Fig.19 As shown, if the action type determined by the first device according to the audio signal is rubbing the left microphone, the action feature determined is rubbing once, that is, the combination of the action type and the action feature exists in the mapping table, so the "lower volume" control command will be triggered later. If the action type determined by the first device according to the audio signal is rubbing the right microphone, the action feature determined is rubbing twice, that is, the combination of the action type and the action feature does not exist in the mapping table, then the process can be ended.

[0233] It should be noted that in the present application, since the action type and the signal type have a corresponding relationship, and the action feature and the signal feature have a corresponding relationship, the embodiment of the present application can also be described as: through S303, the first device has determined the signal type and audio feature corresponding to the audio signal, then the first device can determine whether the combination of the signal type and the audio feature meets the preset condition, and the preset condition is, for example, one or more preset combinations containing the signal type and the audio feature. For example, by querying the mapping relationship between the preset combination (the combination formed by the signal type and the audio feature) saved in S301 and the control command, it is determined whether the combination of the signal type and the audio feature meets the target preset combination in the preset combination, that is, to determine whether the combination exists in the mapping table preset in S301. If so, it means that it meets the target preset combination. At this time, the combination has a corresponding control command in the mapping table, and S306 can be executed later. Otherwise, it means that it does not meet the preset combination, and the combination does not have a corresponding control command, and this process can be terminated.

[0234] S306: Triggering control of the second device according to the control command.

[0235] In a specific embodiment, the control command is used to instruct the second device to perform a certain function. The first device sends the control command to the second device, and the second device can perform related functional operations according to the control command. For example, taking the second device as a smart phone, the first device can control the second device through a control command to perform at least one of making a call, answering / hanging up a call, sending a message, playing / pausing music, playing / pausing a video, adjusting a track, adjusting the volume, locking / unlocking the screen, and turning on / off a specified functional mode. Among them, the specified functional mode can be, for example, a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a listening mode (HearThrough), etc., which is not limited in this application.

[0236] It should be noted that, in a possible embodiment, the control command may also be used to instruct the first device to perform a certain function by itself. For example, in a scenario where the first device is a mobile phone, the first device can perform at least one of making a call, answering / hanging up a call, sending a message, playing / pausing music, playing / pausing a video, adjusting a track, adjusting the volume, locking / unlocking the screen, and turning on / off a specified function mode according to the control command. Among them, the specified function mode can be, for example, a silent mode, a vibration mode, an airplane mode, a power saving mode, an active noise reduction (ANC) function, a listening mode (HearThrough), etc., which is not limited in this application.

[0237] It can be seen that in the embodiment of the present application, the first device pre-stores the mapping relationship between the action type and action characteristics of the silent operation and the control command. When the microphone collects the audio signal caused by the silent operation of the user rubbing or clicking the microphone or the silent operation of fanning or blowing near the microphone, the first device can identify the signal type and audio characteristics of the audio signal, and further determine the action type and action characteristics corresponding to the signal type and audio characteristics, thereby realizing the recognition of the silent operation, and then the control of the second device can be realized according to the control command corresponding to the combination of the action type and action characteristics. In this way, the second device can be controlled even if the user does not make a sound, which expands the user's control form of the device. The input method of silent operation is relatively simple, which improves the concealment of the second device's operations such as making an emergency call. In addition, since the first device used in this application itself has a microphone, the control function can be realized by reusing the microphone on the device, without adding additional sensor equipment, and the cost is low, and the device power consumption is low. Therefore, the implementation of this application can greatly improve the user experience.

[0238] It should be noted that the above embodiments are used to illustrate some implementation plans of the present application. In practical applications, further expansion / deformation / refinement can be performed based on the technical content of the above embodiments.

[0239] For example, in a possible embodiment of the present application, when other sensors other than a microphone are provided in the first device, the other sensors can also be used to assist the first device in detecting whether there is a noise signal corresponding to the silent operation to be identified in the audio signal collected by the microphone, thereby improving the accuracy of identification. For example, if the first device has a motion sensor and two microphones, and the first device is configured to detect the silent operation through a microphone. Then, the first device can determine whether the user is in a stationary state or in a moving state according to the detection result of the motion sensor. Assuming that the user is detected to be in a stationary state, an audio signal corresponding to the silent operation is detected by one microphone, and at the same time, another microphone does not detect the audio signal corresponding to the silent operation, then the first device can determine that the user is performing a silent operation in order to control the second device. Otherwise, if both microphones detect an audio signal, there may be a case of misidentification, and the first device may not perform subsequent control operations. In this way, by using other sensors already in the first device and cooperating with different types of sensors, it is possible to realize the control operation of the second device, and it is also beneficial to improve the reliability of identification, further improve the user experience, and save power consumption caused by misidentification.

[0240] For another example, in a possible embodiment of the present application, the first device can also use the information provided by the second device or other terminal device or server to control the opening or closing of the function of the device control method provided by the present application, thereby improving the reliability of the device. For example, the first device can obtain the weather forecast information provided by the second device. If the weather forecast information indicates that the wind force in the current environment exceeds the preset magnitude, the user can be reminded to turn off the function of the device control method provided by the present application, thereby avoiding the influence of the high wind on the accuracy of the audio signal recognition in the present application solution, further improving the reliability of the solution and improving the user experience.

[0241] See also Fig. 20 , Fig. 20 40 is a schematic diagram of the structure of a first device 40 provided in an embodiment of the present application, and a schematic diagram of a system composed of the first device 40 and the second device 50, wherein the first device and the second device can be connected in communication, and the communication connection is a wireless connection or a wired connection. The first device 40 includes an acquisition module 401, a signal processing module 402, and a control module 403. In some embodiments, the acquisition module 401, the signal processing module 402, and the control module 403 can exist in the form of software code. In a specific implementation, the data / code of the acquisition module 401, the signal processing module 402, and the control module 403 can be stored in a computer program such as a computer program product. Figure 6 The memory 120 shown in FIG. 1 may be operated as follows: Figure 6 The processor 110 is shown. Among them:

[0242] The collection module 401 is used to collect audio signals through the at least one microphone;

[0243] A signal processing module 402, configured to determine a signal type and an audio feature of the audio signal; the signal type at least includes wind noise;

[0244] The control module 403 is used to determine whether the signal type and the audio characteristics meet the preset conditions; when the signal type and the audio characteristics meet the preset conditions, trigger the control command corresponding to the preset conditions, and the control command is used to control the second device.

[0245] In a specific embodiment, the acquisition module 401, the signal processing module 402 and the control module 403 can cooperate with each other to perform the following steps: Figure 7 , Fig.14 ,or Fig.18 The functions of the first device side in the embodiment and the specific implementation contents of each functional module can be referred to the relevant step descriptions of the above method embodiment, and will not be repeated here for the sake of brevity.

[0246] The embodiment of the present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods. If the components of the above signal processing device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0247] Based on such understanding, the embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer or processor, enables the computer or processor to execute any of the methods provided in the embodiments of the present application. The technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device or a processor therein to execute all or part of the steps of the methods described in the embodiments of the present application.

[0248] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application is described in detail with reference to the above-described embodiments, a person skilled in the art should understand that the technical solutions described in the above-described embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. For example, some specific operations in the device embodiment can refer to the previous method embodiment.

Claims

1. A device control method, applied to a first device having at least one microphone and other sensors other than the microphone, wherein the first device is communicatively connected with a second device, characterized in that: include: collecting an audio signal through the at least one microphone; The audio signal includes audio signals generated by multiple action types of silent operation; Determine, based on the signals collected by the other sensors and the audio signals collected by the at least one microphone, whether there is a noise signal corresponding to the silent operation to be identified in the audio signals collected by the at least one microphone; wherein the silent operation includes at least one of the following: rubbing the at least one microphone, clicking the at least one microphone, fanning near the at least one microphone, and blowing toward the at least one microphone; If so, determine the signal type and audio features of the audio signal generated by the multiple action types corresponding to the silent operation; the signal type at least includes wind noise; wherein the signal type has a corresponding relationship with the action type of the silent operation, the audio feature has a corresponding relationship with the action feature of the silent operation, and the action feature includes one or more of action frequency, action strength, action rhythm, and action sequence; Determining whether the signal type and the audio feature meet preset conditions; When the signal type and the audio feature meet a preset condition, a control command corresponding to the preset condition is triggered, and the control command is used to control the second device.

2. The method according to claim 1, characterized in that The control command is used to control the first device.

3. The method according to claim 1, characterized in that The determining of the signal type and audio features of the audio signal generated by the plurality of action types corresponding to the silent operation includes: Obtaining the signal type by detecting one or more of the time domain features and the spectrum features of the audio signal; The audio feature is obtained by extracting the feature of the audio signal.

4. The method according to claim 1, characterized in that: The determining of the signal type and audio features of the audio signal generated by the plurality of action types corresponding to the silent operation includes: According to the audio signal and the neural network model, the signal type is obtained; wherein the neural network model represents the mapping relationship between the audio signal and the signal type; The audio feature is obtained by extracting the feature of the audio signal.

5. The method according to claim 1, characterized in that The audio features include: one or more of the frequency, energy, duration, sound length feature, sound intensity feature, and the order of collecting different signal types of wind noise pulses in the audio signal.

6. The method according to claim 5, characterized in that The preset condition is at least one preset combination of a signal type and an audio feature; The determining whether the signal type and the audio feature meet preset conditions includes: Determine whether a combination formed by one or more of the signal type and the frequency, energy, duration, sound length characteristics, sound intensity characteristics, and the order of collecting different signal types of the wind noise pulse meets the preset combination.

7. The method according to claim 6, characterized in that Each preset combination has a mapping relationship with at least one control instruction; When the signal type and the audio feature meet the preset condition, triggering the control command corresponding to the preset condition includes: When the combination of the signal type and the audio feature meets the target preset combination in the preset combination, a control command corresponding to the target preset combination is triggered.

8. The method according to any one of claims 1 to 7, characterized in that: The control command is used to perform at least one of the following controls: Control the second device to make a call, answer / end a call, send a message, play / pause music, play / pause video, adjust the track, adjust the volume, lock / unlock the screen, or turn on / off a specified function mode.

9. A device for controlling a device, applied to a first device having at least one microphone and other sensors other than the microphone, wherein the first device is communicatively connected with a second device, characterized in that: include: A collection module, used for collecting audio signals through the at least one microphone; The audio signal includes audio signals generated by multiple action types of silent operation; a signal processing module, configured to determine whether there is a noise signal corresponding to the silent operation to be identified in the audio signal collected by the at least one microphone according to the signals collected by the other sensors and the audio signal collected by the at least one microphone; wherein the silent operation includes at least one of the following: rubbing the at least one microphone, clicking the at least one microphone, fanning near the at least one microphone, and blowing toward the at least one microphone; The signal processing module is further used to determine the signal type and audio features of the audio signal generated by the multiple action types corresponding to the silent operation, if any; the signal type at least includes wind noise; wherein the signal type has a corresponding relationship with the action type of the silent operation, the audio feature has a corresponding relationship with the action feature of the silent operation, and the action feature includes one or more of action frequency, action strength, action rhythm, and action sequence; A control module is used to determine whether the signal type and the audio characteristics meet the preset conditions; when the signal type and the audio characteristics meet the preset conditions, a control command corresponding to the preset conditions is triggered, and the control command is used to control the second device.

10. The device according to claim 9, characterized in that The control command is used to control the first device.

11. The device according to claim 9, characterized in that The signal processing module is used for: Obtaining the signal type by detecting one or more of the time domain features and the spectrum features of the audio signal; The audio feature is obtained by extracting the feature of the audio signal.

12. The device according to claim 9, characterized in that The signal processing module is used for: According to the audio signal and the neural network model, the signal type is obtained; wherein the neural network model represents the mapping relationship between the audio signal and the signal type; The audio feature is obtained by extracting the feature of the audio signal.

13. The device according to claim 9, characterized in that The audio features include: one or more of the frequency, energy, duration, sound length feature, sound intensity feature, and the order of collecting different signal types of wind noise pulses in the audio signal.

14. The device according to claim 13, characterized in that The preset condition is at least one preset combination of a signal type and an audio feature; The control module is specifically used to determine whether a combination formed by one or more of the signal type and the frequency, energy, duration, sound length characteristics, sound intensity characteristics, and the order of collecting different signal types of the wind noise pulse meets the preset combination.

15. The device according to claim 14, characterized in that Each preset combination has a mapping relationship with at least one control instruction; The control module is specifically configured to trigger a control command corresponding to the target preset combination when the combination of the signal type and the audio feature matches a target preset combination in the preset combinations.

16. The device according to any one of claims 9 to 15, characterized in that: The control command is used to perform at least one of the following controls: Control the second device to make a call, answer / end a call, send a message, play / pause music, play / pause video, adjust the track, adjust the volume, lock / unlock the screen, or turn on / off a specified function mode.

17. An electronic device, the device being a first device, characterized in that: include: At least one microphone; one or more processors; Memory; and one or more computer programs; wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the first device, cause the first device to perform the method as described in any one of claims 1-8.

Images