Control method, model training method and device, equipment and medium

Abstract

The invention discloses a control method, a model training method and device, equipment and a medium, and belongs to the technical field of electronic equipment. The control method comprises the following steps: acquiring a first sound signal acquired by a microphone; the first sound signal is input into an impulse signal detection model to obtain feature information of the impulse signal, and the impulse signal detection model is used for detecting the impulse signal in the sound signals input into the impulse signal detection model; according to the feature information, determining a knocking operation of the user on the sound outlet hole of the microphone; and executing control operation corresponding to the knocking operation.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to a control method, a model training method, a device, a device and a medium. Background Art

[0002] With the development of smart electronic devices, the buttons of electronic devices have gradually solidified into two volume buttons and one power button. By pressing the volume button and the power button, corresponding control operations can be performed, such as taking screenshots, recording screens, turning on or off a flashlight, etc.

[0003] However, as user demands increase, relying solely on buttons can no longer satisfy people's demands for controlling electronic devices. Therefore, how to satisfy users' demands for controlling electronic devices is an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a control method, a model training method, an apparatus, a device and a medium that can meet the user's control needs for electronic devices.

[0005] In a first aspect, an embodiment of the present application provides a control method, including:

[0006] Acquire a first sound signal collected by a microphone;

[0007] Inputting the first sound signal into an impulse signal detection model to obtain characteristic information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model;

[0008] Determine, based on the feature information, a tapping operation of the user on the microphone sound outlet;

[0009] Execute the control operation corresponding to the tapping operation.

[0010] In a second aspect, an embodiment of the present application provides an impulse signal detection model training method, comprising:

[0011] Acquire a fourth sound signal of the user tapping the microphone sound outlet hole multiple times;

[0012] The impulse signal detection model is trained using the fourth sound signal as sample data until the first condition is met; wherein the impulse signal detection model is used to detect an impulse signal in a first sound signal input to the impulse signal detection model, and the first sound signal is a sound signal collected by a microphone;

[0013] The first condition includes at least one of the following:

[0014] The loss value of the loss function of the impulse signal detection model reaches the minimum value;

[0015] The false rejection rate of impulse signal detection is less than a second threshold;

[0016] The false acceptance rate of impulse signal detection is less than a third threshold.

[0017] In a third aspect, an embodiment of the present application provides a control device, including:

[0018] A first acquisition module, used to acquire a first sound signal collected by a microphone;

[0019] A feature information extraction module, used for inputting the first sound signal into an impulse signal detection model to obtain feature information of the impulse signal, wherein the impulse signal detection model is used for detecting the impulse signal in the sound signal input into the impulse signal detection model;

[0020] A determination module, used to determine a tapping operation of the user on the microphone sound outlet according to the feature information;

[0021] The execution module is used to execute the control operation corresponding to the tapping operation.

[0022] In a fourth aspect, an embodiment of the present application provides an impulse signal detection model training device, comprising:

[0023] A second acquisition module is used to acquire a fourth sound signal caused by the user tapping the microphone sound outlet hole multiple times;

[0024] A training module, used to train an impulse signal detection model using the fourth sound signal as sample data until a first condition is met; wherein the impulse signal detection model is used to detect an impulse signal in a first sound signal input to the impulse signal detection model, and the first sound signal is a sound signal collected by a microphone;

[0025] The first condition includes at least one of the following:

[0026] The loss value of the loss function of the impulse signal detection model reaches the minimum value;

[0027] The false rejection rate of the impulse signal detection model is less than a second threshold;

[0028] The false acceptance rate of the impulse signal detection model is less than a third threshold.

[0029] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the control method or the steps of the impulse signal detection model training method provided in the embodiment of the present application are implemented.

[0030] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the control method or the steps of the impulse signal detection model training method provided in the embodiment of the present application are implemented.

[0031] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the steps of the control method or the steps of the impulse signal detection model training method provided in the embodiment of the present application.

[0032] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the control method or the steps of the impulse signal detection model training method provided in the embodiment of the present application.

[0033] In the embodiment of the present application, the first sound signal collected by the microphone is obtained; the first sound signal is input into the impulse signal detection model to obtain characteristic information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model; according to the characteristic information, the user's tapping operation on the microphone sound outlet is determined; and the control operation corresponding to the tapping operation is executed. In this way, the electronic device can be controlled by the user's tapping operation on the microphone sound outlet, which can meet the user's control needs for the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic diagram of a time domain signal corresponding to a sound signal generated by tapping a sound outlet of a microphone provided in an embodiment of the present application;

[0035] Figure 2 is a schematic diagram of a frequency domain signal corresponding to a sound signal generated by tapping a sound outlet of a microphone provided in an embodiment of the present application;

[0036] Figure 3 is a schematic diagram of an audio signal of knocking on a microphone sound outlet three times provided in an embodiment of the present application;

[0037] Figure 4 It is a flow chart of the control method provided in the embodiment of the present application;

[0038] Figure 5 is a schematic diagram of audio signals when a finger touches and leaves the microphone sound outlet provided in an embodiment of the present application;

[0039] Figure 6 It is a flowchart of the impulse signal detection model training method provided in the embodiment of the present application;

[0040] Figure 7 is a schematic diagram of the structure of a control device provided in an embodiment of the present application;

[0041] Figure 8 It is a structural schematic diagram of an impulse signal detection model training device provided in an embodiment of the present application;

[0042] Fig. 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0043] Fig.10 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0045] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0046] Microphones are hardware used in electronic devices to collect sound signals in the environment. When a user taps the sound hole of the microphone in an electronic device with a finger, in addition to the sound produced by the tapping itself, a unique sound signal is also generated due to the compression and rebound of the air in the sound cavity. By detecting the sound signal, it is possible to detect the user's behavior of tapping the sound hole of the microphone by detecting the sound signal collected by the microphone, and then perform the control operation corresponding to the behavior.

[0047] Analysis of the sound signal generated by tapping the microphone outlet hole found that tapping the microphone outlet hole will generate a short impulse signal with an amplitude significantly higher than the ambient sound. In the time domain, the sound signal generated by tapping the microphone outlet hole each time will show a pattern of first high, then low, and then back to half high. Figure 1 As shown, Figure 1 : is a schematic diagram of a time domain signal corresponding to a sound signal generated by tapping a microphone sound outlet provided in an embodiment of the present application. Figure 1The time domain signal corresponding to the sound signal generated by knocking on the microphone outlet three times is shown in the figure. In the frequency domain, the impulse signal has a strong power distribution from 0 to 8 kHz, with stronger low frequencies and gradually weakening high frequencies, and presents a regular group of resonance peaks. The fundamental frequency is generally below 100 Hz, and shows an obvious tailing effect. Figure 2 As shown, Figure 2 : is a schematic diagram of a frequency domain signal corresponding to a sound signal generated by tapping the sound outlet of a microphone provided in an embodiment of the present application. Figure 2 The frequency domain signal corresponding to the sound signal generated by knocking the microphone sound hole three times is shown in FIG. Therefore, the sound signal generated by knocking the microphone sound hole will form a unique sound feature in both the time domain and the frequency domain. The sound feature can effectively distinguish the sound signal generated by knocking the microphone sound hole from other sound signals. By detecting the sound signal collected by the microphone, it is feasible to obtain the user's behavior of knocking the microphone sound hole and then perform corresponding control operations. For example, Figure 3 As shown, Figure 3 It is a schematic diagram of an audio signal of knocking on the microphone sound outlet three times provided in an embodiment of the present application. Figure 3 The audio signal shown includes three impulse signals, which are impulse signal 31, impulse signal 32 and impulse signal 33 respectively.

[0048] The control, model training method, device, equipment and medium provided in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings through specific embodiments and their application scenarios.

[0049] Figure 4 : is a flow chart of a control method provided in an embodiment of the present application. The control method may include:

[0050] Step 401: Acquire a first sound signal collected by a microphone;

[0051] Step 402: inputting the first sound signal into an impulse signal detection model to obtain characteristic information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model;

[0052] In some possible implementations of the embodiments of the present application, the characteristic information of the impulse signal in the embodiments of the present application includes but is not limited to: the number of impulse signals, the time interval between two adjacent impulse signals, the time when the finger touches the microphone sound outlet each time, and the time when the finger leaves the microphone sound outlet each time, etc.

[0053] Step 403: determining, based on the characteristic information, a tapping operation of the user on the microphone sound outlet;

[0054] Exemplarily, assuming that after the first sound signal collected by the microphone is output to the impulse signal detection model, the impulse signal detection model outputs that the number of impulse signals generated by the user knocking on the microphone sound hole is 0, it is determined that the user did not knock on the microphone sound hole.

[0055] As another example, assuming that after the first sound signal collected by the microphone is output to the impulse signal detection model, the impulse signal detection model outputs that the number of impulse signals formed by the user tapping the microphone sound outlet is 1, it is determined that the user clicks the microphone sound outlet.

[0056] As another example, assuming that after the first sound signal collected by the microphone is output to the impulse signal detection model, the impulse signal detection model outputs the first sound signal including 2 impulse signals formed by the user tapping the microphone sound hole and the time interval between the two impulse signals is less than the preset duration, then it is determined that the user double-clicks the microphone sound hole.

[0057] As another example, assuming that after the first sound signal collected by the microphone is output to the impulse signal detection model, the impulse signal detection model outputs the first sound signal including 2 impulse signals formed by the user tapping the microphone sound hole and the time interval between the two impulse signals is greater than the preset duration, then it is determined that the user clicks the microphone sound hole twice.

[0058] As another example, assuming that after the first sound signal collected by the microphone is output to the impulse signal detection model, the impulse signal detection model outputs the first sound signal including that the number of impulse signals formed by the user tapping the microphone sound outlet is greater than or equal to 3 and the time interval between any two adjacent impulse signals is less than the preset duration, it is determined that the user taps the microphone sound outlet continuously.

[0059] Step 404: Execute a control operation corresponding to the tapping operation.

[0060] In some possible implementations of the embodiments of the present application, different control operations can be pre-set for different tapping operations. For example, the user single-clicks the microphone sound hole to open the camera, the user double-clicks the microphone sound hole to open the file manager, the user repeatedly taps the microphone sound hole to clean up garbage on the electronic device, and so on.

[0061] In the embodiment of the present application, the first sound signal collected by the microphone is obtained; the first sound signal is input into the impulse signal detection model to obtain characteristic information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model; according to the characteristic information, the user's tapping operation on the microphone sound outlet is determined; and the control operation corresponding to the tapping operation is executed. In this way, the electronic device can be controlled by the user's tapping operation on the microphone sound outlet, which can meet the user's control needs for the electronic device.

[0062] In some possible implementations of the embodiments of the present application, the electronic device may be in a complex environment, and various noises may generate signals similar to impulse signals, resulting in erroneous execution of control operations and affecting the user's use of the electronic device. Based on this, the sound signals collected by the two microphones can be detected, wherein the sound outlets of the two microphones are arranged on the electronic device with a certain distance. When the sound signals collected by the two microphones are respectively input into the impulse signal detection model and the characteristic information of the impulse signals can be obtained, since the user cannot knock on the two microphone sound outlets at the same time, at this time, it is considered that the impulse signal is generated by the ambient sound, not by the user knocking on the microphone sound outlet, and the corresponding control operation is not performed.

[0063] Based on the above, step 401 may include obtaining a second sound signal collected by the first microphone and obtaining a third sound signal collected by the second microphone, wherein the first sound signal includes the second sound signal and the third sound signal; step 402 may include inputting the second sound signal and the third sound signal into an impulse signal detection model respectively; step 403 may include: when the second sound signal is input into the impulse signal detection model to obtain the first characteristic information of the impulse signal, and the characteristic information of the impulse signal is not obtained by inputting the third sound signal into the impulse signal detection model, determining the tapping operation according to the first characteristic information; or, when the second sound signal is input into the impulse signal detection model to obtain the characteristic information of the impulse signal, and the third sound signal is input into the impulse signal detection model to obtain the second characteristic information of the impulse signal, determining the tapping operation according to the second characteristic information.

[0064] In some possible implementations of the embodiments of the present application, the second sound signal may be first input into the impulse signal detection model, and then the third sound signal may be input into the impulse signal detection model; or, the third sound signal may be first input into the impulse signal detection model, and then the second sound signal may be input into the impulse signal detection model.

[0065] Exemplarily, when the second sound signal collected by the first microphone is input into the impulse signal detection model, the number of impulse signals output by the impulse signal detection model is 0, that is, the impulse signal detection model does not detect the impulse signal in the second sound signal; when the third sound signal collected by the second microphone is input into the impulse signal detection model, the number of impulse signals output by the impulse signal detection model is 1, determining that the user clicks the sound outlet of the second microphone.

[0066] As another example, when the third sound signal collected by the second microphone is input into the impulse signal detection model, the number of impulse signals output by the impulse signal detection model is 1; when the second sound signal collected by the first microphone is input into the impulse signal detection model, the number of impulse signals output by the impulse signal detection model is 0, determining that the user clicks the sound outlet of the second microphone.

[0067] As another example, when the second sound signal collected by the first microphone is input into the impulse signal detection model, the impulse signal detection model outputs 2 impulse signals, and the interval between the two impulse signals is 0.3 seconds; the third sound signal collected by the second microphone is input into the impulse signal detection model, and the impulse signal detection model outputs 1 impulse signals; that is, an impulse signal is detected from the second sound signal collected by the first microphone and from the third sound signal collected by the second microphone, and it is determined that the impulse signal is caused by ambient sound, and is not generated by the user knocking on the microphone sound outlet.

[0068] In the embodiment of the present application, it is possible to detect that the impulse signal is generated by the ambient sound, thereby avoiding erroneous execution of operations due to the ambient sound.

[0069] In some possible implementations of the embodiments of the present application, different users have different habits of operating electronic devices. When determining the user's tapping operation on the microphone sound outlet, the same preset duration is used, and it is possible to perform corresponding operations on the behavior of tapping the microphone sound outlet of a person who is not the owner of the electronic device, resulting in erroneous execution of the operation. Based on this, step 403 may include: when the number of impulse signals is greater than or equal to 3 and the time interval between any two adjacent impulse signals is within a first time interval range, determining the user's tapping operation on the microphone sound outlet according to the characteristic information; wherein the first time interval range is pre-determined based on the sound signal generated by the user's historical tapping of the microphone sound outlet.

[0070] In some possible implementations of the embodiments of the present application, for continuously tapping the microphone sound outlet, the time interval of the impulse signal in the sound signal of the owner of the electronic device continuously tapping the microphone sound outlet can be obtained in advance to obtain a time interval set, wherein the time interval set includes the time interval of the impulse signal in the sound signal of the owner of the electronic device continuously tapping the microphone sound outlet, and the minimum time interval in the time interval set to the maximum time interval in the time interval set is used as the first time interval range.

[0071] When the number of impulse signals is greater than or equal to 3 but there is at least one time interval between two adjacent impulse signals that is not within the first time interval range, it is determined that the consecutive tapping of the microphone outlet hole is not operated by the owner of the electronic device.

[0072] When the number of impulse signals is greater than or equal to 3 and the time interval between any two adjacent impulse signals is within the first time interval range, it is determined that the continuous tapping of the microphone sound outlet is operated by the owner of the electronic device, and then based on the characteristic information of the impulse signal, it is determined that the owner of the electronic device has performed a tapping operation on the microphone sound outlet.

[0073] In the embodiment of the present application, it is possible to avoid executing corresponding operations on the behavior of a user other than the owner of the electronic device knocking on the microphone sound outlet.

[0074] In some possible implementations of the embodiments of the present application, the characteristic information includes a first time of multiple touches on the microphone sound outlet hole and a second time of multiple separations from the microphone sound outlet hole; accordingly, step 403 may include: when the time difference between the second time of separation from the microphone sound outlet hole for the i-th time and the first time of touching the microphone sound outlet hole for the i-th time is greater than or equal to a first threshold, determining that the user's tapping operation on the microphone sound outlet hole within a first duration is a long press of the microphone sound outlet hole; wherein i is a positive integer, the i-th time is any one of the multiple times, and the first duration is the duration from the first time of the i-th touching the microphone sound outlet hole to the second time of separation from the microphone sound outlet hole for the i-th time.

[0075] In some possible implementations of the present application, by analyzing the impulse signal in the sound signal generated by the user tapping the microphone sound outlet, it can be found that the first of the two peaks corresponding to the impulse signal in the sound signal generated by each tapping of the microphone sound outlet is generated by the finger touching the microphone sound outlet, and the second of the two peaks is generated by the finger leaving the microphone sound outlet, and the microphone detects the ambient sound. Since there is almost no mid-high frequency signal in the sound from the time when the finger touches the microphone sound outlet to the time when the finger leaves the microphone sound outlet, that is, during the period when the finger presses the microphone sound outlet, this section of sound is called the silent frequency area; the ambient sound collected by the microphone after the finger leaves the microphone sound outlet is called the noise frequency area, and the finger touching the microphone sound outlet and the finger leaving the microphone sound outlet can be distinguished by the order of appearance of the silent frequency area and the noise frequency area. When the noise frequency area is before the silent frequency area, the operation at the separation of the noise frequency area and the silent frequency area is determined as the finger touching the microphone sound outlet, and when the noise frequency area is after the silent frequency area, the operation at the separation of the noise frequency area and the silent frequency area is determined as the finger leaving the microphone sound outlet. For example, Figure 5 As shown, Figure 5 Schematic diagram of the audio signal of a finger touching and leaving the microphone sound outlet provided in an embodiment of the present application. Figure 5 The “press down” in the picture means that your finger touches the microphone sound hole, and the “release” means that your finger leaves the microphone sound hole.

[0076] It is understandable that the finger touching the microphone sound outlet and the finger leaving the microphone sound outlet appear in pairs one after another.

[0077] After the first sound signal collected by the microphone is input into the impulse signal detection model, the impulse signal detection model outputs the time when the finger touches the microphone sound outlet hole and the time when the finger leaves the microphone sound outlet hole.

[0078] Exemplarily, the time when the finger touches the microphone sound outlet for the i-th time is recorded as Tai, and the time when the finger leaves the microphone sound outlet for the i-th time is recorded as Tbi. Then the time from the i-th touching the microphone sound outlet to the i-th leaving the microphone sound outlet is Tbi-Tai, where i is a positive integer.

[0079] When the time duration Tbi-Tai from the i-th touching of the microphone sound outlet hole to the i-th leaving of the microphone sound outlet hole is greater than or equal to a first threshold, it is determined that the user's behavior from the i-th touching of the microphone sound outlet hole to the i-th leaving of the microphone sound outlet hole is long pressing the microphone sound outlet hole, and then an operation corresponding to long pressing the microphone sound outlet hole is performed, for example, restarting the electronic device.

[0080] In the embodiment of the present application, it is possible to identify the operation of the user long pressing the microphone sound outlet hole, and then execute the control operation corresponding to the user long pressing the microphone sound outlet hole.

[0081] In some possible implementations of the embodiments of the present application, the various thresholds in the embodiments of the present application can be set according to actual needs.

[0082] The present application also provides a method for training an impulse signal detection model. Figure 6 As shown, Figure 6 : is a flow chart of the impulse signal detection model training method provided in the embodiment of the present application. The impulse signal detection model training method comprises the following steps:

[0083] Step 601: Acquire a fourth sound signal caused by a user tapping the microphone sound outlet hole multiple times;

[0084] Step 602: using the fourth sound signal as sample data to train an impulse signal detection model until a first condition is met; wherein the impulse signal detection model is used to detect an impulse signal in a first sound signal input to the impulse signal detection model, and the first sound signal is a sound signal collected by a microphone;

[0085] The first condition includes at least one of the following:

[0086] The loss value of the loss function of the impulse signal detection model reaches the minimum value;

[0087] The false rejection rate of the impulse signal detection model is less than a second threshold;

[0088] The false acceptance rate of the impulse signal detection model is less than a third threshold.

[0089] In some possible implementations of the embodiments of the present application, sound signals of the user repeatedly tapping the microphone sound hole can be collected in advance and annotated, and the time domain features and frequency domain features are extracted from these sound signals using short-time Fourier transform (STFT) or Mel-Frequency Cepstral Coefficients (MFCC). These features are then preprocessed, for example, normalized, denoised, smoothed, etc. The annotated audio signals of the user repeatedly tapping the microphone sound hole are input into a neural network model to train the neural network model, and the neural network model can be a Gaussian Mixture Model (GMM), a deep learning model, etc., wherein the deep learning model includes but is not limited to a convolutional neural network (CNN), a recurrent neural network (RNN), etc., until the loss value of the loss function of the neural network model reaches a minimum value or the false rejection rate (FRR) of the neural network model is less than a second threshold; the false acceptance rate (FRR) of the neural network model is less than a second threshold; When the user taps the microphone sound outlet hole, the impulse signal detection model is obtained. The input of the impulse signal detection model can be: a sound signal, and the output can be: the sound signal includes the number of impulse signals formed by the user tapping the microphone sound outlet hole, and the time interval between two adjacent impulse signals.

[0090] After the first sound signal collected by the microphone is output to the impulse signal detection model, the impulse signal detection model outputs the first sound signal including the number of impulse signals generated by the user tapping the microphone sound outlet and the time interval between two adjacent impulse signals.

[0091] In the embodiment of the present application, the impulse signal detection model can identify the user's tapping operation on the microphone sound outlet, and then perform the control operation corresponding to the tapping operation. The electronic device can be controlled by the user's tapping operation on the microphone sound outlet, meeting the user's control needs for the electronic device.

[0092] In some possible implementations of the embodiments of the present application, a sound signal of a user tapping the microphone sound outlet hole multiple times can be obtained, and the finger touching the microphone sound outlet hole and the finger leaving the microphone sound outlet hole in the sound signal are marked. The impulse signal detection model is trained using the marked sound signal. The input of the impulse signal detection model can be: a sound signal, and the output can be: the time point when the finger touches the microphone sound outlet hole and the time point when the finger leaves the microphone sound outlet hole in the sound signal.

[0093] It is understandable that the finger touching the microphone sound outlet and the finger leaving the microphone sound outlet appear in pairs one after another.

[0094] After the first sound signal collected by the microphone is input into the impulse signal detection model, the impulse signal detection model outputs the time when the finger touches the microphone sound outlet and the time when the finger leaves the microphone sound outlet. It can be understood that the finger touching the microphone sound outlet and the finger leaving the microphone sound outlet appear in pairs. The time length from the i-th touch of the microphone sound outlet to the i-th leaving of the microphone sound outlet can determine whether the user's behavior from the i-th touch of the microphone sound outlet to the i-th leaving of the microphone sound outlet is a long press of the microphone sound outlet.

[0095] In the embodiment of the present application, it is possible to identify the operation of the user long pressing the microphone sound outlet hole, and then execute the control operation corresponding to the user long pressing the microphone sound outlet hole.

[0096] In the control method provided in the embodiment of the present application, the execution subject may be a control device. In the embodiment of the present application, the control device provided in the embodiment of the present application is described by taking the control method executed by the control device as an example.

[0097] Figure 7 700 is a schematic diagram of the structure of a control device provided in an embodiment of the present application. The control device 700 may include:

[0098] A first acquisition module 701 is used to acquire a first sound signal collected by a microphone;

[0099] The feature information extraction module 702 is used to input the first sound signal into the impulse signal detection model to obtain feature information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model;

[0100] A determination module 703 is used to determine a tapping operation of the user on the microphone sound outlet according to the feature information;

[0101] The execution module 704 is used to execute the control operation corresponding to the tapping operation.

[0102] In the embodiment of the present application, the first sound signal collected by the microphone is obtained; the first sound signal is input into the impulse signal detection model to obtain characteristic information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model; according to the characteristic information, the user's tapping operation on the microphone sound outlet is determined; and the control operation corresponding to the tapping operation is executed. In this way, the electronic device can be controlled by the user's tapping operation on the microphone sound outlet, which can meet the user's control needs for the electronic device.

[0103] In some possible implementations of the embodiments of the present application, the first acquisition module 701 is specifically used to:

[0104] Acquire a second sound signal collected by the first microphone and acquire a third sound signal collected by the second microphone, wherein the first sound signal includes the second sound signal and the third sound signal;

[0105] Accordingly, the feature information extraction module 702 is specifically used to: input the second sound signal and the third sound signal into the impulse signal detection model respectively;

[0106] The determination module 703 is specifically used for: determining a tapping operation based on the first characteristic information when the second sound signal is input into the impulse signal detection model to obtain the first characteristic information of the impulse signal, and the third sound signal is input into the impulse signal detection model to obtain no characteristic information of the impulse signal; or determining a tapping operation based on the second characteristic information when the second sound signal is input into the impulse signal detection model to obtain no characteristic information of the impulse signal, and the third sound signal is input into the impulse signal detection model to obtain second characteristic information of the impulse signal.

[0107] In the embodiment of the present application, it is possible to detect that the impulse signal is generated by the ambient sound, thereby avoiding erroneous execution of operations due to the ambient sound.

[0108] In some possible implementations of the embodiment of the present application, the characteristic information includes the number of impulse signals and the time interval between two adjacent impulse signals; the determination module 703 is specifically used to:

[0109] When the number of impulse signals is greater than or equal to 3 and the time interval between any two adjacent impulse signals is within a first time interval range, the user's tapping operation on the microphone sound hole is determined based on the characteristic information; wherein the first time interval range is determined in advance based on the sound signal generated by the user's historical tapping of the microphone sound hole.

[0110] In the embodiment of the present application, it is possible to avoid executing corresponding operations on the behavior of a user other than the owner of the electronic device knocking on the microphone sound outlet.

[0111] In some possible implementations of the embodiments of the present application, the characteristic information includes a first time of multiple touching of the microphone sound outlet hole and a second time of multiple separations from the microphone sound outlet hole; the determination module 703 is specifically used to: when the time difference between the second time of separation from the microphone sound outlet hole for the i-th time and the first time of touching the microphone sound outlet hole for the i-th time is greater than or equal to a first threshold, determine that the user's tapping operation on the microphone sound outlet hole within a first time length is a long press of the microphone sound outlet hole; wherein i is a positive integer, the i-th time is any one of the multiple times, and the first time length is the time length from the first time of the i-th touching of the microphone sound outlet hole to the second time of separation from the microphone sound outlet hole for the i-th time.

[0112] In the embodiment of the present application, it is possible to identify the operation of the user long pressing the microphone sound outlet hole, and then execute the control operation corresponding to the user long pressing the microphone sound outlet hole.

[0113] The control device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0114] The control device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0115] The control device provided in the embodiment of the present application can achieve Figures 4 to 5 To avoid repetition, the various processes implemented in the control method embodiment will not be described here.

[0116] The impulse signal detection model training method provided in the embodiment of the present application can be executed by an impulse signal detection model training device. In the embodiment of the present application, the impulse signal detection model training method is executed by an impulse signal detection model training device as an example to illustrate the impulse signal detection model training device provided in the embodiment of the present application.

[0117] Figure 8 800 is a schematic diagram of the structure of an impulse signal detection model training device provided in an embodiment of the present application. The impulse signal detection model training device 800 may include:

[0118] The second acquisition module 801 is used to acquire a fourth sound signal caused by the user tapping the microphone sound outlet hole multiple times;

[0119] A training module 802 is used to train an impulse signal detection model using the fourth sound signal as sample data until a first condition is met; wherein the impulse signal detection model is used to detect an impulse signal in a first sound signal input to the impulse signal detection model, and the first sound signal is a sound signal collected by a microphone;

[0120] The first condition includes at least one of the following:

[0121] The loss value of the loss function of the impulse signal detection model reaches the minimum value;

[0122] The false rejection rate of the impulse signal detection model is less than a second threshold;

[0123] The false acceptance rate of the impulse signal detection model is less than a third threshold.

[0124] In the embodiment of the present application, the impulse signal detection model can identify the user's tapping operation on the microphone sound outlet, and then perform the control operation corresponding to the tapping operation. The electronic device can be controlled by the user's tapping operation on the microphone sound outlet, meeting the user's control needs for the electronic device.

[0125] The impulse signal detection model training device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a MID, an AR / VR device, a robot, a wearable device, a UMPC, a netbook or a PDA, etc., and can also be a server, a NAS, a PC, a TV, a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0126] The impulse signal detection model training device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0127] The impulse signal detection model training device provided in the embodiment of the present application can achieve Figure 6 To avoid repetition, the various processes implemented in the embodiment of the impulse signal detection model training method are not described here.

[0128] Alternatively, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores a program or instruction that can be executed on the processor 901, and when the program or instruction is executed by the processor 901, the various steps of the control method embodiment provided in the embodiment of the present application or the various steps of the impulse signal detection model training method embodiment provided in the embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0129] Fig.10 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0130] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0131] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0132] Among them, the processor 1010 is used to: obtain a first sound signal collected by a microphone; input the first sound signal into an impulse signal detection model to obtain characteristic information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model; determine, based on the characteristic information, a user's tapping operation on the microphone outlet; and execute a control operation corresponding to the tapping operation.

[0133] In the embodiment of the present application, the first sound signal collected by the microphone is obtained; the first sound signal is input into the impulse signal detection model to obtain characteristic information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model; according to the characteristic information, the user's tapping operation on the microphone sound outlet is determined; and the control operation corresponding to the tapping operation is executed. In this way, the electronic device can be controlled by the user's tapping operation on the microphone sound outlet, which can meet the user's control needs for the electronic device.

[0134] In some possible implementations of the embodiments of the present application, the processor 1010 is specifically configured to:

[0135] Acquire a second sound signal collected by a first microphone and acquire a third sound signal collected by a second microphone, wherein the first sound signal includes a second sound signal and a third sound signal; input the second sound signal and the third sound signal into an impulse signal detection model respectively; when the first characteristic information of the impulse signal is obtained by inputting the second sound signal into the impulse signal detection model, and the characteristic information of the impulse signal is not obtained by inputting the third sound signal into the impulse signal detection model, determine a tapping operation according to the first characteristic information; or, when the second sound signal is input into the impulse signal detection model, and the characteristic information of the impulse signal is not obtained, and the second characteristic information of the impulse signal is obtained by inputting the third sound signal into the impulse signal detection model, determine a tapping operation according to the second characteristic information.

[0136] In the embodiment of the present application, it is possible to detect that the impulse signal is generated by the ambient sound, thereby avoiding erroneous execution of operations due to the ambient sound.

[0137] In some possible implementations of the embodiments of the present application, the characteristic information includes the number of impulse signals and the time interval between two adjacent impulse signals; the processor 1010 is specifically configured to:

[0138] When the number of impulse signals is greater than or equal to 3 and the time interval between any two adjacent impulse signals is within a first time interval range, the user's tapping operation on the microphone sound hole is determined based on the characteristic information; wherein the first time interval range is determined in advance based on the sound signal generated by the user's historical tapping of the microphone sound hole.

[0139] In the embodiment of the present application, it is possible to avoid executing corresponding operations on the behavior of a user other than the owner of the electronic device knocking on the microphone sound outlet.

[0140] In some possible implementations of the embodiments of the present application, the characteristic information includes a first time of multiple touches on the microphone sound outlet and a second time of multiple detachments from the microphone sound outlet; the processor 1010 is specifically configured to:

[0141] When the time difference between the second time of the i-th time of leaving the microphone sound outlet hole and the first time of the i-th time of touching the microphone sound outlet hole is greater than or equal to the first threshold, it is determined that the user's tapping operation on the microphone sound outlet hole within the first time length is a long press of the microphone sound outlet hole; wherein i is a positive integer, the i-th time is any one of multiple times, and the first time length is the time length from the first time of the i-th time of touching the microphone sound outlet hole to the second time of leaving the microphone sound outlet hole for the i-th time.

[0142] In the embodiment of the present application, it is possible to identify the operation of the user long pressing the microphone sound outlet hole, and then execute the control operation corresponding to the user long pressing the microphone sound outlet hole.

[0143] In some possible implementations of the embodiments of the present application, the processor 1010 is further configured to:

[0144] Obtain a fourth sound signal generated by the user repeatedly tapping the microphone sound outlet hole; use the fourth sound signal as sample data to train an impulse signal detection model until the first condition is met; wherein the impulse signal detection model is used to detect an impulse signal in a first sound signal input to the impulse signal detection model, where the first sound signal is a sound signal collected by the microphone;

[0145] The first condition includes at least one of the following:

[0146] The loss value of the loss function of the impulse signal detection model reaches the minimum value;

[0147] The false rejection rate of the impulse signal detection model is less than a second threshold;

[0148] The false acceptance rate of the impulse signal detection model is less than a third threshold.

[0149] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0150] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0151] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0152] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the control method embodiment provided in the embodiment of the present application or each process of the impulse signal detection model training method embodiment provided in the embodiment of the present application is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0153] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, and examples of computer-readable storage media include non-transitory computer-readable media, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0154] An embodiment of the present application also provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the various processes of the control method embodiment provided in the embodiment of the present application or the various processes of the impulse signal detection model training method embodiment provided in the embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0155] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0156] The embodiment of the present application also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the control method embodiment provided by the embodiment of the present application or the various processes of the impulse signal detection model training method embodiment provided by the embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0157] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0159] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A control method, characterized in that: The method comprises: Acquire a first sound signal collected by a microphone; Inputting the first sound signal into an impulse signal detection model to obtain characteristic information of the impulse signal, wherein the impulse signal detection model is used to detect the impulse signal in the sound signal input into the impulse signal detection model; Determining, according to the characteristic information, a tapping operation of the user on the microphone sound outlet; Execute a control operation corresponding to the tapping operation.

2. The method according to claim 1, characterized in that The step of obtaining a first sound signal collected by a microphone includes: Acquire a second sound signal collected by the first microphone and acquire a third sound signal collected by the second microphone, wherein the first sound signal includes the second sound signal and the third sound signal; The step of inputting the first sound signal into an impulse signal detection model to obtain characteristic information of the impulse signal includes: inputting the second sound signal and the third sound signal into the impulse signal detection model respectively; The step of determining, according to the characteristic information, a tapping operation of the user on the sound outlet of the microphone includes: In a case where the first characteristic information of the impulse signal is obtained by inputting the second sound signal into the impulse signal detection model, and the characteristic information of the impulse signal is not obtained by inputting the third sound signal into the impulse signal detection model, determining the tapping operation according to the first characteristic information; or, When the second sound signal is input into the impulse signal detection model and no characteristic information of the impulse signal is obtained, and the third sound signal is input into the impulse signal detection model and second characteristic information of the impulse signal is obtained, the tapping operation is determined according to the second characteristic information.

3. The method according to claim 1, characterized in that The characteristic information includes the number of impulse signals and the time interval between two adjacent impulse signals; The step of determining, according to the characteristic information, a tapping operation of the user on the sound outlet of the microphone includes: When the number of the impulse signals is greater than or equal to 3 and the time interval between any two adjacent impulse signals is within the first time interval range, determining, according to the characteristic information, that the user has performed a tapping operation on the microphone sound outlet; The first time interval range is determined according to the sound signal generated by the user tapping the microphone sound outlet hole in the past.

4. The method according to claim 1, characterized in that The characteristic information includes a first time of multiple touches on the microphone sound outlet and a second time of multiple detachments from the microphone sound outlet; The step of determining, according to the characteristic information, a tapping operation of the user on the sound outlet of the microphone includes: When the time difference between the second time of the i-th time of leaving the microphone sound outlet hole and the first time of the i-th time of touching the microphone sound outlet hole is greater than or equal to the first threshold, determining that the tapping operation of the user on the microphone sound outlet hole within the first time period is a long press of the microphone sound outlet hole; Here, i is a positive integer, and the first duration is the duration from a first time when the microphone sound outlet is touched for the i-th time to a second time when the microphone sound outlet is separated for the i-th time.

5. A method for training an impulse signal detection model, characterized in that: The method comprises: Acquire a fourth sound signal of the user tapping the microphone sound outlet hole multiple times; Using the fourth sound signal as sample data to train an impulse signal detection model until a first condition is met; wherein the impulse signal detection model is used to detect an impulse signal in a first sound signal input to the impulse signal detection model, wherein the first sound signal is a sound signal collected by a microphone; The first condition includes at least one of the following: The loss value of the loss function of the impulse signal detection model reaches a minimum value; The false rejection rate of the impulse signal detection model is less than a second threshold; The false acceptance rate of the impulse signal detection model is less than a third threshold.

6. A control device, characterized in that: The device comprises A first acquisition module, used to acquire a first sound signal collected by a microphone; a feature information extraction module, configured to input the first sound signal into an impulse signal detection model to obtain feature information of the impulse signal, wherein the impulse signal detection model is configured to detect an impulse signal in the sound signal input into the impulse signal detection model; A determination module, used to determine a tapping operation of the user on the microphone sound outlet according to the feature information; The execution module is used to execute the control operation corresponding to the tapping operation.

7. The device according to claim 6, characterized in that The first acquisition module is specifically used for: Acquire a second sound signal collected by the first microphone and acquire a third sound signal collected by the second microphone, wherein the first sound signal includes the second sound signal and the third sound signal; The feature information extraction module is specifically used for: inputting the second sound signal and the third sound signal into the impulse signal detection model respectively; The determination module is specifically used for: In a case where the first characteristic information of the impulse signal is obtained by inputting the second sound signal into the impulse signal detection model, and the characteristic information of the impulse signal is not obtained by inputting the third sound signal into the impulse signal detection model, determining the tapping operation according to the first characteristic information; or, When the second sound signal is input into the impulse signal detection model and no characteristic information of the impulse signal is obtained, and the third sound signal is input into the impulse signal detection model and second characteristic information of the impulse signal is obtained, the tapping operation is determined according to the second characteristic information.

8. An impulse signal detection model training device, characterized in that: The device comprises: A second acquisition module is used to acquire a fourth sound signal caused by the user tapping the microphone sound outlet hole multiple times; A training module, used for training an impulse signal detection model using the fourth sound signal as sample data until a first condition is met; wherein the impulse signal detection model is used for detecting an impulse signal in a first sound signal input to the impulse signal detection model, wherein the first sound signal is a sound signal collected by a microphone; The first condition includes at least one of the following: The loss value of the loss function of the impulse signal detection model reaches a minimum value; The false rejection rate of the impulse signal detection model is less than a second threshold; The false acceptance rate of the impulse signal detection model is less than a third threshold.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the control method according to any one of claims 1 to 4 or the steps of the impulse signal detection model training method according to claim 5 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the control method according to any one of claims 1 to 4 or the steps of the impulse signal detection model training method according to claim 5 are implemented.

Images