Interaction method and device, electronic equipment and storage medium

Abstract

The invention discloses an interaction method and device, electronic equipment and a storage medium. The interaction method comprises the steps that first motion data of the electronic equipment and collected first audio data are acquired; performing knock detection based on the first motion data and the first audio data to obtain a knock detection result; and executing a corresponding interaction operation according to the knocking detection result. According to the method, the knocking of the electronic equipment can be accurately detected, the interactive operation is prevented from being mistakenly triggered, and the accuracy of executing the interactive operation is improved.

Description

Technical Field

[0001] The present application relates to the technical field of electronic devices, and more specifically, to an interaction method, device, electronic device and storage medium. Background Art

[0002] With the rapid advancement of technology and living standards, electronic devices (such as smart phones, tablet computers, etc.) have become one of the commonly used electronic products in people's lives. Usually, when using electronic devices, they usually interact with the electronic devices through buttons, but in some cases it is inconvenient for users to operate through buttons, so there is an interactive method of interacting by tapping the electronic devices. However, when interacting by tapping the electronic devices, it is easy to misdetect the tapping event and mistrigger the interactive operation, thereby affecting the accuracy of the electronic device in performing the interactive operation. Summary of the invention

[0003] The present application proposes an interaction method, device, electronic device and storage medium, which can accurately detect tapping on an electronic device, thereby avoiding false triggering of interactive operations and improving the accuracy of the electronic device in performing interactive operations.

[0004] In a first aspect, an embodiment of the present application provides an interaction method, which is applied to an electronic device, and the method includes: obtaining first motion data and collected first audio data of the electronic device; performing knock detection based on the first motion data and the first audio data to obtain a knock detection result; and performing a corresponding interaction operation according to the knock detection result.

[0005] In a second aspect, an embodiment of the present application provides an interactive device, which is applied to an electronic device, and the device includes: a data acquisition module, a tap recognition module and an interactive execution module, wherein the data acquisition module is used to acquire first motion data and collected first audio data of the electronic device; the tap recognition module is used to perform tap detection based on the first motion data and the first audio data to obtain a tap detection result; and the interactive execution module is used to perform corresponding interactive operations according to the tap detection result.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the interaction method provided in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the interaction method provided in the first aspect above.

[0008] The solution provided by the present application obtains the first motion data of the electronic device and the first audio data collected; performs tapping detection based on the first motion data and the first audio data to obtain a tapping detection result; and performs a corresponding interactive operation according to the tapping detection result. Since the electronic device not only uses the motion data for tapping detection, but also combines the collected audio data, the accuracy of the tapping detection can be improved, thereby avoiding the false triggering of the interactive operation and improving the accuracy of the electronic device in performing the interactive operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram of an application environment provided by an embodiment of the present application is shown.

[0011] Figure 2 A flow chart of an interaction method according to an embodiment of the present application is shown.

[0012] Figure 3 A schematic diagram of the change of acceleration data provided in an embodiment of the present application is shown.

[0013] Figure 4 A flow chart of an interaction method according to another embodiment of the present application is shown.

[0014] Figure 5 A flow chart of an interaction method according to yet another embodiment of the present application is shown.

[0015] Figure 6 A schematic diagram of the change of acceleration data provided in an embodiment of the present application is shown.

[0016] Figure 7 A schematic diagram of the change of acceleration data provided in an embodiment of the present application is shown.

[0017] Figure 8 A schematic diagram of the change of acceleration data provided in an embodiment of the present application is shown.

[0018] Fig. 9 A schematic diagram showing the principle of a knock detection model provided in an embodiment of the present application is shown.

[0019] Fig.10 A flow chart of an interaction method according to yet another embodiment of the present application is shown.

[0020] Fig.11 A flow chart of an interaction method according to yet another embodiment of the present application is shown.

[0021] Fig.12 A block diagram of an interaction device according to an embodiment of the present application is shown.

[0022] Fig.13 It is a block diagram of an electronic device according to an embodiment of the present application for executing the interaction method according to an embodiment of the present application.

[0023] Fig.14 It is a storage unit of an embodiment of the present application for storing or carrying program codes for implementing the interaction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0025] Usually, when people use electronic devices, they can interact with the electronic devices by operating the buttons displayed on the screen of the electronic device or the physical buttons set on the electronic device. With the development of electronic devices, there are fewer and fewer physical buttons on electronic devices, which makes the main component for triggering interactive operations change from buttons to touch screens. However, when users actually interact with electronic devices, in some cases, they cannot interact with virtual buttons or touch operations more conveniently. For example, when users hold the electronic device with one hand, they cannot quickly complete some interactions through buttons or touch; and when some interactive operations need to be triggered, cumbersome operations are usually required to complete the interaction. For example, when a QR code needs to be scanned through an electronic device, if the application currently running in the foreground is not an application that can scan QR codes, it is necessary to switch to the main interface to start the application that can scan QR codes, and then after starting the application, trigger the QR code scanning in the application interface. The operation during the entire interactive process is relatively cumbersome, which affects the user experience.

[0026] Therefore, in current electronic equipment products, new interaction methods have emerged, which involve tapping the back of an electronic device or the side of an electronic device. For example, when an electronic device detects that a user has triggered a tapping operation on the back cover of a smartphone, the smartphone executes a preset interactive function, such as triggering a QR code scanning function of a target application. This interactive method can quickly trigger preset interactive operations, thereby greatly improving the user experience. However, in the related art, tapping detection for electronic devices is mainly implemented through motion sensors, and its usage scenario requirements are relatively harsh, requiring the product to be hit with a small area and a light weight, and unconventional operations such as when the electronic device is shaken may also be detected as tapping events. Therefore, the detection accuracy of tapping on electronic devices in the related art is insufficient, and interactive operations are easily triggered by mistake.

[0027] In response to the above problems, the inventors have proposed the interactive method, device, electronic device and storage medium provided in the embodiments of the present application. On the basis of using motion data for tapping detection, the collected audio data is also combined, so that the accuracy of tapping detection can be improved, thereby avoiding false triggering of interactive operations and improving the accuracy of electronic devices in performing interactive operations. Among them, the specific interactive method is described in detail in the subsequent embodiments.

[0028] The following first introduces the hardware environment involved in the control method provided in the embodiment of the present application.

[0029] See also Figure 1 , Figure 1 The control method provided in the embodiment of the present application is applied to an electronic device 100, and the electronic device 100 includes a motion sensor 101 and an audio acquisition device 102. The motion sensor 101 is used to collect motion data of the electronic device 100, and the motion sensor 101 may be an acceleration sensor, a gyroscope, etc.; the audio acquisition device 102 is used to collect sound information in the environment and obtain the collected audio data, and the audio acquisition device 102 may be a microphone, etc.

[0030] In an embodiment of the present application, the electronic device can collect motion data through the motion sensor 101, and collect sound information in the environment through the audio collection device 102 to obtain audio data, and then detect the tapping of the electronic device 100 based on the motion data of the electronic device and the collected audio data, and perform interactive operations based on the tapping detection results. Therefore, the motion data and the collected audio data are combined when performing tapping detection, which can improve the accuracy of tapping detection and avoid false triggering of interactive operations.

[0031] The interactive method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0032] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of an interactive method provided by an embodiment of the present application. In a specific embodiment, the interactive method is applied to Fig.12 The interactive device 400 and the electronic device 100 ( Fig.13 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment can be a smart phone, a tablet computer, a smart watch, an e-book, etc., which is not limited here. Figure 2 The process shown is described in detail, and the interaction method may specifically include the following steps:

[0033] Step S110: Acquire first motion data of the electronic device and collected first audio data.

[0034] In an embodiment of the present application, the electronic device can collect its motion data as the first motion data through a motion sensor, and can also collect sound information in the environment through an audio collection device and obtain the first audio data, so as to perform knock detection on the electronic device based on the acquired first motion data and the first audio data. Among them, the motion data of the electronic device can be data such as angular acceleration and linear acceleration. It can be understood that when the shell of the electronic device is knocked, its angular velocity, linear acceleration and other motion data will change, and the motion data when the shell of the electronic device is knocked is different from the motion data when the electronic device is not knocked. Therefore, it is possible to determine whether the electronic device is knocked through the motion data of the electronic device. Taking the linear acceleration data as an example, please refer to Figure 3 , Figure 3 The figure shows the change of acceleration data within 1 second. The length of the horizontal axis is 200, representing 200 frames of acceleration data. It can be seen that when the back cover of the electronic device is knocked twice, two obvious accelerations will appear on the Z axis, corresponding to Figure 3 Similarly, when other surfaces of the electronic device are struck, the acceleration data of the X-axis and Y-axis will also increase significantly. In addition, when the shell of the electronic device is struck, a sound is usually generated, and the sound generated when struck is also usually different from other sounds in the environment. Therefore, it is also possible to determine whether the electronic device is struck by combining the audio data collected by the electronic device.

[0035] In some embodiments, an acceleration sensor is provided in the electronic device, and the linear acceleration of the electronic device can be obtained through the acceleration sensor. The acceleration of three mutually perpendicular axes can be obtained through the acceleration sensor, and the three axes can be the x-axis, the y-axis, and the z-axis. The plane formed by the x-axis and the y-axis is the plane where the screen of the electronic device is located, and the x-axis and the y-axis are perpendicular to each other, and the z-axis is perpendicular to the plane formed by the x-axis and the y-axis. The z-axis can take the vertical downward direction as the positive direction.

[0036] In some embodiments, a gyroscope is provided in the electronic device, and the angular velocity can be detected by the gyroscope. The gyroscope is also called an angular velocity sensor. The electronic device is usually equipped with a three-axis gyroscope, which can track the displacement changes in 6 directions and obtain the acceleration of the electronic device in the three directions of x, y, and z. The rotation and deflection of the electronic device can be well measured, so as to perform corresponding operations on the electronic device. In the electronic device, the applications that usually use gyroscopes include games, camera anti-shake, navigation, etc. Of course, the specific method for the electronic device to detect the above motion data is not limited. Since the above motion data includes motion data of multiple dimensions, it is possible to accurately determine whether the electronic device is knocked based on the motion data.

[0037] In some embodiments, since the motion data of the housing of the electronic device changes when it is tapped, the first motion data can be the motion data within a continuous duration to detect whether the electronic device is tapped by detecting the change in the motion data. In addition, when using the audio data collected by the electronic device to detect whether the electronic device is tapped, the external sound when the electronic device is tapped is different from the external sound when it is not tapped, so the first audio data can also be the audio data within a continuous duration to improve the accuracy of the tapping detection.

[0038] In some embodiments, the electronic device can obtain the first motion data of the electronic device and the first audio data collected when the tapping interaction mode is turned on, and perform subsequent steps. That is to say, the electronic device performs the interaction method provided in the embodiment of the present application only when the tapping interaction mode is turned on. Among them, the electronic device can control the above tapping interaction mode to be in an on state or an off state according to the input setting operation for the above tapping interaction mode, thereby realizing the control of the state of the tapping interaction mode according to the user's usage requirements.

[0039] Step S120: Perform tapping detection based on the first motion data and the first audio data to obtain a tapping detection result.

[0040] In an embodiment of the present application, after obtaining the first motion data of the electronic device and the first audio data collected, tap detection can be performed based on the first motion data and the first audio data to obtain a tap detection result. The above specific tap detection result can be determined according to actual needs. For example, the tap detection result may include: whether the electronic device is tapped, the number of times it is tapped, the location of the tap, etc.

[0041] In some embodiments, data features can be extracted for the above first motion data and the first audio data, and tap detection can be performed based on the extracted data features. In one possible manner, the first motion data can include linear acceleration data, and the electronic device can extract the acceleration features of the X-axis, Y-axis, and Z-axis as the first data feature based on the linear acceleration data. For example, the acceleration value when the amplitude of any axis among the X-axis, Y-axis, and Z-axis appears a peak can be determined, and then the acceleration value when the axis appears a peak is used as the first data feature; in addition, rhythmic features, spectrum, power spectrum, inverse spectrum, spectrum envelope and other features can be extracted for the first audio data as the second data feature; then the acquired first data feature is matched with the first preset feature stored in the electronic device, and the second data feature is matched with the second preset feature stored in the electronic device to obtain a matching result, and the tap detection result is determined based on the matching result. Among them, the first preset feature can be a data feature of motion data obtained in advance when the electronic device is knocked, and the second preset feature can be a data feature of audio data obtained in advance when the electronic device is knocked, and the feature extraction method of the first preset feature is the same as the feature extraction method of the above first data feature, and the feature extraction method of the second preset feature is the same as the feature extraction method of the above second data feature.

[0042] Optionally, the above matching results may include a target matching degree, based on which the probability of the electronic device being tapped can be determined, and the determined probability can be used as the tap detection result. For example, if the matching degree is 0.8, 0.8 can be directly used as the probability that the electronic device is tapped; or, the determined target matching degree can be compared with the target matching degree. If the target matching degree is greater than the matching degree threshold, the tap detection result that the electronic device is tapped can be determined. If the target matching degree is not greater than the target matching degree, the tap detection result that the electronic device is not tapped can be determined. In addition, the acquired first data feature is matched with the first preset feature stored in the electronic device, and the second data feature is matched with the second preset feature stored in the electronic device. When the matching result is obtained, the matching degree of the first data feature with the first preset feature and the matching degree of the second data feature with the second preset feature can be weighted and summed according to the weights of the first data feature and the second data feature, thereby obtaining the target matching degree.

[0043] Optionally, the first preset feature may include data features of motion data corresponding to different tapping situations (number of tapping and / or tapping positions), and the second preset feature may include data features of audio data corresponding to different tapping situations (number of tapping and / or tapping positions). The electronic device may match the acquired first data feature with the first preset feature corresponding to the different tapping situations to obtain a first matching result; match the acquired second data feature with the second preset feature corresponding to the different tapping situations to obtain a second matching result; and then determine the tapping detection result based on the first matching result and the second matching result.

[0044] Among them, the first matching result may include the first matching degree between the first data feature and the first preset feature corresponding to each tapping situation (as the first matching degree corresponding to each tapping situation), and the second matching result may include the second matching degree between the second data feature and the second preset feature corresponding to each tapping situation (as the second matching degree corresponding to each tapping situation). According to the first matching degree and the second matching degree corresponding to the above tapping situations, the first matching degree and the second matching degree with the largest values ​​are determined; if the first matching degree and the second matching degree with the largest values ​​both correspond to the same tapping situation, it is determined whether the first matching degree and the second matching degree with the largest values ​​are both greater than the matching degree threshold; if the first matching degree and the second matching degree with the largest values ​​are both greater than the matching degree threshold, it can be determined that the electronic device has been tapped, and the tapping situation is the tapping situation corresponding to the first matching degree and the second matching degree with the largest values; if the first matching degree and the second matching degree with the largest values ​​do not correspond to the same tapping situation, or any of the first matching degree and the second matching degree with the largest values ​​is not greater than the matching degree threshold, it can be determined that the electronic device has not been tapped.

[0045] Of course, after obtaining the first matching degree and the second matching degree corresponding to the above different knocking situations, for each knocking situation, according to the weights corresponding to the first matching degree and the second matching degree, the first matching degree and the second matching degree corresponding to the same knocking situation are weightedly summed to obtain the third matching degree corresponding to each knocking situation, and the third matching degree is used as the knocking probability corresponding to each knocking situation, so as to obtain the knock detection result.

[0046] In some embodiments, a pre-trained knock detection model is stored in the electronic device. After acquiring the above first motion data and first audio data, the electronic device can input the first motion data and the first audio data into the knock detection model to obtain a knock detection result. The knock detection model can be pre-trained based on sample motion data when the target device is knocked and the collected sample audio data. The knock detection model can be a tree model, such as a lightGBM model, or a neural network model, such as a deep neural network model. Of course, the specific model type of the knock detection model may not be limited.

[0047] Step S130: performing corresponding interactive operations according to the tapping detection result.

[0048] In an embodiment of the present application, after the first motion data of the electronic device and the first audio data collected are input into a pre-trained knock detection model and the knock detection result output by the knock detection model is obtained, the electronic device can perform an interactive operation according to the knock detection result. Among them, the electronic device can obtain information such as whether the electronic device is knocked, the number of knocks, the knock position, etc. according to the knock detection result, and determine whether an interactive operation needs to be performed based on the obtained information, and perform the corresponding interactive operation if it is determined that an interactive operation needs to be performed; the above interactive operations performed by the electronic device can be pre-set, for example, can be default settings, or can be set according to detected user operations.

[0049] In some embodiments, the electronic device performs corresponding interactive operations according to the above tap detection results, which may be to open the target Near Field Communication (NFC) card, run the target application, adjust the volume, adjust the screen display brightness, adjust the communication mode, adjust the playback progress, hide the message notification, answer the call, or hang up the call. Of course, the interactive operations that the electronic device can perform according to the tap detection results are not limited to these. For example, the electronic device can also control the flash to turn on to realize the flashlight function according to the acquired tap detection results.

[0050] The interactive method provided in the embodiment of the present application, when performing tapping detection on an electronic device, not only uses motion data for tapping detection, but also combines the collected audio data, so as to improve the accuracy of tapping detection, thereby avoiding false triggering of interactive operations and improving the accuracy of the electronic device in performing interactive operations.

[0051] See also Figure 4 , Figure 4 A flow chart of an interaction method provided by another embodiment of the present application is shown. The interaction method is applied to the above electronic device. Figure 4The process shown is described in detail, and the interaction method may specifically include the following steps:

[0052] Step S210: Acquire first motion data of the electronic device and collected first audio data.

[0053] In the embodiment of the present application, step S210 can refer to the contents of other embodiments and will not be repeated here.

[0054] Step S220: Perform tap detection based on the first motion data and the first audio data to obtain a tap detection result, wherein the tap detection result includes a tap probability, and the tap probability is used to indicate the probability that the electronic device is tapped.

[0055] In an embodiment of the present application, the knock detection result obtained above may include a knock probability, which indicates the probability that the electronic device is knocked, and the knock probability is positively correlated with the possibility that the electronic device is knocked, that is, the greater the knock probability, the greater the possibility that the electronic device is knocked, and the smaller the knock probability, the smaller the possibility that the electronic device is knocked. Therefore, based on the knock detection result, it can be determined whether the electronic device is knocked.

[0056] In some embodiments, the knock detection result is output by a pre-trained knock detection model based on the above first motion data and the first audio data. The knock detection model can be used to determine the label probability of the first label corresponding to the input data based on the input motion data and audio data. The label probability of the first label represents the possibility that the knock detection result is the information represented by the first label. The first label is used to represent that the electronic device is knocked. The larger the label probability of the first label, the greater the possibility of representing that the electronic device is knocked.

[0057] Step S230: If the tapping probability is greater than the first threshold, a corresponding interactive operation is performed.

[0058] In an embodiment of the present application, after obtaining the tap detection result, it can be determined whether to perform an interactive operation based on the above tap probability in the tap detection result. Among them, the electronic device can compare the above tap probability with the first threshold value, and the first threshold value is used as a basis for determining whether the electronic device is tapped. If the tap probability in the tap detection result is greater than the first threshold value, it means that the electronic device is tapped. Therefore, in the case of determining that the tap probability is greater than the first threshold value, the corresponding interactive operation can be performed in response to the electronic device being tapped (i.e., in response to the user's tapping action on the electronic device); if the tap probability in the tap detection result is less than or equal to the first threshold value, it means that the electronic device is not tapped, in which case, the corresponding interactive operation may not be performed. Among them, the specific value of the first threshold value may not be limited, and the first threshold value may be determined according to the numerical range of the tap probability. For example, if the numerical range of the tap probability is 0 to 1, the first threshold value may be 0.5, 0.6, 0.7, 0.9, etc.

[0059] In some embodiments, the knock detection result output by the above knock detection model for the input motion data and audio data may be a knock detection result for a specified number of knocks, that is, the above knock probability is the probability that the electronic device is knocked for a specified number of knocks. In addition, the interactive operation executed by the knock trigger may also be an interactive operation set for the specified number of knocks, and the interactive operation set for the specified number of knocks may be a default setting or may be set according to user operation, which is not limited here. Among them, the above specified number of knocks may not be limited, for example, it may be 1 time, 2 times, 3 times, 4 times, etc. If the above knock probability is greater than the first threshold, it means that the electronic device is knocked and the specified number of knocks is knocked, so in this case, the above interactive operation can be performed; if the above knock probability is less than or equal to the first threshold, it means that the electronic device is not knocked, or the electronic device is knocked but the number of knocks is not the specified number of knocks, in this case, the above interactive operation may not be performed.

[0060] Exemplarily, the electronic device can turn on the camera and display the QR code scanning interface according to the user's double-click operation on the back cover of the electronic device. The knock probability output by the above knock detection model according to the input motion data and audio data is the probability that the electronic device is knocked twice. Therefore, if the knock probability output by the knock detection model is greater than the first threshold, it means that the electronic device has been knocked twice, that is, the user has double-clicked the back cover of the electronic device, and at this time, the camera can be turned on and the QR code scanning interface can be displayed.

[0061] In some embodiments, the knock probability in the above knock detection result may include a first probability that the electronic device is knocked with different knock times, that is, when the above knock detection is performed based on the first motion data and the first audio data, the probability that the electronic device is knocked with different knock times can be detected simultaneously. In addition, for each of the different knock times, a corresponding interactive operation can be set, and the interactive operations corresponding to different knock times can be the same or different. For example, the interactive operation corresponding to knock number 1 is to turn on the NFC card, the interactive operation corresponding to knock number 2 can be to adjust the communication mode, and the interactive operation corresponding to knock number 3 can be to answer the call. For another example, the interactive operations corresponding to knock number 1 and knock number 3 can both be to answer the call, and the interactive operation corresponding to knock number 2 can be to hang up the call.

[0062] In the above implementation, when the electronic device performs the corresponding interactive operation according to the tap detection result, the first probability corresponding to each number of taps in the above tap probability can be compared with the above first threshold value. If the first probability corresponding to the target number of taps is greater than the above first threshold value, it means that the electronic device is tapped and the target number of taps is tapped. In this case, the interactive operation corresponding to the target number of taps can be performed. Among them, the target number of taps is any one of the above different numbers of taps. If the first probability corresponding to each number of taps is less than or equal to the first threshold value, it means that the electronic device has not been tapped, or the electronic device has been tapped but the number of taps is not any of the above different numbers of taps, so the electronic device may not perform the interactive operation. In the above manner, the corresponding interactive operation can be set for the number of taps, so that the interactive operations that can be realized through the tapping operation can be enriched, which facilitates the user's use of the electronic device.

[0063] In some embodiments, the knocking probability in the above knocking detection result may include a second probability that each knocking surface of the electronic device is knocked, that is, when the knocking detection is performed according to the above first motion data and the first audio data, the probability of each knocking surface of the electronic device being knocked can be detected at the same time, and the knocking surface can also be understood as the position of the knocking. In addition, for different knocking surfaces, corresponding interactive operations can be set, and the knocking surface can include the back cover (i.e., the back) of the electronic device and the four surfaces of the middle frame of the electronic device located in different directions; the interactive operations corresponding to different knocking surfaces can be the same or different, for example, the interactive operation corresponding to the back cover is to open the NFC card, the interactive operation corresponding to the knocking surface of the middle frame located in the first direction can be to increase the volume, the interactive operation corresponding to the knocking surface of the middle frame located in the second direction can be to reduce the volume, the interactive operation corresponding to the knocking surface of the middle frame located in the third direction can be to answer the call, and the interactive operation corresponding to the knocking surface of the middle frame located in the fourth direction can be to hang up the call, and for example, for example, the interactive operations corresponding to the back cover (i.e., the back) of the electronic device and the four surfaces of the middle frame of the electronic device located in different directions can all be to open the camera and display the QR code scanning interface.

[0064] In the above embodiments, when the electronic device performs the corresponding interactive operation according to the knock detection result, the second probability corresponding to each knock surface in the above knock probability can be compared with the above first threshold value. If the second probability corresponding to the target knock surface is greater than the above first threshold value, it means that the electronic device is knocked, and the target knock surface of the electronic device is knocked. In this case, the interactive operation corresponding to the target knock surface can be performed. Among them, the target knock surface is any knock surface among the above different knock surfaces. If the first probability corresponding to each knock surface is less than or equal to the first threshold value, it means that the electronic device is not knocked, so the electronic device may not perform the interactive operation. In the above manner, the corresponding interactive operation can be set for the knock surface, so that the interactive operations that can be realized through the knock operation can be enriched, which facilitates the user's use of the electronic device.

[0065] In one possible implementation, after the electronic device is used by the user for the first time, the side that is held each time it is held with one hand can be detected as the holding side, and the holding side detected each time can be stored as the historical holding side; after the number of times it is detected that it is held with one hand reaches a target number of times, the user's preferred holding side can be determined for the stored historical holding sides, and then a recommended setting plan for interactive operations corresponding to different tapping surfaces is generated based on the preferred holding side, and the generated recommended setting plan is displayed for the user to select, modify, and perform other operations on the recommended setting plan. Among them, the historical number of interactive operations performed in the user's history can be classified into first, second, third, fourth and fifth categories of interactive operations, and the historical numbers corresponding to the first, second, third, fourth and fifth categories of interactive operations decrease in sequence; then the operation of the first category is determined as the operation corresponding to the back cover of the electronic device, and the operation of the second category is determined as the interactive operation corresponding to the first target surface of the middle frame, and the first target surface is one of the four surfaces of the middle frame corresponding to the above preferred holding side; the interactive operations of the third, fourth and fifth categories are respectively determined as interactive operations corresponding to one of the four surfaces of the middle frame except the above first target surface according to the target rules, wherein the interactive operations of the third, fourth and fifth categories correspond one-to-one to the other three surfaces of the four surfaces of the middle frame except the above first target surface. It can be understood that when a user usually taps an electronic device while holding the electronic device, the back cover is the side that is most convenient for the user to tap, so the first category of interactive operations that the user has performed the most can be determined as the interactive operations corresponding to the back cover; and when the user holds the electronic device, one of the four sides of the middle frame that corresponds to the preferred holding side is also the side that is convenient for the user to tap, so the first category of interactive operations can be determined as the interactive operations corresponding to the back cover. The recommended setting scheme generated in the above manner can be more in line with the user's usage habits and needs, thereby reducing the user's setting operations when setting interactive operations for each tapping surface.

[0066] Optionally, the above target rule may be to randomly determine the interaction operations of the third category, the fourth category and the fifth category as the interaction operations corresponding to one of the four surfaces of the middle frame except the first target surface mentioned above; the above target rule may also be to determine the interaction operations of the third category as the interaction operations corresponding to the second target surface of the middle frame, to determine the interaction operations of the fourth category as the interaction operations corresponding to the upper surface of the middle frame, and to determine the interaction operations of the fifth category as the interaction operations corresponding to the lower surface of the middle frame.

[0067] In some embodiments, the above embodiments can also be combined, that is, the knock probability in the above knock detection result can include the first probability that the electronic device is knocked with different knock times and the second probability that each knock surface of the electronic device is knocked, that is, when the knock detection is performed based on the first motion data and the first audio data, the probability that each knock surface of the electronic device is knocked and the probability that the electronic device is knocked with different knock times can be detected at the same time, that is, the probability that each knock surface of the electronic device is knocked with different knock times can be detected based on the first motion data and the second motion data. In this embodiment, different interactive operations can be pre-set for different combinations of knock surfaces and knock times; when the electronic device performs the corresponding interactive operation according to the knock probability, the first probability can be compared with the above first threshold, and the second probability can be compared with the first threshold. If the first probability corresponding to the target knock number is greater than the above first threshold, and the second probability corresponding to the target knock surface is greater than the above first threshold, it means that the electronic device is knocked, and the target knock surface is knocked for the target knock number of times. In this case, the interactive operation corresponding to the target knock surface and the target knock number can be performed.

[0068] Exemplarily, the tapping surface includes the back cover, the first surface of the middle frame, the second surface of the middle frame, the third surface of the middle frame and the fourth surface of the middle frame, the tapping times include 1 and 2, and the interactive operations corresponding to the combination of the tapping surface and the tapping times can be shown in the following table:

[0069] Tap 1 time Tap 2 times Back cover Turn on the camera Turn off camera The first side of the middle frame Volume Up Volume Down The second side of the middle frame Answering a call Hang up the phone The third side of the middle frame Increase screen brightness Lower the screen brightness The fourth side of the middle frame Open NFC card 1 Open NFC card 2

[0070] It can be seen that, through the above method, corresponding interactive operations can be set for the combination of the tapping surface and the number of tapping times, thereby enriching the interactive operations achievable through the tapping operation and facilitating the user's use of the electronic device.

[0071] The interactive method provided in the embodiment of the present application improves the accuracy of tap detection, thereby avoiding false triggering of interactive operations and improving the accuracy of electronic devices in performing interactive operations. In addition, by performing corresponding interactive operations based on the number of taps and the tapping surface, the interactive operations that can be achieved through tapping operations can be enriched, thereby facilitating users' use of electronic devices.

[0072] See also Figure 5 , Figure 5 A flow chart of an interaction method provided by another embodiment of the present application is shown. The interaction method is applied to the above electronic device. Figure 5 The process shown is described in detail, and the interaction method may specifically include the following steps:

[0073] Step S310: Acquire first motion data of the electronic device and collected first audio data.

[0074] In some embodiments, when acquiring data for performing knock detection, the motion data of the electronic device in the current time period can be acquired, and the motion data in the current time period can be used as the first motion data, and the audio data collected in the current time period can be acquired, and the audio data collected in the current time period can be used as the first audio data. It can be understood that since it is necessary to detect the change of motion data to detect whether the electronic device is knocked, and according to the difference between the external sound when the electronic device is knocked and the external sound when it is not knocked, whether the electronic device is knocked is detected, so by acquiring the audio data of the current time period (i.e., the audio data within the continuous duration) for knock detection, the accuracy of knock detection can be improved. Wherein, the above current time period is the time period from the first moment before the current moment to the current moment, and the first moment can be determined by the duration of the current time period. For example, the duration of the current time period is 1 second, and the current time is the 1st second, then the time period from the 0th second to the 1st second can be used as the current time period.

[0075] In a possible implementation, the electronic device may obtain the motion data of the current time period as the first motion data and the audio data collected in the current time period as the first audio data at intervals of the first time length. The specific value of the first time length may not be limited. For example, the first time length may be 0.1 seconds, and the current time period is 1 second. The electronic device may obtain the motion data of the most recent 1 second and the audio data collected in the most recent 1 second at intervals of 0.1 seconds. For example, during the period from the 0th second to the 1st second, since the duration is less than the duration of the current time period (i.e., 1 second), the motion data of the current time period as the first motion data and the audio data collected in the current time period as the first audio data are not obtained at this time. At the 1st second, the motion data in the time period from the 0th second to the 1st second and the audio data collected in the time period are obtained; at the 1.1th second, the motion data in the time period from the 0th second to the 1st second and the audio data collected in the time period are obtained. It can be understood that the above method can be regarded as using a sliding window method to obtain data for tap detection. Taking the motion data as acceleration data as an example, please refer to Figure 6 , Figure 7 and Figure 8 , Figure 6 The acceleration data in the middle window A1 is the acceleration data obtained at time T1. Figure 7 The acceleration data in the middle window is the acceleration data obtained at time T2. Figure 8The acceleration data in the middle window is the acceleration data acquired at time T3, wherein the interval between T3 and T2 is the first time length, and the interval between T2 and T1 is the first time length.

[0076] In a possible implementation, a motion sensor for detecting motion data in an electronic device can produce data at a frequency of N frames per second, where N can be 50, 100, 200, etc., and the specific value can be unlimited, that is, the motion sensor can generate N data per second; an audio acquisition device for collecting audio data in an electronic device can produce data at a target frequency, and the target frequency can be 16kHz, 32kHz, 48kHz, etc., where the amount of data produced at 16kHz is the smallest, and the sound quality when tapping has met the recognition requirements, and the amount of calculation under the condition of meeting the accuracy is the smallest. Thus, the motion sensor in the electronic device can continuously collect motion data, the audio acquisition device can continuously collect audio data, and the electronic device can obtain the motion data of the current time period from the continuously collected motion data when executing the interactive method provided in the embodiment of the present application, and obtain the audio data collected in the current time period from the continuously collected audio data.

[0077] It should be noted that the above method of obtaining the first motion data and the first audio data for tapping detection can be applied to other embodiments.

[0078] Step S320: input the first motion data into a first feature extraction module of a knock detection model to obtain a first motion feature output by the first feature extraction module.

[0079] In an embodiment of the present application, when the electronic device performs knock detection based on the first motion data and the first audio data, it can use a pre-trained knock detection model, wherein the electronic device can input the first motion data and the first audio data into the pre-trained knock detection model to obtain a knock detection result. The knock detection model is trained based on sample motion data when the target device is knocked and the collected sample audio data. The sample motion data and sample audio data of the target device at the same time can be used as a group of sample data and marked with corresponding labels. The labels are used to indicate whether the electronic device is knocked or not, the number of knocks, the knock position, etc. The target device can be the above electronic device, or it can be a device of the same device type or model as the above electronic device.

[0080] In some embodiments, the sample data used to train the above knock detection model may include, in addition to the sample motion data when the target device is knocked and the sample audio data collected, the sample motion data when the target device is not knocked and the sample audio data collected. It can be understood that the sample motion data when the target device is knocked and the sample audio data collected can be used as positive sample data when training the knock detection model, and the sample motion data when the target device is not knocked and the sample audio data collected can be used as negative sample data when training the knock detection model.

[0081] In an embodiment of the present application, the knock detection model may include a first feature extraction module, a second feature extraction module and a classification module, the first feature extraction module is used to extract motion features of input motion data, and the second feature extraction module is used to extract audio features of input audio data. After acquiring the above first motion data and first audio data, the electronic device can input the first motion data into the first feature extraction module to obtain the motion features output by the first feature extraction module (i.e., the first motion features).

[0082] The first feature extraction module may be a neural network, for example, the neural network may be a deep neural network composed of a convolutional layer and a fully connected layer, the size of its input data may be 3*200, and the size of its output data may be 1024. Of course, the specific type of the first feature extraction module may not be limited, for example, the first feature extraction module may also be a model for extracting features such as MobileNetV2 for mobile terminals.

[0083] Step S330: input the first audio data into the second feature extraction module of the knock detection model to obtain the first audio feature output by the second feature extraction module.

[0084] In an embodiment of the present application, after acquiring the above first motion data and the first audio data, the first audio data can also be input into the above second feature extraction module to obtain the first audio feature extracted by the second feature extraction module for the first audio data.

[0085] In some embodiments, the second feature extraction module for extracting audio features may include a first feature extraction submodule and a second feature extraction submodule, wherein the first feature extraction submodule is used to extract Mel spectrum coefficient features for audio data, and the second feature extraction module is used to further perform feature extraction on the Mel spectrum coefficient features to obtain audio features. After obtaining the above first audio data, the first audio data may be input into the first feature extraction submodule to obtain the Mel spectrum coefficient features corresponding to the first audio data, and then the Mel spectrum coefficient features may be input into the second feature extraction submodule to obtain the audio features output by the second feature extraction submodule as the above first audio features. The first feature extraction submodule is cascaded with the second feature extraction submodule, that is, the output data of the first feature extraction submodule is used as input data and input into the second feature extraction submodule, so that after the first audio data is input into the first feature extraction submodule, the Mel spectrum coefficient features extracted by the first feature extraction submodule for the first audio data are input into the second feature extraction submodule, and then the above first audio features may be obtained.

[0086] In the above implementation, the first feature extraction submodule converts the audio data to the frequency domain to obtain more significant audio features (Mel spectrum coefficient features). When the first feature extraction submodule extracts the Mel spectrum coefficient features, the input audio data can be pre-emphasized, framed, windowed, Fourier transformed to obtain the power spectrum, and the Mel filter bank is filtered to obtain the Mel spectrum map, and then the discrete cosine transform (DCT for Discrete Cosine Transform, DCT) is used to obtain the Mel spectrum coefficients, and finally a matrix output of 96*64 size (i.e., Mel spectrum coefficient features) is obtained. The second feature extraction submodule can be a neural network. For example, the neural network can be a deep neural network composed of a convolutional layer and a fully connected layer, and its input data is a Mel spectrum coefficient feature of 96*64 size, and the size of the output data is 1024. Of course, the specific type of the second feature extraction submodule can be unlimited. For example, the second feature extraction submodule can also be a model for extracting features such as MobileNetV2 for mobile terminals.

[0087] Step S340: input the first motion feature and the first audio feature into a classification module of a knock detection model to obtain a knock detection result.

[0088] In an embodiment of the present application, after obtaining the first motion feature output by the first feature extraction module and the first audio feature output by the second feature extraction module, the first motion feature and the first audio feature can be input into the above classification module to obtain the knock detection result output by the classification module. The knock detection result output by the classification module can be a probability of characterizing the electronic device being knocked, the probability of the electronic device being knocked N times, the probability of the target knocking surface of the electronic device being knocked, etc., where N is a positive integer. Among them, when the first motion feature and the first audio feature are input into the above classification module, the first motion feature and the first audio feature can be fused, and then the fused features are input into the classification module to obtain the knock detection result output by the classification module. The classification module can be a Softmax logical regression model (Softmaxlogical regression), or a support vector machine (Support Vector Machine, SVM), etc., and the specific classification module may not be limited.

[0089] Next, combine Fig. 9 The principle of the above tap detection model is explained.

[0090] See also Figure 6 The knock detection model 300 in the embodiment of the present application includes a first feature extraction module 310, a second feature extraction module 320 and a classification module 330, and the second feature extraction module 320 includes a first feature extraction sub-module 321 and a second feature extraction sub-module 322; the first motion data can be input into the first feature extraction module 310 to obtain the first motion feature; the first audio data can be input into the first feature extraction sub-module 321 to obtain the Mel spectrum coefficient feature; after the Mel spectrum coefficient feature is input into the second feature extraction sub-module 322, the first audio feature is obtained; then the first motion feature and the first audio feature are fused and input into the classification module 330 to obtain the knock detection result.

[0091] Regarding the knock detection model in the embodiment of the present application, a training method for the knock detection model is also provided in the present embodiment. It is worth noting that the training of the knock detection model can be performed in advance based on the acquired sample data. Subsequently, each time a knock detection is required, the trained knock detection model can be used for it, without the need to train the knock detection model each time a knock detection is performed.

[0092] In some embodiments, a sample data set used to train a knock detection model includes multiple groups of sample data, each group of sample data includes sample motion data of a target device and collected sample audio data, the sample motion data may include first sample motion data and second sample motion data, the sample audio data includes first sample audio data and second sample audio data, the first sample motion data and the first sample audio data are respectively motion data and collected audio data when the target device is knocked, the second sample motion data and the second sample audio data are respectively motion data and collected audio data when the target device is not knocked, each of the above groups of sample data is marked with a corresponding knock detection label, and the knock detection label is used to characterize information such as whether the electronic device is knocked, the knocking surface, and the number of knocks.

[0093] After obtaining the above sample data set, the sample motion data and sample audio data in the sample data set can be input into the initial detection model to obtain the output result of the initial detection model, and then the loss value can be determined based on the difference between the output result of the initial detection model and the knock detection label marked by the sample motion data and the sample audio data; after obtaining the loss value, the initial detection model can be iteratively trained according to the loss value to obtain the final knock prediction model.

[0094] In a possible implementation, the initial detection model can be iteratively trained using the Adam optimizer according to the total loss value until the loss value of the output result of the initial detection model converges, and the model at this time is saved to obtain a trained knock detection model. Among them, the Adam optimizer combines the advantages of the two optimization algorithms AdaGra (Adaptive Gradient) and RMSProp, comprehensively considers the first moment estimation (First Moment Estimation, i.e. the mean of the gradient) and the second moment estimation (Second Moment Estimation, i.e. the uncentered variance of the gradient), and calculates the update step size. The training end conditions of the iterative training may include: the number of iterative training reaches the target number of times; or the loss value of the output result of the initial detection model meets the set conditions. Optionally, the loss value meeting the set conditions may include: the loss value is less than the loss threshold, or the absolute value of the difference between the loss values ​​obtained twice in a row is less than the target threshold. Of course, the specific set conditions may not be used as a limitation.

[0095] It is worth noting that the training of the tap detection model can be performed by an electronic device that executes the interaction method provided in the embodiment of the present application, or it can be trained by other devices (such as a server) and then sent to the electronic device that executes the interaction method provided in the embodiment of the present application.

[0096] Step S350: Execute corresponding interactive operations according to the tapping detection result.

[0097] In the embodiment of the present application, step S350 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0098] The interactive method provided in the embodiment of the present application obtains motion data and collected audio data of the electronic device when performing knock detection on the electronic device, extracts motion features for the motion data through a first feature extraction module, and extracts audio features for the audio data through a second feature extraction module, and then identifies the motion features and audio data through a classification module to obtain a knock detection result. Therefore, on the basis of using motion data for knock detection, the collected audio data is combined, so that the accuracy of knock detection can be improved, thereby avoiding false triggering of interactive operations and improving the accuracy of electronic devices in performing interactive operations.

[0099] See also Fig.10 , Fig.10 A flow chart of an interaction method provided by another embodiment of the present application is shown. The interaction method is applied to the above electronic device. Fig.10 The process shown is described in detail, and the interaction method may specifically include the following steps:

[0100] Step S410: If the electronic device is in a held state, controlling an audio collection device of the electronic device to collect audio data, and obtaining first motion data of the electronic device and first audio data collected by the audio collection device.

[0101] In the embodiment of the present application, considering that the audio collection device of the electronic device is usually not working, and when the electronic device executes the interactive method provided in the embodiment of the present application, the electronic device needs to continuously collect audio data, which will lead to an increase in power consumption. Therefore, when the electronic device executes the interactive method provided in the embodiment of the present application, it can determine whether the electronic device is in a held state or an unheld state. If the electronic device is in a held state, it means that the electronic device is currently being used by the user. In this case, the user may achieve the required interactive operation by tapping, so the electronic device can control the audio collection device of the electronic device to collect audio data, and obtain the first motion data of the electronic device and the first audio data collected by the audio collection device; if the electronic device is not in a held state, the electronic device may not control the audio collection device to collect audio data, thereby reducing the power consumption of the electronic device.

[0102] In some embodiments, the back cover and the middle frame of the electronic device may be provided with touch sensors, pressure sensors, etc., and based on the data detected by the sensors provided on the back cover and the middle frame, and the touch data collected by the touch screen of the electronic device, it can be determined whether the electronic device is in a held state or a not held state. For example, if the touch sensors provided on the back cover and the middle frame do not detect touch data, and the touch screen does not collect touch data, it can be determined that the electronic device is in an not held state; if the touch sensors provided on the back cover and the middle frame detect touch data, and / or the touch screen collects touch data, it can be determined that the electronic device is in a held state.

[0103] Step S420: Perform tapping detection based on the first motion data and the first audio data to obtain a tapping detection result.

[0104] Step S430: Execute corresponding interactive operations according to the tapping detection result.

[0105] In the embodiment of the present application, step S420 and step S430 may refer to the contents of other embodiments and will not be described in detail here.

[0106] The interactive method provided in the embodiment of the present application, when performing tapping detection on the electronic device, not only uses motion data for tapping detection, but also combines the collected audio data, so as to improve the accuracy of tapping detection, thereby avoiding false triggering of interactive operations and improving the accuracy of the electronic device in performing interactive operations; in addition, the electronic device controls the audio collection device to collect audio data and executes the interactive method provided in the embodiment of the present application only when it detects that it is in a held state, which can effectively reduce the power consumption of the electronic device.

[0107] See also Fig.11 , Fig.11 A flow chart of an interaction method provided by another embodiment of the present application is shown. The interaction method is applied to the above electronic device. Fig.11 The process shown is described in detail, and the interaction method may specifically include the following steps:

[0108] Step S510: If the electronic device is in a held state and a swinging operation on the electronic device is detected, the audio collection device of the electronic device is controlled to collect audio data, and obtain first motion data of the electronic device and first audio data collected by the audio collection device.

[0109] In the embodiment of the present application, in order to more accurately identify the situation where the user needs to implement the interactive operation through the tapping action, the electronic device can also detect the swinging operation on the electronic device when it is determined that it is in the state of being held. The swinging operation is an operation for swinging the body of the electronic device; if the swinging operation on the electronic device is detected, the audio acquisition device of the electronic device can be controlled to collect audio data, and the first motion data of the electronic device and the first audio data collected by the audio acquisition device can be obtained; if the above swinging operation is not detected, that is, the electronic device is in the state of not being held but the swinging operation is not detected, the electronic device may not control the audio acquisition device to collect audio data, thereby reducing the power consumption of the electronic device. In this way, when the user holds the electronic device, the body of the electronic device can be shaken to trigger the implementation of the interactive method provided in the embodiment of the present application, thereby satisfying the user's need to implement the corresponding interactive operation through the tapping operation.

[0110] In some embodiments, the electronic device can detect the swinging operation through at least one of a gravity sensor, an acceleration sensor, a gyroscope sensor, and a six-axis sensor (or a nine-axis sensor or a twelve-axis sensor) that is a fusion of an acceleration sensor and a gyroscope sensor.

[0111] As a possible implementation, the motion sensor provided in the electronic device may include a gravity sensor and a gyroscope sensor. The gravity sensor is implemented through the piezoelectric effect. There is a weight inside the gravity sensor that is integrated with the piezoelectric sheet. The horizontal direction is calculated through the voltage generated in two orthogonal directions. When the gravity sensor is used in an electronic device, it can usually be used to switch between the horizontal and vertical screen directions. The gyroscope, also known as the angular velocity sensor, is usually configured in the electronic device with a three-axis gyroscope, which can track the displacement changes in 6 directions, obtain the angular acceleration of the electronic device in the three directions of x, y, and z, and make a good measurement of the rotation and deflection of the mobile phone, so as to perform corresponding operations on the electronic device. In the electronic device, the applications that usually use gyroscopes include games, camera anti-shake, navigation, etc. Among them, the electronic device can detect that it has a certain rotation angle through the gyroscope, and when it detects that there is a gravity dump through the gravity sensor, it determines that a swing operation is detected.

[0112] Of course, the specific manner in which the electronic device detects the swinging operation thereof may not be limited, and is not limited to the above-mentioned implementation manner. For example, whether a swinging operation occurs may also be determined by signals received by a set camera or antenna.

[0113] In the above implementation, when the electronic device detects a swinging operation on it through the motion sensor, it can also detect the tilt direction of the electronic device through the motion sensor, so that when performing subsequent interactive operations, corresponding interactive operations can be performed according to the detected tilt direction.

[0114] Step S520: Perform tapping detection based on the first motion data and the first audio data to obtain a tapping detection result.

[0115] In the embodiment of the present application, step S520 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0116] Step S530: performing a corresponding interactive operation according to the first direction in which the electronic device is tilted when the swing operation is detected, and the tapping detection result.

[0117] In the embodiment of the present application, since triggering the electronic device to execute the interactive method provided in the embodiment of the present application requires the user to perform a swinging operation, when performing the interactive operation, in addition to based on the tapping detection result, it can also be based on the first direction in which the electronic device is tilted when the swinging operation is detected (i.e., the swinging direction of the swinging operation detected above), that is, the electronic device can perform the corresponding interactive operation based on the tapping detection result and the above first direction, thereby enriching the interactive operations that can be achieved through the tapping operation and facilitating the user's use of the electronic device.

[0118] In some embodiments, when it is determined that the electronic device is tapped based on the tap detection result, the number of taps of the electronic device can be determined based on the tap detection result, and then, based on the number of taps of the electronic device and the above first direction, interactive operations corresponding to the number of taps and the first direction are performed; when it is determined that the electronic device is tapped based on the tap detection result, the tapping surface of the electronic device can be determined based on the tap detection result, and then, based on the tapping surface of the electronic device and the above first direction, interactive operations corresponding to the tapping surface and the first direction are performed; when it is determined that the electronic device is tapped based on the tap detection result, the number of taps and the tapping surface of the electronic device can be determined based on the tap detection result, and then, based on the number of taps, the tapping surface and the above first direction, interactive operations corresponding to the number of taps, the tapping surface and the first direction are performed.

[0119] The interactive method provided in the embodiment of the present application, when performing tap detection on the electronic device, not only uses motion data for tap detection, but also combines the collected audio data, so as to improve the accuracy of tap detection, thereby avoiding false triggering of interactive operations and improving the accuracy of the electronic device in performing interactive operations; and, the electronic device controls the audio collection device to collect audio data and executes the interactive method provided in the embodiment of the present application only when it is determined that the electronic device is in a held state and a swing operation is detected on the electronic device, which can effectively reduce the power consumption of the electronic device; in addition, after obtaining the tap detection result, the corresponding interactive operation is performed according to the first direction in which the electronic device is tilted when the swing operation is detected and the tap detection result, which can enrich the interactive operations that can be realized through the tap operation and facilitate the user's use of the electronic device.

[0120] See also Fig.12 , which shows a structural block diagram of an interactive device 600 provided in an embodiment of the present application. The interactive device 600 applies the above-mentioned electronic device, and the interactive device 600 includes: a data acquisition module 610, a tapping recognition module 620 and an interactive execution module 630. Among them, the data acquisition module 610 is used to obtain the first motion data of the electronic device and the collected first audio data; the tapping recognition module 620 is used to perform tapping detection based on the first motion data and the first audio data to obtain a tapping detection result; the interactive execution module 630 is used to perform a corresponding interactive operation according to the tapping detection result.

[0121] In some implementations, the tap detection result includes a tap probability, where the tap probability is used to indicate the probability that the electronic device is tapped. The interaction execution module 630 may be specifically configured to execute a corresponding interaction operation if the tap probability is greater than a first threshold.

[0122] In a possible implementation, the tap probability includes a first probability of the electronic device being tapped at different tap times. The interaction execution module 630 may be specifically configured to execute the interaction operation corresponding to the target tap times if the first probability corresponding to the target tap times is greater than the first threshold.

[0123] In a possible implementation, the tap probability includes a second probability that each tap surface of the electronic device is tapped. The interaction execution module 630 may be specifically configured to execute the interaction operation corresponding to the target tap surface if the second probability corresponding to the target tap surface is greater than the first threshold.

[0124] In some embodiments, the knock recognition module 620 can be specifically used to input the first motion data and the first audio data into a pre-trained knock detection model to obtain a knock detection result. The knock detection model is trained based on sample motion data when the target device is knocked and collected sample audio data.

[0125] In a possible implementation, the knock detection model includes a first feature extraction module, a second feature extraction module, and a classification module. The knock recognition module 620 can be specifically used to input the first motion data into the first feature extraction module to obtain the first motion feature output by the first feature extraction module; input the first audio data into the second feature extraction module to obtain the first audio feature output by the second feature extraction module; input the first motion feature and the first audio feature into the classification module to obtain a knock detection result.

[0126] Optionally, the second feature extraction module includes a first feature extraction submodule and a second feature extraction submodule. The knock recognition module 620 can also be used to input the first audio data into the first feature extraction submodule to obtain the Mel spectrum coefficient feature corresponding to the first audio data; input the Mel spectrum coefficient feature into the second feature extraction submodule to obtain the audio feature output by the second feature extraction submodule as the first audio feature.

[0127] In some embodiments, the data acquisition module 610 can be specifically used to control the audio acquisition device of the electronic device to collect audio data if the electronic device is in a held state, and to obtain the first motion data of the electronic device and the first audio data collected by the audio acquisition device.

[0128] In a possible implementation, the data acquisition module 610 can be specifically used to control the audio acquisition device of the electronic device to collect audio data if the electronic device is in a held state and a swing operation is detected on the electronic device, and to obtain the first motion data of the electronic device and the first audio data collected by the audio acquisition device.

[0129] Optionally, the interaction execution module 630 may be specifically configured to execute a corresponding interaction operation according to detecting a first direction in which the electronic device is tilted when the swing operation is performed, and the tap detection result.

[0130] In some embodiments, the data acquisition module 610 can be specifically used to obtain motion data of the electronic device in a current time period as the first motion data, where the current time period is a time period from a first moment before the current moment to the current moment; and obtain audio data collected by the electronic device in the current time period as the first audio data.

[0131] In some embodiments, the interactive execution module 610 can be specifically used to turn on the target near-field communication NFC card, run the target application, adjust the volume, adjust the screen display brightness, adjust the communication mode, adjust the playback progress, hide message notifications, answer a call, or hang up a call based on the tap detection results.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0133] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0134] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0135] In summary, the solution provided by the present application obtains the first motion data of the electronic device and the first audio data collected; performs tap detection based on the first motion data and the first audio data to obtain a tap detection result; and performs a corresponding interactive operation according to the tap detection result. Since the electronic device not only uses the motion data for tap detection, but also combines the collected audio data, it can improve the accuracy of the tap detection, thereby avoiding the false triggering of the interactive operation and improving the accuracy of the electronic device in performing the interactive operation.

[0136] Please refer to Fig.13 , which shows a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, a smart watch, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to execute the method described in the aforementioned method embodiment.

[0137] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0138] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0139] Please refer to Fig.14 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0140] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An interactive method, characterized in that: Applied to electronic equipment, the method comprises: Acquire first motion data of the electronic device and collected first audio data; Performing tap detection based on the first motion data and the first audio data to obtain a tap detection result; According to the tap detection result, a corresponding interactive operation is performed.

2. The method according to claim 1, characterized in that The tap detection result includes a tap probability, where the tap probability is used to indicate a probability that the electronic device is tapped. The performing of a corresponding interactive operation according to the tap detection result includes: If the tapping probability is greater than the first threshold, the corresponding interactive operation is performed.

3. The method according to claim 2, characterized in that The knock probability includes a first probability that the electronic device is knocked at different knock times, and if the knock probability is greater than a first threshold, performing a corresponding interactive operation includes: If the first probability corresponding to the target tapping number is greater than the first threshold, the interactive operation corresponding to the target tapping number is performed.

4. The method according to claim 2, characterized in that: The tapping probability includes a second probability that each tapping surface of the electronic device is tapped, and if the tapping probability is greater than a first threshold, performing a corresponding interactive operation includes: If the second probability corresponding to the target tapping surface is greater than the first threshold, the interactive operation corresponding to the target tapping surface is performed.

5. The method according to claim 1, characterized in that The performing knock detection based on the first motion data and the first audio data to obtain a knock detection result includes: The first motion data and the first audio data are input into a pre-trained knock detection model to obtain a knock detection result, wherein the knock detection model is trained based on sample motion data when the target device is knocked and collected sample audio data.

6. The method according to claim 5, characterized in that The knock detection model includes a first feature extraction module, a second feature extraction module and a classification module. The first motion data and the first audio data are input into the pre-trained knock detection model to obtain a knock detection result, including: Inputting the first motion data into the first feature extraction module to obtain a first motion feature output by the first feature extraction module; Inputting the first audio data into the second feature extraction module to obtain the first audio feature output by the second feature extraction module; The first motion feature and the first audio feature are input into the classification module to obtain a knock detection result.

7. The method according to claim 6, characterized in that The second feature extraction module includes a first feature extraction submodule and a second feature extraction submodule, and the first audio data is input into the second feature extraction module to obtain the first audio feature output by the second feature extraction module, including: Inputting the first audio data into the first feature extraction submodule to obtain Mel frequency spectrum coefficient features corresponding to the first audio data; The Mel-spectrum coefficient feature is input into the second feature extraction submodule, and the audio feature output by the second feature extraction submodule is obtained as the first audio feature.

8. The method according to any one of claims 1 to 7, characterized in that: The acquiring the first motion data of the electronic device and the collected first audio data includes: If the electronic device is in a held state, the audio collection device of the electronic device is controlled to collect audio data, and the first motion data of the electronic device and the first audio data collected by the audio collection device are obtained.

9. The method according to claim 8, characterized in that If the electronic device is in a held state, controlling an audio collection device of the electronic device to collect audio data, and obtaining first motion data of the electronic device and first audio data collected by the audio collection device, includes: If the electronic device is in a held state and a swinging operation on the electronic device is detected, the audio collection device of the electronic device is controlled to collect audio data, and the first motion data of the electronic device and the first audio data collected by the audio collection device are obtained.

10. The method according to claim 9, characterized in that The performing a corresponding interactive operation according to the tap detection result includes: According to detecting the first direction in which the electronic device is tilted when the swing operation is performed, and the tapping detection result, a corresponding interactive operation is performed.

11. The method according to any one of claims 1 to 7, characterized in that: The acquiring the first motion data of the electronic device and the collected first audio data includes: Acquire motion data of the electronic device in a current time period as the first motion data, where the current time period is a time period from a first moment before the current moment to the current moment; The audio data collected by the electronic device in the current time period is acquired as the first audio data.

12. The method according to any one of claims 1 to 7, characterized in that: The performing a corresponding interactive operation according to the tap detection result includes: According to the tapping detection result, the target near field communication NFC card is turned on, the target application is run, the volume is adjusted, the screen display brightness is adjusted, the communication mode is adjusted, the playback progress is adjusted, the message notification is hidden, the call is answered, or the call is hung up.

13. An interactive device, characterized in that: Applied to electronic equipment, the device comprises: a data acquisition module, a tapping recognition module and an interactive execution module, wherein: The data acquisition module is used to acquire the first motion data of the electronic device and the collected first audio data; The knock recognition module is used to perform knock detection based on the first motion data and the first audio data to obtain a knock detection result; The interaction execution module is used to execute corresponding interaction operations according to the tap detection result.

14. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 12.

Images