Device interaction method, smart glasses, readable storage medium and program product

Abstract

The present application relates to the technical field of wearable devices, and provides a device interaction method, smart glasses, a readable storage medium, and a program product, which are applied to smart glasses, wherein the smart glasses include at least one signal transmitter, at least one first signal receiver corresponding to the left eye area, and at least one second signal receiver corresponding to the right eye area; the method includes: controlling at least one signal transmitter to transmit a signal, and the transmitted signal reaches the user's eye area to form a reflection signal; receiving at least one first reflection signal through at least one first signal receiver, and receiving at least one second reflection signal through at least one second signal receiver; in response to the detected target blinking action, executing a target control operation corresponding to the target blinking action; wherein the target blinking action is determined based on at least one first reflection signal and the at least one second reflection signal. Thus, the user can interact with the smart glasses through the blinking action, thereby improving the user experience.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of wearable devices, and in particular, to a device interaction method, smart glasses, a readable storage medium, and a program product. Background Art

[0002] Smart glasses are a new type of wearable electronic device that is mainly worn on the user's eye area. At present, the interaction between users and smart glasses is mostly manual or voice commands. In this way, in scenarios where it is inconvenient for users to issue commands manually or by voice, the existing interaction methods cannot meet the user's interaction needs, resulting in a poor user experience. Summary of the invention

[0003] The embodiments of the present application provide a device interaction method, smart glasses, a readable storage medium and a program product, which mainly provide an interaction method in which a user can interact with a device through blinking actions, thereby improving the user's experience.

[0004] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, a device interaction method is provided, which is applied to smart glasses, wherein the smart glasses include at least one signal transmitter, at least one first signal receiver and at least one second signal receiver; the first signal receiver corresponds to the left eye area, and the second signal receiver corresponds to the right eye area; the method includes: controlling at least one signal transmitter to transmit a signal, and the transmitted signal reaches the user's eye area to form a reflection signal; receiving at least one first reflection signal through at least one first signal receiver, and receiving at least one second reflection signal through at least one second signal receiver; in response to a detected target blinking action, executing a target control operation corresponding to the target blinking action; wherein the target blinking action is determined based on at least one first reflection signal and at least one second reflection signal.

[0006] In this implementation, because different eye movements correspond to different orbicularis oculi muscle states, for example, when a user blinks, the orbicularis oculi muscle contracts and the upper and lower eyelids move, so different orbicularis oculi muscle states will form different reflection paths, and the frequencies of the reflection signals generated by different reflection paths will have different offsets, and then the target blinking action can be determined based on the changes in the reflection signals of different frequencies, and then the corresponding control operation can be triggered. In this way, device interaction can be achieved through the user's blinking action, so that the device interaction method is no longer limited to manual or voice, so as to meet the user's interaction needs in various scenarios and improve the user experience.

[0007] In a possible implementation of the first aspect, the smart glasses also include a frame, a left lens, and a right lens; wherein the frame is used to fix the left lens and the right lens; at least one signal transmitter is arranged on the frame near the left lens or the right lens; at least one first signal receiver is arranged on the frame near the left lens; and at least one second signal receiver is arranged on the frame near the right lens.

[0008] In a possible implementation of the first aspect, the smart glasses include four signal transmitters, two first signal receivers and two second signal receivers; wherein, two of the four signal transmitters are arranged on the frame close to the left lens and form a diagonal line, and the other two signals are arranged on the frame close to the right lens and form a diagonal line; the two first signal transmitters are arranged on the frame close to the left lens and form a diagonal line, and the two second signal receivers are arranged on the frame close to the right lens and form a diagonal line.

[0009] In a possible implementation manner of the first aspect, the signal transmitter is an ultrasonic transmitter, the first signal receiver and the second signal receiver are both ultrasonic receivers; and the first reflection signal and the second reflection signal are both ultrasonic signals.

[0010] In a possible implementation manner of the first aspect, the signal transmitter is a millimeter wave transmitter, the first signal receiver and the second signal receiver are both millimeter wave receivers; and the first reflected signal and the second reflected signal are both millimeter wave signals.

[0011] In a possible implementation of the first aspect, the smart glasses are connected to an electronic device; in response to a target blinking action, a target control operation corresponding to the target blinking action is executed, including: an operation instruction corresponding to the target control operation is sent to the electronic device, and the electronic device responds to the operation instruction and executes the target control operation.

[0012] In this implementation, the smart glasses are connected to electronic devices so that the user can interact with the electronic devices through the blinking action of the smart glasses, so that the device interaction method is no longer limited to manual or voice, thereby meeting the user's interaction needs in various scenarios and improving the user experience.

[0013] In a possible implementation manner of the first aspect, the device interaction method may further include: performing Fourier transform processing on at least one first reflection signal and at least one second reflection signal, respectively, to obtain at least one first time-spectrum graph sequence and at least one second time-spectrum graph sequence; wherein the first time-spectrum graph sequence and the second time-spectrum graph sequence each include N time-spectrum graphs, N≥1, N is a positive integer, and N corresponds to the number of fragments into which the signal is divided by Fourier transform processing; merging at least one first time-spectrum graph sequence and at least one second time-spectrum graph sequence to obtain a merged time-spectrum graph; detecting the user's blinking action based on the merged time-spectrum graph to obtain a target blinking action.

[0014] In this implementation, by dividing the signal into segments and using a window to select different signal segments in each segment for Fourier transform processing, a time-frequency spectrum diagram that can simultaneously display the time domain and frequency domain of the signal can be obtained. Therefore, the frequency change of the reflected signal at different time points can be accurately understood through the time-frequency spectrum diagram, and the dynamic characteristics of the reflected signal can be analyzed to detect and determine the dynamic target blinking action, realize device interaction, and improve user experience.

[0015] In a possible implementation manner of the first aspect, the Fourier transform includes a short-time Fourier transform, a fast Fourier transform, a continuous Fourier transform, or a discrete Fourier transform.

[0016] In a possible implementation of the first aspect, Fourier transform processing is performed on at least one first reflected signal and at least one second reflected signal respectively to obtain at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence, including: filtering low-frequency signals in at least one first reflected signal and at least one second reflected signal respectively to obtain at least one first high-frequency signal and at least one second high-frequency signal; wherein the low-frequency signal is a signal less than a preset frequency, and the high-frequency signal is a signal greater than or equal to a preset frequency; Fourier transform processing is performed on at least one first high-frequency signal and at least one second high-frequency signal respectively to obtain at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence.

[0017] In this implementation, before Fourier transform processing is performed on the reflected signal, the low-frequency signal is first filtered to retain the high-frequency signal, thereby removing interference from low-frequency signals such as the environment.

[0018] In a possible implementation of the first aspect, low-frequency signals in at least one first reflected signal and at least one second reflected signal are filtered respectively to obtain at least one first high-frequency signal and at least one second high-frequency signal, including: using a high-pass filter to perform high-pass filtering on at least one first reflected signal and at least one second reflected signal respectively to obtain at least one first high-frequency signal and at least one second high-frequency signal; wherein the preset frequency is the cutoff frequency of the high-pass filter.

[0019] In this implementation, the low-frequency signal is filtered by a high-pass filter to retain the high-frequency signal, thereby avoiding interference from the low-frequency signal.

[0020] In a possible implementation of the first aspect, Fourier transform processing is performed on at least one first high-frequency signal and at least one second high-frequency signal respectively to obtain at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence, including: performing signal screening on at least one first high-frequency signal and at least one second high-frequency signal based on a target frequency band to obtain at least one first target signal and at least one second target signal; Fourier transform processing is performed on at least one first target signal and at least one second target signal respectively to obtain at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence.

[0021] In this implementation, signals of required frequency bands are screened, for example, frequency bands required by a classification model are screened, thereby ensuring the accuracy of blinking action detection.

[0022] In a possible implementation of the first aspect, the device interaction method may further include: before merging the time-spectrogram sequences, performing static information elimination processing on N time-spectrograms in at least one first time-spectrogram sequence and at least one second time-spectrogram sequence, respectively, and retaining dynamic information corresponding to the blinking action in the N time-spectrograms.

[0023] In this implementation, static information is eliminated while retaining dynamic information corresponding to the blinking action, thereby removing useless signal data and improving detection accuracy.

[0024] In a possible implementation manner of the first aspect, each of the N time-spectrograms includes M columns of data, M≥1, N is a positive integer, and M is determined based on the size of the window used for Fourier transform processing, the length of the segment, and the moving step of the window; the window is used to move and select different signal segments, and each time the window moves, the signal segment included in the window is Fourier transformed once to form a corresponding column of data; based on this, static information elimination is performed on the N time-spectrograms in at least one first time-spectrogram sequence and at least one second time-spectrogram sequence, including: for each column of data of each time-spectrogram in the N time-spectrograms, a difference calculation is performed with the adjacent previous column of data, and the obtained difference time-spectrogram is the result of the static information elimination processing.

[0025] In a possible implementation manner of the first aspect, merging at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence to obtain a merged time-frequency spectrum, including: splicing the N time-frequency spectrums in each first time-frequency spectrum sequence respectively to obtain at least one first spliced ​​time-frequency spectrum corresponding to the at least one first time-frequency spectrum sequence; splicing the N time-frequency spectrums in each second time-frequency spectrum sequence respectively to obtain at least one second spliced ​​time-frequency spectrum corresponding to the at least one second time-frequency spectrum sequence; splicing at least one first spliced ​​time-frequency spectrum and at least one second spliced ​​time-frequency spectrum to obtain a merged time-frequency spectrum.

[0026] In this implementation, the time-frequency spectrograms of the reflected signals corresponding to different signal receivers are combined by splicing, which can avoid eliminating the frequency offset and ensure the integrity of the information.

[0027] In a possible implementation of the first aspect, when the first reflected signal is subjected to multiple Fourier transform processing based on windows of different sizes, there are first time-frequency spectrum sequences with different resolutions corresponding to windows of different sizes, and the first time-frequency spectrum sequences with different resolutions respectively correspond to a spliced ​​time-frequency spectrum, and the first spliced ​​time-frequency spectrum is obtained by splicing the spliced ​​time-frequency spectrum corresponding to each resolution; wherein the window is used to move and select different signal segments in the first reflected signal, and the Fourier transform is processed based on the signal segments; when the second reflected signal is subjected to multiple Fourier transform processing based on windows of different sizes, there are second time-frequency spectrum sequences with different resolutions corresponding to windows of different sizes, and the second time-frequency spectrum sequences with different resolutions respectively correspond to a spliced ​​time-frequency spectrum, and the second spliced ​​time-frequency spectrum is obtained by splicing the spliced ​​time-frequency spectrum corresponding to each resolution.

[0028] In this implementation, if time-frequency spectrograms of different resolutions are obtained by Fourier transforming windows of different sizes, the time-frequency spectrograms of different resolutions can also be combined by splicing, thereby providing richer information and improving the accuracy of detection.

[0029] In a possible implementation of the first aspect, a user's blinking action is detected based on a merged time-frequency spectrum to obtain a target blinking action, including: inputting the merged time-frequency spectrum into a trained classification model, and determining the target blinking action by the classification model; wherein, when the classification model is a multi-classification model and outputs multiple blinking action types, the blinking action type with the highest confidence is selected as the target blinking action.

[0030] In this implementation, the classification model is used to detect the blinking action, which can ensure the accuracy of the detection.

[0031] In a possible implementation of the first aspect, the device interaction method includes: acquiring simultaneously collected video data and signal data; the video data includes different blinking movements, and the signal data is a reflected signal received by a signal receiver; after aligning the timestamps of the video data and the signal data, based on a target time period, intercepting a corresponding signal segment from the signal data as a positive sample, and based on a non-target time period, intercepting a corresponding signal segment from the signal data as a negative sample; wherein the target time period is a time period including blinking movements, and the non-target time period is a time period not including blinking movements; and using the positive samples and negative samples to perform model training to obtain a classification model.

[0032] In a possible implementation of the first aspect, the target blinking action includes a single blink and L consecutive blinks; a single blink includes a single blink of both eyes, a single blink of the left eye, and a single blink of the right eye; L consecutive blinks include L consecutive blinks of both eyes, L consecutive blinks of the left eye, and L consecutive blinks of the right eye, where L≥2 and L is a positive integer; wherein different target blinking actions correspond to different or the same target control operations; the same target blinking action corresponds to one target control operation or to multiple different target control operations.

[0033] In a possible implementation of the first aspect, the target control operation corresponding to a single blink of both eyes includes a dry eye prompt; based on this, the target control operation corresponding to the target blinking action is executed, including: counting the blinking frequency of a single blink of both eyes; when the blinking frequency is less than a preset frequency, executing the dry eye prompt to prompt the user to blink.

[0034] In this implementation, when the user has dry eyes, the dry eye prompt can remind the user to actively blink, thereby preventing the dry eyes from getting worse and providing the user with a better user experience.

[0035] In a second aspect, the present application provides a pair of smart glasses, comprising: at least one signal transmitter, at least one first signal receiver, at least one second signal receiver, one or more processors and memories, wherein the signal transmitter, the first signal receiver, the second signal receiver and the memory are coupled to the processor; the signal transmitter is used to transmit signals, and the first signal receiver and the second signal receiver are used to receive reflected signals; one or more computer program codes are stored in the memory, and the computer program codes include computer instructions; when the processor executes the computer instructions, the smart glasses perform the following steps: control at least one signal transmitter to transmit signals, and the transmitted signals reach the user's eye area to form a reflected signal; receive at least one first reflected signal through at least one first signal receiver, and receive at least one second reflected signal through at least one second signal receiver; in response to a detected target blinking action, perform a target control operation corresponding to the target blinking action; wherein the target blinking action is determined based on at least one first reflected signal and at least one second reflected signal.

[0036] In a possible implementation of the second aspect, the smart glasses also include a frame, a left lens, and a right lens; wherein the frame is used to fix the left lens and the right lens; at least one signal transmitter is arranged on the frame near the left lens or the right lens; at least one first signal receiver is arranged on the frame near the left lens; and at least one second signal receiver is arranged on the frame near the right lens.

[0037] In a possible implementation of the second aspect, the smart glasses include four signal transmitters, two first signal receivers and two second signal receivers; wherein, two of the four signal transmitters are arranged on the frame close to the left lens and form a diagonal, and the other two signals are arranged on the frame close to the right lens and form a diagonal; the two first signal transmitters are arranged on the frame close to the left lens and form a diagonal, and the two second signal receivers are arranged on the frame close to the right lens and form a diagonal.

[0038] In a possible implementation manner of the second aspect, the signal transmitter is an ultrasonic transmitter, the first signal receiver and the second signal receiver are both ultrasonic receivers; and the first reflection signal and the second reflection signal are both ultrasonic signals.

[0039] In a possible implementation manner of the second aspect, the signal transmitter is a millimeter wave transmitter, the first signal receiver and the second signal receiver are both millimeter wave receivers; and the first reflected signal and the second reflected signal are both millimeter wave signals.

[0040] In a possible implementation of the second aspect, the smart glasses are connected to an electronic device; when the above-mentioned computer instructions are executed by the processor, the smart glasses also perform the following steps: sending an operation instruction corresponding to the target control operation to the electronic device, and the electronic device responds to the operation instruction to perform the target control operation.

[0041] In a possible implementation manner of the second aspect, when the above-mentioned computer instructions are executed by the processor, the smart glasses further perform the following steps: perform Fourier transform processing on at least one first reflection signal and at least one second reflection signal, respectively, to obtain at least one first time-spectrum graph sequence and at least one second time-spectrum graph sequence; wherein the first time-spectrum graph sequence and the second time-spectrum graph sequence each include N time-spectrum graphs, N≥1, N is a positive integer, and N corresponds to the number of fragments into which the signal is divided by Fourier transform processing; merge at least one first time-spectrum graph sequence and at least one second time-spectrum graph sequence to obtain a merged time-spectrum graph; detect the user's blinking action based on the merged time-spectrum graph to obtain a target blinking action.

[0042] In a possible implementation manner of the second aspect, the Fourier transform includes a short-time Fourier transform, a fast Fourier transform, a continuous Fourier transform, or a discrete Fourier transform.

[0043] In a possible implementation manner of the second aspect, when the above-mentioned computer instructions are executed by the processor, the smart glasses further perform the following steps: respectively filter the low-frequency signals in at least one first reflection signal and at least one second reflection signal to obtain at least one first high-frequency signal and at least one second high-frequency signal; wherein the low-frequency signal is a signal less than a preset frequency, and the high-frequency signal is a signal greater than or equal to the preset frequency; and respectively perform Fourier transform processing on at least one first high-frequency signal and at least one second high-frequency signal to obtain at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence.

[0044] In a possible implementation of the second aspect, when the above-mentioned computer instructions are executed by the processor, the smart glasses further perform the following steps: using a high-pass filter to perform high-pass filtering on at least one first reflection signal and at least one second reflection signal respectively to obtain at least one first high-frequency signal and at least one second high-frequency signal; wherein the preset frequency is the cutoff frequency of the high-pass filter.

[0045] In a possible implementation manner of the second aspect, when the above-mentioned computer instructions are executed by the processor, the smart glasses further perform the following steps: performing signal screening on at least one first high-frequency signal and at least one second high-frequency signal based on the target frequency band to obtain at least one first target signal and at least one second target signal; performing Fourier transform processing on at least one first target signal and at least one second target signal to obtain at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence.

[0046] In a possible implementation manner of the second aspect, when the above-mentioned computer instructions are executed by the processor, the smart glasses further perform the following steps: before merging the time-spectrogram sequences, performing static information elimination processing on N time-spectrograms in at least one first time-spectrogram sequence and at least one second time-spectrogram sequence, respectively, and retaining dynamic information corresponding to the blinking action in the N time-spectrograms.

[0047] In a possible implementation of the second aspect, each of the N time-frequency spectrograms includes M columns of data, M≥1, N is a positive integer, and M is determined based on the size of the window used for Fourier transform processing, the length of the segment, and the moving step of the window; the window is used to move and select different signal segments, and each time the window moves, the signal segment included in the window is Fourier transformed once to form a corresponding column of data; when the above-mentioned computer instructions are executed by the processor, the smart glasses also perform the following steps: for each column of data of each time-frequency spectrogram of the N time-frequency spectrograms, the difference between it and the adjacent previous column of data is calculated, and the obtained difference time-frequency spectrogram is the result of static information elimination processing.

[0048] In a possible implementation manner of the second aspect, when the above-mentioned computer instructions are executed by the processor, the smart glasses further perform the following steps: respectively splicing the N time-frequency spectrum graphs in each first time-frequency spectrum graph sequence to obtain at least one first spliced ​​time-frequency spectrum graph corresponding one-to-one to at least one first time-frequency spectrum graph sequence; respectively splicing the N time-frequency spectrum graphs in each second time-frequency spectrum graph sequence to obtain at least one second spliced ​​time-frequency spectrum graph corresponding one-to-one to at least one second time-frequency spectrum graph sequence; and splicing at least one first spliced ​​time-frequency spectrum graph and at least one second spliced ​​time-frequency spectrum graph to obtain a merged time-frequency spectrum graph.

[0049] In a possible implementation of the second aspect, when the first reflected signal is subjected to multiple Fourier transform processing based on windows of different sizes, there are first time-frequency spectrum sequences with different resolutions corresponding to windows of different sizes, and the first time-frequency spectrum sequences with different resolutions respectively correspond to a spliced ​​time-frequency spectrum, and the first spliced ​​time-frequency spectrum is obtained by splicing the spliced ​​time-frequency spectrum corresponding to each resolution; wherein the window is used to move and select different signal segments in the first reflected signal, and the Fourier transform is processed based on the signal segments; when the second reflected signal is subjected to multiple Fourier transform processing based on windows of different sizes, there are second time-frequency spectrum sequences with different resolutions corresponding to windows of different sizes, and the second time-frequency spectrum sequences with different resolutions respectively correspond to a spliced ​​time-frequency spectrum, and the second spliced ​​time-frequency spectrum is obtained by splicing the spliced ​​time-frequency spectrum corresponding to each resolution.

[0050] In a possible implementation of the second aspect, when the above-mentioned computer instructions are executed by the processor, the smart glasses further perform the following steps: inputting the merged time spectrum diagram into a trained classification model, and determining the target blinking action by the classification model; wherein, when the classification model is a multi-classification model and outputs multiple blinking action types, selecting the blinking action type with the highest confidence as the target blinking action.

[0051] In a possible implementation of the second aspect, when the above-mentioned computer instructions are executed by the processor, the smart glasses also perform the following steps: acquiring simultaneously collected video data and signal data; the video data includes different blinking actions, and the signal data is a reflected signal received by the signal receiver; after aligning the timestamps of the video data and the signal data, based on the target time period, intercepting the corresponding signal segment from the signal data as a positive sample, and based on the non-target time period, intercepting the corresponding signal segment from the signal data as a negative sample; wherein the target time period is a time period including the blinking action, and the non-target time period is a time period not including the blinking action; using the positive samples and the negative samples to perform model training to obtain a classification model.

[0052] In a possible implementation of the second aspect, the target blinking action includes a single blink and L consecutive blinks; a single blink includes a single blink of both eyes, a single blink of the left eye, and a single blink of the right eye; L consecutive blinks include L consecutive blinks of both eyes, L consecutive blinks of the left eye, and L consecutive blinks of the right eye, where L≥2 and L is a positive integer; wherein different target blinking actions correspond to different or the same target control operations; the same target blinking action corresponds to one target control operation or to multiple different target control operations.

[0053] In a possible implementation of the second aspect, the target control operation corresponding to a single blink of both eyes includes a dry eye prompt; when the above-mentioned computer instructions are executed by the processor, the smart glasses also perform the following steps: counting the blinking frequency of a single blink of both eyes; when the blinking frequency is less than a preset frequency, executing a dry eye prompt to prompt the user to blink.

[0054] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor in smart glasses, the smart glasses execute a device interaction method as in the first aspect and any possible implementation thereof.

[0055] In a fourth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the device interaction method of the first aspect and any possible implementation thereof. The computer may be the above-mentioned smart glasses.

[0056] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute a device interaction method as described in any one of the first aspects.

[0057] It can be understood that the beneficial effects that can be achieved by the smart glasses of any possible implementation of the second aspect, the computer-readable storage medium of the third aspect, the computer program product of the fourth aspect, and the chip of the fifth aspect can be referred to as the beneficial effects in the first aspect and any possible implementation thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of the appearance of a pair of smart glasses provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of a single eye blink provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of the connection between smart glasses and electronic devices provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the structure of a pair of smart glasses provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of the distribution of a signal transmitter and a signal receiver provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of the distribution of another signal transmitter and signal receiver provided in an embodiment of the present application;

[0064] Figure 7 A schematic diagram of a flow chart of a device interaction method provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of the principle of a short-time Fourier transform provided in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of the principle of eliminating static information provided by an embodiment of the present application;

[0067] Fig.10 A schematic diagram of the principle of splicing a time-frequency spectrum provided in an embodiment of the present application;

[0068] Fig.11 A schematic diagram of a data collection scenario provided in an embodiment of the present application;

[0069] Fig.12 A schematic diagram of a process for determining a target blinking action provided in an embodiment of the present application;

[0070] Fig.13 A schematic diagram of a process for executing a target control operation provided in an embodiment of the present application;

[0071] Fig.14 Application scenario of a device interaction method provided in an embodiment of the present application Figure 1 ;

[0072] Fig.15 Application scenario of a device interaction method provided in an embodiment of the present application Figure 2 ;

[0073] Fig.16 A structural block diagram of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical scheme of the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Among them, in the description of the embodiment of the present application, the terms used in the following embodiments are only for the purpose of describing a specific embodiment, and are not intended to limit the present application. In addition, in order to facilitate the clear description of the technical scheme of the embodiment of the present application, in the embodiment of the present application, if the words "first", "second" and the like are used to distinguish the same items or similar items with basically the same functions and effects. It can be understood by those skilled in the art that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit necessarily different. And, in the description of the embodiment of the present application, unless otherwise specified, the meaning of "multiple" means two or more.

[0075] Smart glasses are a new type of wearable electronic device that is mainly worn on the user's eye area. For example, Figure 1 A schematic diagram of the appearance of smart glasses is shown.

[0076] In some embodiments, the smart glasses can be worn like traditional glasses, supported and fixed on the user's eye area by the ears, nose, etc. Figure 1 The smart glasses shown in (1) can be supported and fixed on the user's eye area through the ears, nose and other parts.

[0077] In other embodiments, the smart glasses (such as virtual reality (VR) glasses) can also be worn by fixing them on the user's eye area through head-mounted wear. Figure 1 The smart glasses shown in (2) can be fixed on the user's eye area by wearing it on the head.

[0078] Understandably, Figure 1 The two smart glasses shown are only examples of the embodiments of the present application. Based on the design of the actual smart glasses, the wearing method (i.e., the method of fixing the smart glasses on the eye area), the appearance, etc. of the smart glasses may be different. Figure 1 There is no limitation on the wearing method and appearance of smart glasses.

[0079] Because of the special wearing and usage mode / attributes of smart glasses, smart glasses can provide users with a new type of device interaction experience. In some embodiments, users can directly interact with smart glasses to use the functions on smart glasses. In other embodiments, smart glasses can also be connected to electronic devices such as mobile phones and tablets. Furthermore, when it is inconvenient for users to directly operate the connected electronic devices, they can send instructions to the electronic devices to control the electronic devices by interacting with smart glasses, so that users can use the functions provided by the electronic devices through smart glasses. In other words, users can interact with electronic devices through smart glasses.

[0080] For example, taking a mobile phone as an example, when it is inconvenient for the user to check mobile phone messages or answer calls, the user can use smart glasses to control the electronic device to read mobile phone messages and answer calls. It should be noted that based on the actual interaction design, based on the connection between smart glasses and electronic devices, the control of electronic devices is not limited to reading mobile phone messages and answering calls. For example, smart glasses can also be used to control electronic devices to set alarms, turn off alarms, etc., or some other mobile phone display controls can be implemented, and the embodiments of the present application do not impose any restrictions on this.

[0081] However, whether interacting directly with smart glasses or interacting with electronic devices by reversely controlling the electronic devices through smart glasses, most of the existing interaction methods require users to manually issue commands on the smart glasses side (for example, by manual clicking, manual sliding, etc.), or, commands can be issued through voice.

[0082] Thus, in scenarios where it is inconvenient for users to issue commands manually or by voice, and for some disabled people, the existing interaction methods cannot meet the interaction needs of users, resulting in a poor user experience.

[0083] Based on this, in order to meet the user's interaction needs in various scenarios (such as scenarios where it is inconvenient to interact manually or by voice) and ensure the user experience, an embodiment of the present application provides a device interaction method.

[0084] The device interaction method provided in the embodiment of the present application is applied to smart glasses. The appearance of the smart glasses can be referred to as Figure 1 The smart glasses shown in (1) or (2) in FIG. In a specific embodiment, the appearance of the smart glasses can be Figure 1 The smart glasses shown in (1) in the embodiment of the present application are provided with a signal transmitter and a signal receiver.

[0085] The signal transmitter provided in the smart glasses is used to transmit a signal to the user's eye area. At the same time, the signal transmitted by the signal transmitter reaches / contacts the user's eye area and forms a corresponding reflection signal. The reflected signal is received by the signal receiver provided in the smart glasses. That is, the eye area can receive the direct signal transmitted by the signal transmitter, and the signal receiver can receive the reflected signal reflected by the eye area.

[0086] It is understandable that after the signal transmitter is fixedly installed in the smart glasses, the position of the signal transmitter will not change, so the frequency of the direct signal emitted by the signal transmitter usually does not change significantly. In other words, the signal directly emitted to the user's eye area does not change significantly because the position and distance of the device do not change.

[0087] However, although the signal receiver is fixed in the smart glasses, the position of the signal receiver will not change. However, the difference is that the frequency of the reflected signal received by the signal receiver will change. The reason is that the reflected signal received by the signal receiver is a signal reflected by the user's eye area. At the same time, when the user has eye movements, such as blinking, the eye area will change accordingly due to blinking. Then, the reflection path of the reflected signal will change due to eye movements such as blinking, so that the frequency of the reflected signal will have a certain deviation, that is, the reflected signal received by the signal receiver will have a frequency deviation due to the blinking action.

[0088] For example, Figure 2 A schematic diagram of a single eye blink is shown.

[0089] refer to Figure 2 , Figure 2 (1) is a schematic diagram of no blinking (i.e. eyes open). Figure 2 (2) shows a schematic diagram of a single eye blink (i.e., with eye closure). Figure 2 As can be seen in (1) and (2), when blinking (including closing and opening the eyes), the orbicularis oculi muscle contracts, the upper and lower eyelids move, and the upper eyelid moves more than the lower eyelid. The orbicularis oculi muscle is one of the facial expression muscles, which is ring-shaped and distributed around the eye sockets. It is mainly responsible for closing the eyes.

[0090] That is to say, because blinking changes the eye movement, the frequency of the reflected signal received by the signal receiver will be offset to a certain extent. In addition, because the amplitude caused by different eye movements is different, the orbicularis oculi muscle has different corresponding states, so the offset of the reflected signal will also be different.

[0091] For example, through Figure 2 The deviation of the signal reflected by the eye movement (i.e., eye opening) shown in (1) is related to the Figure 2 The offset of the signal reflected back from the eye movement (i.e. closing the eyes) shown in (2) is different.

[0092] Therefore, by analyzing the characteristics of the continuous reflection signal, it is possible to detect whether the user has blinked. Therefore, the device interaction method provided in the embodiment of the present application is mainly implemented through the user's blinking action. Specifically:

[0093] The smart glasses control the signal transmitter to transmit a signal to the user's eye area, and then receive the reflected signal reflected back from the user's eye area through the signal receiver. Then, the smart glasses detect and determine the user's blinking action through the reflected signal, thereby determining the target blinking action. Furthermore, different blinking actions are pre-regarded as different operation instructions, and different operation instructions can be pre-defined with different control operations. Furthermore, after the smart glasses detect the user's blinking action, it is equivalent to receiving the operation instruction issued by the user through the blinking action. Finally, the smart glasses can respond to the user's operation instruction and execute the control operation corresponding to the target blinking action (i.e., the target control operation), thereby realizing device interaction.

[0094] It can be seen that by using the user's blinking action to achieve device interaction, the device interaction mode is no longer limited to manual or voice, so that the user's interaction needs in various scenarios can be met and the user experience can be improved. For example, after wearing smart glasses, even if the user is in a scenario where it is not convenient to send commands manually or by voice, or for some disabled people, the user can send commands by blinking, thereby achieving device interaction and ensuring the user experience.

[0095] In some embodiments, the embodiments of the present application implement device interaction through blinking actions, which may be direct interaction with smart glasses. That is, the user can use the functions on the smart glasses through different blinking actions.

[0096] In other embodiments, the device interaction achieved through the blinking action can also be to control the electronic device to achieve interaction with the electronic device. That is to say, in the embodiments of the present application, different control operations corresponding to different blinking actions can not only directly act on the smart glasses themselves, but also act on the electronic devices connected to the smart glasses through the smart glasses.

[0097] For example, Figure 1 Take the smart glasses and mobile phone shown in (1) as an example. Figure 3 A schematic diagram of connecting smart glasses to electronic devices is shown. Figure 3 As shown in (1), the smart glasses can be connected to the mobile phone via a data cable. Figure 3 As shown in (2), smart glasses can also be wirelessly connected to mobile phones via Bluetooth / Wi-Fi and other methods.

[0098] For example, Figure 4 A structural schematic diagram of smart glasses is shown.

[0099] like Figure 4As shown, the smart glasses 40 may include one or more processors 41, one or more memories 42, a power supply component 43, an interface component 44, a signal transmitter 45, a signal receiver 46, a wireless communication module 47, etc.

[0100] The processor 41 may include one or more processing units. The processor 41 may generate an operation control signal according to an instruction operation code and a timing signal to complete the control of fetching and executing instructions.

[0101] In the embodiment of the present application, the processor 41 in the smart glasses can control the signal transmitter 45 to transmit a signal, and the processor 41 can perform feature analysis on the reflected signal received by the signal receiver 46 to detect and determine the user's blinking action. At the same time, different blinking actions represent different operation instructions, and different operation instructions correspond to different control operations. Furthermore, the processor 41 can respond to the user's blinking action and execute corresponding control operations to achieve device interaction.

[0102] The memory 42 can be used to store computer executable program code, which includes instructions. In some embodiments, the memory 42 can be set in the processor 41 to store instructions and data. In some embodiments, the memory 42 in the processor 41 can be a cache memory. The memory can save instructions or data that the processor has used or uses frequently. If the processor 41 needs to use the instruction or data, it can be directly called from the memory 42. Repeated access is avoided, the waiting time of the processor is reduced, and the efficiency of the system is improved.

[0103] The power supply component 43 is used to supply power to the smart glasses 40. The interface component 44 is used to provide various types of interfaces, such as a network interface for connecting to a network. The interface component 44 may also provide a universal serial bus (USB) interface for connecting to peripheral devices. Exemplarily, the smart glasses 40 may be connected to an electronic device via a USB interface provided by the interface component, that is, the smart glasses 40 may be connected to the electronic device by wire.

[0104] The wireless communication module 47 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which can be applied to the smart glasses 40. For example, the smart glasses 40 can be connected to the electronic device through the wireless communication module 47, that is, the smart glasses 40 are wirelessly connected to the electronic device.

[0105] The signal transmitter 45 is used to transmit signals. The signal receiver 46 is used to receive reflected signals. In some embodiments, the smart glasses 40 include at least one signal transmitter 45, and the smart glasses 40 include at least two signal receivers 46. The reason why there are at least two signal receivers 46 is that they need to be used to receive reflected signals from the left eye area and the right eye area respectively. In this way, it is convenient to determine whether the user's blinking action is a double eye blink or a single eye blink through the reflected signal analysis, and in the case of a single eye blink, it can be accurately determined whether it is a left eye single eye blink or a right eye single eye blink.

[0106] For the convenience of distinction and scheme description, the at least one signal receiver 46 corresponding to the left eye area in the present embodiment of the application will be referred to as the first signal receiver, and the at least one signal receiver 46 corresponding to the right eye area in the present embodiment of the application will be referred to as the second signal receiver. That is, the at least one first signal receiver is used to receive the reflected signal of the left eye area. The at least one second signal receiver is used to receive the reflected signal of the right eye area.

[0107] In some embodiments, the signal transmitter and the signal receiver can be set on the frame of the smart glasses, and the specific setting position selected on the frame needs to be a position that can transmit signals to the user's eye area and receive reflected signals.

[0108] For example, the eye area is usually the lens of the smart glasses, so the position of the lens is the best position to transmit signals to the eye area and receive signals reflected from the eye area. However, in order not to damage the integrity of the lens and affect the user's visual experience, the signal transmitter and the signal receiver can be set on the frame close to the lens.

[0109] For example, Figure 1 Taking the smart glasses shown in (1) as an example, and taking a signal transmitter, a first signal receiver and a second signal receiver as an example, Figure 5 A schematic diagram of the distribution of a signal transmitter and a signal receiver is shown.

[0110] like Figure 5 As shown, the frame of the smart glasses includes a lens frame and temples (also called temples). The lens frame is used to fix the lens, which includes a left lens and a right lens. The temples are used to fix and support the lens frame.

[0111] refer to Figure 5 As shown in (1), the signal transmitter may be disposed on the lens frame between the left lens and the right lens. Figure 5 As shown in (2), the signal transmitter may also be disposed on the lens frame near the left lens. Figure 5 As shown in (3), the signal transmitter can also be arranged on the temple close to the left lens. Figure 5 The signal transmitter arrangement positions shown in (1), (2) and (3) are only three examples of the embodiments of the present application. Figure 5 There is no limitation on the location of the signal transmitter. For example, the signal transmitter may be located on the lens frame near the right lens. Alternatively, the signal transmitter may be located on the temple near the right lens.

[0112] refer to Figure 5 In (1), (2) and (3), in the embodiment of the present application, because the first signal receiver is a signal receiver corresponding to the left eye area, and the second signal receiver is a signal receiver corresponding to the right eye area. Then, the first signal receiver can be set on the lens frame close to the left lens, and the second signal receiver can be set on the lens frame close to the right lens. It should be noted that Figure 5 The arrangement positions of the first signal receiver and the second signal receiver shown in (1), (2) and (3) are only three examples of the embodiments of the present application. Figure 5 There is no limitation on the arrangement positions of the first signal receiver and the second signal receiver. For example, the first signal receiver may be arranged on the temple near the left lens, and the second signal receiver may be arranged on the temple near the right lens.

[0113] Understandably, because the temple area away from the lens, e.g. Figure 5The dotted elliptical area shown in (1) is the temple area that mainly serves to fix the smart glasses to the user's eye area. Therefore, if the signal transmitter and the signal receiver are set in this temple area, the signal transmitter may not be able to transmit signals to the user's eye area because it is far away from the eye area, or because of the occlusion of the ears and face, and the signal receiver will not be able to receive the reflected signal reflected from the eye area. Therefore, the signal transmitter and the signal receiver cannot be set in this temple area.

[0114] In a specific embodiment, the smart glasses may include four signal transmitters and four signal receivers. The four signal receivers include two first signal receivers and two second signal receivers. The four signal transmitters and four signal receivers, i.e., a total of eight devices, may be respectively arranged at the frame positions of eight diagonal points of two lenses.

[0115] For example, Figure 1 Take the smart glasses shown in (1) as an example. Figure 6 A schematic diagram of the distribution of 8 devices is shown. Figure 6 in (1) and (2), Figure 6 Neutral (1) and Figure 6 (2) shows two distribution modes. It can be understood that Figure 6 The two distributions shown are just two examples of where to place the eight devices. Figure 6 There is no limitation on the arrangement positions of the eight devices.

[0116] In a specific embodiment, the signal transmitter may be an ultrasonic transmitter, and the signal receiver may also be an ultrasonic receiver. Based on this, the direct signal emitted by the signal transmitter in the smart glasses is an ultrasonic signal, and the reflected signal received by the signal receiver is also an ultrasonic signal.

[0117] In a specific embodiment, the signal transmitter may be a millimeter wave transmitter, and the signal receiver may be a millimeter wave receiver. Based on this, the direct signal emitted by the signal transmitter in the smart glasses is a millimeter wave signal, and the reflected signal received by the signal receiver is also a millimeter wave signal.

[0118] It should be noted that the types of signal transmitters and signal receivers are not limited to the ultrasonic and millimeter waves in the above specific embodiments. As long as the transmitted signal can form a reflected signal in the eye area, and the reflected signal will change the reflection path due to eye movement to form a signal type with frequency deviation, it can be applied to the embodiments of the present application.

[0119] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on smart glasses. In other embodiments of the present application, smart glasses may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware. Exemplarily, smart glasses may also include speakers, receivers, microphones, buttons, etc. In some embodiments, the electronic devices connected to the smart glasses may include mobile phones, foldable electronic devices, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (ultra-mobile personal computers, UMPC), netbooks, cellular phones, personal digital assistants (personal digital assistants, PDAs), augmented reality (augmented reality, AR) devices, virtual reality (virtual reality, VR) devices, artificial intelligence (artificial intelligence, AI) devices, wearable devices, vehicle-mounted devices, smart home devices, smart city devices, car machines, large-screen devices, or at least one of interactive routing devices. It is understandable that the embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.

[0120] In some embodiments, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, an antenna, an antenna, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, an earphone interface, a sensor module, a button, a motor, an indicator, a camera module, a display screen, and a subscriber identification module (SIM) card interface, etc. Among them, the sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0121] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0122] The device interaction method proposed in the embodiment of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the device interaction method in the following embodiments can be implemented in the smart glasses 40 having the above hardware structure. At the same time, for the convenience of describing the solution, the device interaction method will be mainly described below in the embodiment of the present application using ultrasonic signals as an example.

[0123] Figure 7 A schematic diagram of a device interaction method is shown. Figure 7 The device interaction method provided in the embodiment of the present application includes steps S701-S707.

[0124] S701, the smart glasses transmit a signal to the eye area of ​​the user.

[0125] After the smart glasses are fixed on the user's eye area, the smart glasses can control the signal transmitter to transmit a signal to the user's eye area. In some embodiments, the signal transmitter can be an ultrasonic transmitter, so the smart glasses can control the ultrasonic transmitter to transmit an ultrasonic signal to the user's eye area.

[0126] Based on actual needs, at least one ultrasonic transmitter can be provided in the smart glasses. The smart glasses can control the at least one ultrasonic transmitter to transmit ultrasonic signals to the user's eye area. In a specific embodiment, if an ultrasonic transmitter is provided in the smart glasses, the smart glasses can control the ultrasonic transmitter to transmit ultrasonic signals to the user's eye area.

[0127] In another specific embodiment, if two or more ultrasonic transmitters are provided in the smart glasses, the smart glasses control the two or more ultrasonic transmitters to simultaneously transmit ultrasonic signals to the user's eye area. For example, if four ultrasonic transmitters are provided in the smart glasses, the smart glasses can control the four ultrasonic transmitters to simultaneously transmit ultrasonic signals to the user's eye area.

[0128] In some embodiments, when it is necessary to control two or more ultrasonic transmitters to transmit ultrasonic signals to the eye area of ​​the user, the ultrasonic signals transmitted by different ultrasonic transmitters may be signals of the same frequency. Alternatively, the ultrasonic signals transmitted by different ultrasonic transmitters may also be signals of different frequencies. Among them, signals of the same frequency refer to signals with the same frequency / frequency band. Signals of different frequencies refer to signals with different frequencies.

[0129] It is understandable that the higher the frequency of the signal, the stronger the anti-interference ability of the signal. Therefore, based on the actual signal anti-interference requirements, each ultrasonic transmitter can be controlled to transmit ultrasonic signals of different or same frequencies. Exemplarily, the frequency of the ultrasonic signal emitted by the ultrasonic transmitter can be 18 khz, 20 khz, or even higher 40 khz, and the embodiments of the present application do not impose any limitation on this.

[0130] S702: The smart glasses receive the reflected signal.

[0131] After the smart glasses control the signal transmitter to transmit a signal to the user's eye area, the transmitted signal will form a corresponding reflection signal when it reaches or contacts the user's eye area. Then, the smart glasses can receive the reflection signal through the signal receiver. In some embodiments, the signal receiver can be an ultrasonic receiver. That is, the ultrasonic receiver can receive the reflection signal formed by the ultrasonic signal transmitted by the ultrasonic transmitter.

[0132] In an embodiment of the present application, the ultrasonic receiver includes a first signal receiver and a second signal receiver. The first signal receiver is used to receive a reflection signal reflected from the left eye area, and the second signal receiver is used to receive a reflection signal reflected from the right eye area. Then, in an embodiment of the present application, the reflection signal includes a reflection signal received by the first signal receiver and a reflection signal received by the second signal receiver. Hereinafter, for ease of distinction, the reflection signal received by the first signal receiver is referred to as the first reflection signal in the embodiment of the present application, and the reflection signal received by the second signal receiver is referred to as the second reflection signal in the embodiment of the present application.

[0133] It should be noted that, based on actual needs, at least one first signal receiver and at least one second signal receiver can be set in the smart glasses, and the embodiments of the present application do not impose any limitation on this.

[0134] S703, the smart glasses screen the reflected signal based on the target frequency band to obtain the target signal.

[0135] The target frequency band is a frequency band predefined for blinking action detection. In a specific embodiment, in the scenario where a classification model is selected to detect blinking action, the target frequency band is the frequency band used when the classification model is trained for blinking action detection. The target frequency band can be selected according to actual needs, and the embodiment of the present application does not impose any limitation on this.

[0136] Therefore, after obtaining the reflection signal, in order to facilitate the blink detection and ensure the accuracy of the blink detection, the smart glasses first need to filter the reflection signal based on the predefined target frequency band to obtain the target signal. In other words, the target signal is the filtered signal that belongs to the target frequency band.

[0137] The reflected signal includes a first reflected signal and a second reflected signal, that is, the smart glasses first need to screen the first reflected signal and the second reflected signal respectively based on a predefined target frequency band, so as to obtain a first target signal corresponding to the first reflected signal and a second target signal corresponding to the second reflected signal.

[0138] Of course, it is understandable that if the signal analysis uses full-band signals, such as the classification model training uses full-band data for training, then the reflected signal may not be screened.

[0139] In some embodiments, due to external interference such as the environment, the received reflected signal may contain some unnecessary interference signals. For example, if the signal transmitter and the signal receiver are ultrasonic transmitters and timeout receivers, then there will be some environmental audio interference in the ultrasonic signal received by the ultrasonic receiver. Because the signal is interfered, the accuracy of blink detection may be affected.

[0140] Therefore, in order to reduce signal interference and to facilitate accurate extraction of the required target frequency band signal when signal screening is required, the embodiment of the present application is based on the characteristic that ambient audio is usually an audible low-frequency signal, while ultrasonic signals are inaudible high-frequency signals. Before screening the target frequency band signal, the smart glasses can also filter the reflected signal for low-frequency signals. The purpose of low-frequency signal filtering is to retain the high-frequency signal in the reflected signal, that is, to remove the interference of low-frequency signals such as ambient audio by filtering the low-frequency signal, so as to obtain a pure ultrasonic signal.

[0141] The low-frequency signal filtering may specifically include: the smart glasses perform low-frequency signal filtering on the first reflection signal and the second reflection signal respectively, thereby obtaining the corresponding first high-frequency signal and the second high-frequency signal. In some embodiments, the low-frequency signal filtering may filter out signals less than a preset frequency, while retaining signals greater than or equal to the preset frequency.

[0142] Then, after the low-frequency signal is filtered, the smart glasses filter the signal of the target frequency band based on the filtered high-frequency signal to obtain the target signal. That is, the smart glasses filter the first high-frequency signal and the second high-frequency signal based on the target frequency band to obtain the corresponding first target signal and second target signal.

[0143] In a specific embodiment, low-frequency signal filtering can be performed by high-pass filtering. High-pass filtering is a signal processing technology that allows high-frequency signals to pass through while attenuating low-frequency signals. In other words, the main feature of high-pass filtering is that high-frequency signals can be retained normally, while low-frequency signals below a set critical value (i.e., a preset frequency), such as a low-frequency signal below a cutoff frequency, are blocked or weakened. Therefore, the embodiment of the present application can perform high-pass filtering on the reflected signal by using a high-pass filter, thereby filtering out the low-frequency signal in the reflected signal and retaining the high-frequency signal in the reflected signal.

[0144] That is, in the embodiment of the present application, the above S703 may include: the smart glasses use a high-pass filter to perform high-pass filtering on the first reflection signal and the second reflection signal to obtain a first high-frequency signal and a second high-frequency signal. The smart glasses perform signal screening on the first high-frequency signal and the second high-frequency signal based on the target frequency band to obtain a first target signal and a second target signal.

[0145] S704, the smart glasses transform and process the target signal to obtain a time-frequency spectrum diagram.

[0146] After the smart glasses obtain the target signal, in order to facilitate analysis based on the target signal to detect the user's blinking action, the smart glasses need to further transform the target signal to obtain a time-frequency spectrum of the target signal.

[0147] A time-spectrogram is a graph that can display both the time domain and the frequency domain of a signal. In other words, a time-spectrogram is two-dimensional and provides information about the distribution of a signal at different times (i.e., in the time domain) and at different frequencies (i.e., in the frequency domain). Furthermore, a time-spectrogram can be used to understand the changes in the frequency components of a signal at different time points, and thus to analyze the dynamic characteristics of the signal. Therefore, a time-spectrogram can be understood as a spectrogram that is more suitable for analyzing time-varying signals (i.e., signals whose dependent variables change over time).

[0148] In some embodiments, the transformation process may be implemented by short-time Fourier transform (STFT), and in other embodiments, the transformation process may be implemented by fast Fourier transform, continuous Fourier transform, discrete Fourier transform or other Fourier transform methods.

[0149] The short-time Fourier transform (STFT) is a Fourier transform with fixed segmentation and windowing, which can also be called a window Fourier transform. Simply put, the short-time Fourier transform (STFT) mainly divides the signal into multiple segments based on time, and performs Fourier transform on the signal segments in the window by sliding a fixed-size window on the signal in each segment. It can be understood that the short-time Fourier transform (STFT) will slide a window to select a part of the signal segment in each segmented segment, and then perform Fourier transform on the signal segment selected by the window. Therefore, the short-time Fourier transform (STFT) of the signal can directly obtain a time-frequency spectrum including the time domain and the frequency domain.

[0150] It is understandable that the size of the window selected by STFT (i.e., window length, such as 128, 256 or 512) will affect the resolution of time and frequency. That is, the key parameters of the short-time Fourier transform STFT include the window length. The larger the window length, the higher the frequency resolution and the lower the time resolution. The smaller the window length, the lower the frequency resolution, but the higher the time resolution. Furthermore, by selecting different window lengths for short-time Fourier transform, time-frequency spectrograms with different resolutions can be obtained.

[0151] However, the conventional processing method of non-short-time Fourier transforms such as fast Fourier transform, continuous Fourier transform, and discrete Fourier transform is to process the entire signal, which usually does not directly involve the signal segmentation and windowing processing. Therefore, although the signal can be represented in the frequency domain by transforming through fast Fourier transform, continuous Fourier transform, discrete Fourier transform, etc., it is usually not related to the representation of the time dimension. In other words, if the signal is directly transformed by non-short-time Fourier transforms such as fast Fourier transform, continuous Fourier transform, and discrete Fourier transform, it may only be possible to obtain a spectrum diagram including a one-dimensional frequency domain. Therefore, in an embodiment of the present application, if it is necessary to use non-short-time Fourier transform methods such as fast Fourier transform, continuous Fourier transform, and discrete Fourier transform to obtain a time-frequency spectrum diagram, it is necessary to add segmentation and windowing processing on the basis of traditional fast Fourier transform, continuous Fourier transform, and discrete Fourier transform.

[0152] Based on this, in a specific embodiment, the acquisition of the time-frequency spectrum can be directly obtained by performing a short-time Fourier transform (STFT) process on the target signal. In addition, in the embodiment of the present application, the blinking action is an action that changes over time, so the reflected signal in the embodiment of the present application is essentially a time-varying signal. For time-varying signals, time domain information is also important information. Therefore, compared to the spectrum diagram that only includes one-dimensional frequency domain information, the embodiment of the present application is more suitable for detecting the user's blinking action through the time-frequency spectrum diagram.

[0153] In summary, the above S704 may include: the smart glasses perform short-time Fourier transform STFT processing on the target signal to obtain a time-frequency spectrum corresponding to the target signal. Wherein, the target signal includes a first target signal and a second target signal, then the smart glasses need to perform short-time Fourier transform STFT processing on the first target signal and the second target signal respectively, so as to obtain a first time-frequency spectrum corresponding to the first target signal and a first time-frequency spectrum corresponding to the second target signal.

[0154] It is understandable that in a scenario where signal screening is not required, the smart glasses can also directly obtain a time-frequency spectrum by performing short-time Fourier transform (STFT) processing on the high-frequency signal. That is, the above S704 can also include: the smart glasses perform short-time Fourier transform (STFT) processing on the high-frequency signal to obtain a time-frequency spectrum corresponding to the high-frequency signal. Among them, the high-frequency signal includes a first high-frequency signal and a second high-frequency signal. The smart glasses perform short-time Fourier transform (STFT) processing on the first high-frequency signal and the second high-frequency signal, respectively, to obtain a first time-frequency spectrum corresponding to the first high-frequency signal and a first time-frequency spectrum corresponding to the second high-frequency signal.

[0155] It should be noted that when the smart glasses use the short-time Fourier transform (STFT) to transform the target signal, the STFT is a transform process with segmentation and windowing, that is, in the STFT, the signal to be processed will be segmented into a series of short-time segments, and the window will slide in each segmented segment to select the signal segment for analysis. Therefore, a segmented segment can correspond to a time-frequency spectrum.

[0156] Therefore, the number of time-spectrograms obtained by performing short-time Fourier transform (STFT) on the signal depends on the number of segments into which the signal is segmented. Specifically, if the signal is segmented into N≥1 segments, it can be determined that the signal will obtain N≥1 time-spectrograms, thereby forming a time-spectrogram sequence.

[0157] Exemplarily, if the first target signal or the first high-frequency signal is divided into N≥1 segments, then the first target signal or the first high-frequency signal corresponds to N≥1 first time-spectrograms, thereby forming a first time-spectrogram sequence. Similarly, if the second target signal or the second high-frequency signal is divided into N≥1 segments, then the second target signal or the second high-frequency signal corresponds to N≥1 second time-spectrograms, thereby forming a second time-spectrogram sequence.

[0158] For example, taking ultrasonic signals as an example, Figure 8 A schematic diagram of the principle of short-time Fourier transform is shown.

[0159] refer to Figure 8, the smart glasses can divide the ultrasonic signal into N≥3 segments according to time, namely segment 1, segment 2, segment 3... In other words, Figure 8 The ultrasonic signal in includes at least 3 segments, and correspondingly, the obtained time-spectrogram sequence includes 3 time-spectrograms.

[0160] Exemplary, reference Figure 8 The first segment in the image, i.e., reference segment 1, in the process of short-time Fourier transform (STFT) processing of the ultrasonic signal, a fixed-size window will slide in segment 1 according to the window step size, so that multiple signal segments can be selected in segment 1 for Fourier transform respectively, so as to obtain the time-frequency spectrum corresponding to segment 1 Figure 1 The window step size refers to the step size of the window sliding / moving within each segment, which can also be called the moving step size.

[0161] like Figure 8 As shown, the time spectrum Figure 1 Each column of data in corresponds to a signal segment in a window. Therefore, the number of columns in the time-spectrum diagram is related to the size of the window, the length of the segment, and the moving step of the window.

[0162] Exemplary, reference Figure 8 , assuming that the window size is 256 (i.e., the window length is 256), the segment length of segment 1 is 4096, and the window step size is 1, then the time spectrum Figure 1 There are 4096-256+1 columns in it, that is, M=4096-256+1. And the length of each column is equal to the size of the window, for example, the length of each column is equal to 256. In other words, Figure 8 Time spectrum in Figure 1 Includes 256 rows*4096-256+1 columns.

[0163] In the same way, we can get the time-frequency spectrum corresponding to segment 2 Figure 2 , the time spectrum corresponding to segment 3 Figure 3 ……, the principle and processing process are similar to Figure 8 The principle and processing process of the fragment 1 shown are the same and will not be described in detail.

[0164] It should be noted that the embodiments of this application Figure 8 The three moves of the window shown in fragment 1, Figure 8 The purpose of showing the window in an up-and-down manner is to illustrate the movement of the window through the diagram, so as to show the window step length, which does not limit the window in essence. At the same time, it can be understood that the window length and the window step length can be set based on actual needs, and the embodiment of the present application does not make any limitation on this.

[0165] In other embodiments, for the same ultrasonic signal, the smart glasses may perform short-time Fourier transform using one or more different window lengths to obtain time-frequency spectrograms with different resolutions.

[0166] for example, Figure 8 The ultrasonic signal in can be transformed into a short-time Fourier transform using a window length A (such as 256), so as to obtain N time-frequency spectrograms A corresponding to the window length A. At the same time, Figure 8 The ultrasonic signal in can also be short-time Fourier transformed using a window length B (such as 512) to obtain N time-frequency spectrograms B corresponding to the window length B.

[0167] S705, the smart glasses eliminate static information in the time-frequency spectrum diagram and merge features to obtain a merged time-frequency spectrum diagram.

[0168] Because blinking is a dynamic action, the reflected signal offset caused by blinking is also dynamically expressed in the signal. Correspondingly, the static information in the signal is not very helpful for blinking detection and can be regarded as useless data. Furthermore, in order to reduce the interference of useless data, the smart glasses can eliminate the static information and retain only the dynamic information before merging the time-frequency spectrum. Therefore, after obtaining the time-frequency spectrum corresponding to the reflected signal, the smart glasses can first eliminate the static information in the time-frequency spectrum to ensure that only pure dynamic information is retained in the time-frequency spectrum.

[0169] In some embodiments, eliminating static information may be achieved by performing a difference calculation on two adjacent columns of data in the time-frequency spectrum diagram. Fig. 9 FIG. 1 shows a schematic diagram of the principle of static information elimination. It can be understood that Fig. 9 The static information elimination shown in is the static information in the spectrum diagram when it is eliminated.

[0170] refer to Fig. 9 , taking the time-frequency spectrum 1 as an example, for each column of data in the time-frequency spectrum 1, such as Fig. 9 The 1st, 2nd, 3rd, ... M-1, M columns of data shown in the figure are respectively subjected to difference calculation with the adjacent previous column of data, so as to eliminate the static information in the data of this column. That is, the 2nd column of data after the static information is eliminated is the difference between the original 2nd column of data and the 1st column of data (i.e. the difference between 2 and 1), the 3rd column of data after the static information is eliminated is the difference between the original 3rd column of data and the 2nd column of data (i.e. the difference between 3 and 2), ..., the Mth column of data after the static information is eliminated is the difference between the original Mth column of data and the M-1th column of data (i.e. the difference between M and M-1). Among them, because the 1st column of data in the time-frequency spectrum 1 does not have an adjacent previous column of data, the 1st column of data after the static information is eliminated can be a column with all 0 data.

[0171] After the static information is eliminated, the smart glasses can perform feature merging of the time-frequency spectra corresponding to all signal receivers, such as the ultrasonic receiver, to obtain a merged time-frequency spectra. In the embodiment of the present application, feature merging mainly refers to splicing. That is, the embodiment of the present application obtains a merged time-frequency spectra by splicing all the time-frequency spectra.

[0172] It is understandable that because the time-frequency spectrum is a spectrum with time characteristics, and the timing of the corresponding signals of different ultrasonic receivers may not be uniform or there may be slight differences, so if the time-frequency spectrum is combined by adding, it may be difficult to align the timing. At the same time, the offset of the reflected signal is positive or negative, and it is easy to eliminate the offset by adding to combine the time-frequency spectrum, so the embodiment of the present application is not suitable for the combination method of feature addition.

[0173] In a specific embodiment, the feature merging may be to first splice the time-frequency spectrum corresponding to each ultrasonic receiver to obtain a spliced ​​time-frequency spectrum corresponding to each ultrasonic receiver, and then splice the spliced ​​time-frequency spectrum corresponding to each ultrasonic receiver to obtain a merged time-frequency spectrum.

[0174] For example, taking a first ultrasonic receiver, a second timeout receiver, and N=2 (the instant spectrogram sequence includes 2 time-spectrograms), Fig.10 A schematic diagram showing the principle of splicing time-frequency spectra.

[0175] refer to Fig.10 The smart glasses can first splice the two time-frequency spectra of the ultrasonic signal 1 corresponding to the first ultrasonic receiver to obtain a spliced ​​time-frequency spectra. Figure 1 And, the two time-frequency spectra of the ultrasonic signal 2 corresponding to the second ultrasonic receiver are spliced ​​to obtain a spliced ​​time-frequency spectra Figure 2 Then, the smart glasses calculate the spliced ​​time-frequency spectrum corresponding to the first ultrasonic receiver. Figure 1 The spliced ​​time-frequency spectrum corresponding to the second ultrasonic receiver Figure 2 Splice them together to get a combined time-frequency spectrum.

[0176] That is to say, in the embodiment of the present application, the principle that needs to be followed in feature merging is that the time-frequency spectrum diagrams corresponding to the same ultrasonic receiver need to be continuously spliced ​​together, while the time-frequency spectrum diagrams corresponding to different ultrasonic receivers cannot be cross-spliced.

[0177] In some embodiments, when multiple short-time Fourier transforms are performed on the same ultrasonic signal using multiple different window lengths, it is necessary to splice the time-frequency spectrograms of the ultrasonic signal with multiple different window lengths when merging features. For example, taking the above-mentioned window length A and window length B as an example, the smart glasses need to splice the N time-frequency spectrograms A corresponding to the window length A to obtain a spliced ​​graph A. And, splice the N time-frequency spectrograms B corresponding to the window length B to obtain a spliced ​​graph B. Then, splice the spliced ​​graph A and the spliced ​​graph B to obtain a spliced ​​time-frequency spectrogram.

[0178] Exemplarily, if the ultrasonic signal corresponding to the first ultrasonic receiver is subjected to two short-time Fourier transforms with window length A and window length B, respectively, then the spliced ​​time-frequency spectrum obtained by splicing spliced ​​image A and spliced ​​image B is the above-mentioned spliced ​​time-frequency spectrum 1. Similarly, if the ultrasonic signal corresponding to the second ultrasonic receiver is subjected to two short-time Fourier transforms with window length A and window length B, respectively, then the spliced ​​time-frequency spectrum obtained by splicing spliced ​​image A and spliced ​​image B is the above-mentioned spliced ​​time-frequency spectrum 2. Of course, the above-mentioned spliced ​​time-frequency spectrum 1 and spliced ​​time-frequency spectrum 2 can also be spliced ​​time-frequency spectrums that have undergone two short-time Fourier transforms at the same time.

[0179] S706, the smart glasses detect the blinking action based on the combined time spectrum graph and determine the target blinking action.

[0180] After obtaining the combined time spectrum graph, the smart glasses perform corresponding feature analysis on the combined time spectrum graph, thereby detecting and determining the user's blinking action based on the combined time spectrum graph. In some embodiments, the smart glasses can detect the user's blinking action through a trained classification model. That is, the smart glasses output the obtained combined time spectrum graph to the trained classification model, and the classification model outputs the classification result. Because the classification model in the embodiment of the present application is a network model for predicting the user's blinking action, in the embodiment of the present application, the classification result output by the classification model is the user's blinking action.

[0181] In some embodiments, the classification model may be a binary classification model. That is, when the prediction result of blink detection is a classification problem with only two categories, the classification model may be a binary classification model. For a binary classification model, the prediction result outputted by the model may be represented by 0 and 1, or by "yes" and "no", "positive" and "negative", etc.

[0182] In a specific embodiment, the blinking action detected by the binary classification model may include any of the following: no blinking, single blink of both eyes, L≥2 consecutive blinks of both eyes (e.g., L=2 rapid blinks of both eyes, L=3 rapid blinks of both eyes), single blink of the left eye, L≥2 consecutive blinks of the left eye (e.g., L=2 rapid blinks of the left eye, L=3 rapid blinks of the left eye), single blink of the right eye, L≥2 consecutive blinks of the right eye (e.g., L=2 rapid blinks of the right eye, L=3 rapid blinks of the right eye). Wherein, L is a positive integer.

[0183] For example, taking a single blink of both eyes as an example, after the smart glasses input the merged time spectrum graph into the binary classification model, the binary classification model detects whether the user's blinking action is a single blink of both eyes (i.e., the target blinking action) based on the merged time spectrum graph. In the case where 1 represents yes and 0 represents no, if the binary classification model determines that the user's blinking action is a single blink of both eyes, the binary classification model can output 1, and if it determines that the user's blinking action is not a single blink of both eyes, the binary classification model outputs 0. The smart glasses can determine whether the user has a single blink of both eyes based on the 1 or 0 output by the binary classification model.

[0184] For example, taking a single blink of the left eye as an example, after the smart glasses input the merged time spectrum graph into the binary classification model, the binary classification model will detect whether the user's blinking action is a single blink of the left eye (i.e., the target blinking action) based on the merged time spectrum graph. Similarly, 1 represents a single blink of the left eye, and 0 represents a single blink of the left eye. If the binary classification model determines that the user's blinking action is a single blink of the left eye, the binary classification model outputs 1, and if it determines that the user's blinking action is not a single blink of the left eye, the binary classification model outputs 0. Then the smart glasses determine whether the user has a single blink of the left eye based on the 1 or 0 output by the binary classification model.

[0185] In some other embodiments, the classification model may be a multi-classification model. That is, in the case where the prediction result of the blink detection is a classification problem with multiple (greater than two) categories, the classification model may be a multi-classification model. For the multi-classification model, different blinking actions may also be represented by characters such as 0, 1, 2, 3, etc. At the same time, when the multi-classification model outputs multiple blinking actions, these multiple blinking actions have confidence levels one by one. The accuracy of different blinking actions is distinguished by the confidence levels.

[0186] In a specific embodiment, the blinking action detected by the multi-classification model may include any one or more of no blinking, a single blink of both eyes, L≥2 consecutive blinks of both eyes, a single blink of the left eye, L≥2 consecutive blinks of the left eye, a single blink of the right eye, and L≥2 consecutive blinks of the right eye.

[0187] Exemplarily, after the smart glasses input the combined time-frequency spectrogram into the multi-classification model, the multi-classification model can detect whether the user has any one or more of the above blinking actions based on the combined time-frequency spectrogram detection, and output one or more classification results accordingly. For example, the classification results output by the multi-classification model may include a single blink of both eyes, a single blink of the left eye, and L=2 consecutive blinks of the left eye. At the same time, a single blink of both eyes, a single blink of the left eye, and L=2 consecutive blinks of the left eye correspond one-to-one to confidence 1, confidence 2, and confidence 3, respectively. Furthermore, after the smart glasses obtain the three classification results output by the multi-classification model, they can determine the target blink action from these multiple blink actions according to the confidence.

[0188] In a specific embodiment, if confidence 1>confidence 2>confidence 3, then the blinking action corresponding to confidence 1 is the target blinking action. Alternatively, if confidence 2>confidence 3>confidence 1, then the blinking action corresponding to confidence 2 is the target blinking action. Alternatively, if confidence 3>confidence 2>confidence 1, then the blinking action corresponding to confidence 3 is the target blinking action.

[0189] For example, Fig.11 A schematic diagram of a data collection scenario is shown. Fig.11 The training process of the classification model is described below.

[0190] First, refer to Fig.11 , the user wears smart glasses and then faces a device with a camera, such as a mobile phone or a computer with a camera to collect video data with rich and different facial movements.

[0191] Among them, the collected video data needs to include the user's conscious blinking action. For example, the video data includes the user's conscious single blink of both eyes, continuous blinking of both eyes L≥2 times, single blink of the left eye, continuous blinking of the left eye L≥2 times, single blink of the right eye, continuous blinking of the right eye L≥2 times, etc. In addition, while collecting video data, it is also necessary to collect corresponding signal data through the smart glasses worn. The signal data is the reflected signal received by the signal receiver in the smart glasses (such as the timeout signal received by the ultrasonic receiver). In other words, it is necessary to collect video data and reflected signals including blinking actions at the same time.

[0192] Then, the collected video data and the signal data collected at the same time are aligned with the timestamps. After the timestamp alignment is completed to ensure the time is unified, the signal segments related to the blinking action are intercepted from the signal data based on the time of the video data and the preset time length as the interception unit, so that the reflection signal corresponding to each different blinking action can be obtained, and the signal segment related to the blinking action is used as a positive sample. At the same time, the signal segment that does not include the blinking action is intercepted with the preset time length as the interception unit. This type of signal segment that does not include the blinking action is a negative sample. That is, based on the target time period that includes the blinking action, the signal segment corresponding to the time can be intercepted from the signal data as a positive sample. Similarly, based on the non-target time period that does not include the blinking action, the signal segment corresponding to the time can be intercepted from the signal data as a negative sample.

[0193] Among them, the preset duration can be determined according to the actual duration of the blinking action required. For example, after actual measurement, the duration of a single blink is generally between 0.2s and 0.4s, and the time taken for the user to consciously blink twice in a row is about 0.5s to 1.5s. Therefore, in the scenario where only a single blink and two consecutive blinks are detected, the preset duration can be 2s. That is, with 2s as the interception unit, the signal segment related to the blinking action is intercepted from the reflected signal. Similarly, with 2s as the interception unit, the signal segment that does not include the blinking action is intercepted from the reflected signal.

[0194] For example, in a scenario where the preset duration is 2s, assuming that there is a blink at 30s in the video data, the target time period can be 28s-30s, 29s-31s, and 30s-32s. In some embodiments, in order to ensure the quality of the positive sample, the blink can be in the middle area of ​​the intercepted signal segment. That is, taking the above 30s as an example, the positive sample is the reflection signal in the time period of 29s-31s.

[0195] Finally, the positive samples are labeled with the corresponding blink action type according to the actual blink action of the sample, and the negative samples can be labeled with no blink action, that is, all the training data are labeled with the corresponding blink action type. The training data labeled with the blink action type, including positive samples and negative samples, are input into the classification model for model training, so as to obtain a classification model with the ability to detect blink actions.

[0196] In some embodiments, the classification model can be any existing model network structure, and the embodiments of the present application do not impose any limitation on this. For example, the classification model can be an ordinary convolutional network or a large model.

[0197] S707, the smart glasses execute a target control operation in response to the target blinking action.

[0198] Because different blinking actions are predefined as different operation instructions, and different operation instructions are predefined to correspond to different control operations. Therefore, after the smart glasses determine the target blinking action through blinking action detection, they can determine the control operation corresponding to the target blinking action, that is, determine the target control operation. Then, the smart glasses execute the target control operation. In other words, the smart glasses can execute different control operations in response to the user's blinking action.

[0199] It is understandable that different operation instructions corresponding to different blinking actions can be factory settings of the device, or can be customized by the user with custom permissions provided.

[0200] In a specific embodiment, the control operations corresponding to different blinking actions may include any one or more of the following: dry eye prompt, answering a call, hanging up a call, stopping the alarm, stopping music, taking a photo, starting the camera, starting the semantic assistant, and playing music. It can be understood that these nine control operations are only an example in the embodiment of the present application. Based on actual interaction needs, the control operations may include more or less, and the embodiment of the present application does not impose any limitation on this.

[0201] In some embodiments, the target control operation can act on the smart glasses themselves or on the electronic device connected to the smart glasses. At the same time, based on actual needs, the control operations corresponding to different blinking actions can be the same or different. For example, multiple different blinking actions can correspond to instructions to perform different control operations. Exemplarily, blinking twice in a row with both eyes can be used to indicate that a call is answered. Blinking twice in a row with the left eye can be used to indicate starting the camera, and blinking twice in a row with the right eye can be used to indicate playing music. For another example, multiple different blinking actions can also indicate the execution of the same control operation. Exemplarily, blinking twice in a row with both eyes can be used to indicate that the call is closed or the call is ended. Blinking twice in a row with the left eye and blinking twice in a row with the right eye can both be used to indicate starting the camera. For another example, the same blinking action can also indicate the execution of different control operations. Exemplarily, blinking twice in a row with both eyes can be used to indicate that a call is answered and to indicate that a call is ended. That is, in the application scenario where the smart glasses are connected to a mobile phone, if the mobile phone receives an incoming call, the user can send instructions to the mobile phone through the smart glasses to trigger the mobile phone to answer the call by continuously blinking. After answering a call, if the phone is in a call state, the user can also trigger the phone to end the call by continuously blinking their eyes.

[0202] In this way, device interaction with smart glasses can be achieved through various blinking actions, so that the interaction with smart glasses is no longer limited to manual or voice, thereby meeting the user's interaction needs in various scenarios and improving the user experience.

[0203] Hereinafter, the embodiment of the present application takes four signal receivers as an example, namely, two first signal receivers and two second signal receivers as an example, to illustrate the device interaction method provided in the embodiment of the present application.

[0204] For example, Fig.12 A schematic diagram of a process for determining a target blinking action is shown.

[0205] refer to Fig.12 , ultrasonic receiver 1, ultrasonic receiver 2, ultrasonic receiver 3 and ultrasonic receiver 4 correspond to the two first signal receivers and the two second signal receivers one by one. That is to say, after the ultrasonic transmitter in the smart glasses sends a direct signal, the ultrasonic receiver 1, ultrasonic receiver 2, ultrasonic receiver 3 and ultrasonic receiver 4 in the smart glasses will receive the audio signal 1, audio signal 2, audio signal 3 and audio signal 4 respectively. It can be understood that the audio signal 1, audio signal 2, audio signal 3 and audio signal 4 are the ultrasonic signals received by the ultrasonic receiver, that is, the reflected signal in the above embodiment.

[0206] After the audio signal is acquired, due to the influence of the surrounding environment, some low-frequency environmental audio may exist in the audio signal 1, audio signal 2, audio signal 3 and audio signal 4 received by the ultrasonic receiver 1, ultrasonic receiver 2, ultrasonic receiver 3 and ultrasonic receiver 4. Therefore, in order to reduce the interference of low-frequency signals such as environmental audio, the smart glasses first perform high-pass filtering on the audio signal 1, audio signal 2, audio signal 3 and audio signal 4 respectively, so as to filter out the low-frequency signals in the audio signal 1, audio signal 2, audio signal 3 and audio signal 4, thereby retaining the high-frequency signals in the audio signals, and thus obtaining the corresponding high-frequency signals. Fig.12 As shown, audio signal 1, audio signal 2, audio signal 3 and audio signal 4 are processed by high-pass filtering to obtain high-frequency signal 1, high-frequency signal 2, high-frequency signal 3 and high-frequency signal 4 respectively. It can be understood that Fig.12 The specific process of the high-pass filtering shown in can refer to the record of S703 above, the principle is the same, and the embodiment of the present application will not be repeated here.

[0207] Then, after obtaining the high-frequency signal, the smart glasses can perform short-time Fourier transform processing on the four high-frequency signals respectively, thereby obtaining N time-frequency spectrograms for each high-frequency signal. Fig.12 As shown, high-frequency signal 1, high-frequency signal 2, high-frequency signal 3 and high-frequency signal 4 are subjected to short-time Fourier transform processing respectively, and N time-frequency spectrograms corresponding to each high-frequency signal are obtained accordingly, that is, a total of 4*N time-frequency spectrograms. It can be understood that Fig.12The specific process of the short-time Fourier transform processing shown in can refer to the record of S704 above, the principle is the same, and the embodiment of the present application will not be repeated here.

[0208] In some embodiments, before performing short-time Fourier transform on the high-frequency signal, the high-frequency signal may be first subjected to signal screening to obtain a target signal of a desired frequency band. In a specific embodiment, signal screening may be performed based on a predefined target frequency band. Exemplarily, based on the predefined target frequency band, four high-frequency signals, namely, high-frequency signal 1, high-frequency signal 2, high-frequency signal 3, and high-frequency signal 4, are respectively subjected to signal screening, so that target signal 1, target signal 2, target signal 3, and target signal 4 may be obtained respectively. Then, short-time Fourier transform processing is performed on the four target signals, namely, target signal 1, target signal 2, target signal 3, and target signal 4, to obtain Fig.12 4*N time-frequency spectrograms are shown.

[0209] Next, the smart glasses perform static information elimination processing on the 4*N time-frequency spectrograms respectively, thereby eliminating static information (i.e. eliminating direct signal information with unchanged paths), thereby retaining pure dynamic information. At the same time, for the time-frequency spectrogram after the static information is eliminated, the smart glasses perform feature merging to obtain a merged time-frequency spectrogram. In the embodiment of the present application, the feature merging of the time-frequency spectrograms is specifically the splicing of the time-frequency spectrograms. It can be understood that Fig.12 The specific process of static information elimination and feature merging shown in can refer to the record of S705 above, the principle is the same, and the embodiment of the present application will not be repeated here.

[0210] Finally, the smart glasses input the spliced ​​combined time-frequency spectrum into the trained classification model, and the classification model outputs the classification result based on the combined time-frequency spectrum. Since the classification model in the embodiment of the present application is a trained model mainly used for blinking action detection, the classification result output by the classification model is the detected blinking action.

[0211] Specifically, when the classification model outputs only one blink action, that is, when the classification model is a binary classification model, the blink action output by the classification model is the target blink action in the embodiment of the present application. When the classification model can output multiple (two or more) types of blink actions, that is, when the classification model is a multi-classification model, the blink action type with the highest confidence is the target blink action in the embodiment of the present application.

[0212] After the smart glasses determine that the target has blinked, the smart glasses can respond to the target blink and perform the target control operation. Fig.12The specific process of determining the target blinking action by the classification model shown in and the specific process of executing the target control operation can refer to the above-mentioned S706 and S707. The principles are the same and will not be repeated in this embodiment of the present application.

[0213] For example, taking the blinking action in the above embodiment as an example, Fig.13 A schematic diagram of a flow chart for executing a target control operation is shown.

[0214] refer to Fig.13 , the four audio signals received by the four signal receivers, namely, audio signal 1, audio signal 2, audio signal 3 and audio signal 4 and other reflected signals, are subjected to a series of processing such as high-pass filtering, Fourier transform, static information elimination, feature merging and classification model, and the obtained classification results may include: no blinking, single blink of both eyes, two consecutive blinks of both eyes, single blink of the left eye, two consecutive blinks of the left eye, single blink of the right eye, and two consecutive blinks of the right eye. In some embodiments, after high-pass filtering, the audio signal can also be screened. In a specific embodiment, Fig.13 The Fourier transform shown in can be a short-time Fourier transform or a non-short-time Fourier transform with added segmentation and window processing.

[0215] refer to Fig.13 , the control operation corresponding to no blinking is no response. Single blinking of both eyes corresponds to dry eye prompt. That is to say, through a series of processing of the audio signal received by the ultrasonic receiver, if it is determined that the user has no blinking action, indicating that the user has not issued any operation instructions, then no response is required, so the control operation corresponds to no response. In the case where it is determined that the user has blinked a single time with both eyes, this kind of blinking is a natural eye movement, so the corresponding control operation is not predefined. However, if Fig.13 As shown, the smart glasses can further count the blinking frequency of the user's eyes in a single blink, and determine whether the user has dry eyes based on the counted blinking frequency, thereby reminding the user.

[0216] For example, Fig.13 As shown, when the blinking frequency is less than 10 times per minute, that is, <10 times / min (i.e., the preset frequency), it is determined that the user has dry eyes, and the control operation of dry eye prompt is performed. In some embodiments, the dry eye prompt can be through voice prompts or through displayed text prompts. In this way, the dry eye prompt can be used to remind the user to actively blink to moisten the eyeballs, avoid the deterioration of dry eyes, and ensure the user experience.

[0217] As for the control operations corresponding to blinking actions such as blinking twice in a row of both eyes, blinking the left eye once, blinking twice in a row of the left eye, blinking the right eye once, and blinking twice in a row of the right eye, different operation instructions can be customized according to needs.

[0218] like Fig.13 As shown, the customized operation instructions may include customized control instructions and customized shortcut functions. Among them, the control operations corresponding to the customized control instructions may include answering / hanging up calls, stopping alarms / music, taking photos, etc. The control operations corresponding to customized shortcut functions may include starting the camera, turning on the voice assistant, and playing music, etc.

[0219] For example, blinking twice in a row can correspond to a custom control instruction, specifically corresponding to the control operation of answering / hanging up a call. Then, when the detected target blinking action is blinking twice in a row, the corresponding target control operation is answering / hanging up a call. Alternatively, blinking twice in a row can correspond to a custom shortcut function, specifically corresponding to the control operation of starting the camera. Then, when the detected target blinking action is blinking twice in a row, the corresponding target control operation is starting the camera.

[0220] Understandably, Fig.13 The classification results shown, and the control operations corresponding to different classification results are only examples in the embodiments of the present application. Fig.13 There is no limitation on the classification results and the corresponding target control operations. Based on actual needs, the classification results may include more or less. For example, the classification results may also include three consecutive blinks of both eyes, three consecutive blinks of the left eye, and three consecutive blinks of the right eye. Similarly, the target control operations may also be more or less based on actual needs. For example, the target control operations may also include playing a video, closing a video, taking a screenshot, turning on recording, turning off recording, etc.

[0221] For example, taking the electronic device as a mobile phone, the target blinking action as a single blink of the left eye, and the target control operation as playing music as an example, Fig.14 and Fig.15 An application scenario diagram of a device interaction method is shown respectively.

[0222] refer to Fig.14 In the scenario where the user is wearing smart glasses and the smart glasses are connected to a mobile phone, the ultrasonic transmitter of the smart glasses transmits ultrasonic signals to the user's eye area. The ultrasonic signal is reflected by the user's eye area, and the reflected audio signal is received by the ultrasonic receiver. The audio signal undergoes a series of processing such as high-pass filtering, Fourier transform, static information elimination, and feature merging (see Appendix). Fig.14 (not shown) is then input into the classification model, so that the classification model detects the target blinking action. Fig.14As shown, when the detected target blinking action is a single blink of the left eye, the smart glasses can determine that the corresponding operation instruction is a music play instruction. Then, the smart glasses send the music play instruction to the connected mobile phone, and the mobile phone responds to the music play instruction and plays the music. In this way, by interacting with the smart glasses through the blinking action, the interaction with the electronic device can be realized, so that there is no need to be limited to the traditional interaction methods of manual or voice, which improves the user experience.

[0223] refer to Fig.15 Similarly, when the user is wearing smart glasses, the ultrasonic transmitter of the smart glasses transmits ultrasonic signals to the user's eye area. The ultrasonic signal is reflected by the user's eye area, and the reflected audio signal is received by the ultrasonic receiver. The audio signal undergoes a series of processing such as high-pass filtering, Fourier transform, static information elimination, and feature merging (see Appendix). Fig.15 (not shown) is input into the classification model, so that the target blinking action is detected by the classification model. Fig.15 As shown, when the target blinking action is a single blink of both eyes, the smart glasses can further count the blinking frequency. Then, when it is determined that the blinking frequency is less than the preset frequency, such as less than 10 times / min, the smart glasses perform dry eye prompts. For example, the smart glasses can perform dry eye prompts through voice. Fig.15 , the smart glasses play a prompt voice "Dry eyes reminder, please blink". In this way, the user's dry eyes can be prevented from getting worse and the user experience can be ensured.

[0224] Another embodiment of the present application provides a pair of smart glasses, comprising: at least one signal transmitter, at least one first signal receiver, at least one second signal receiver, one or more processors and memories, wherein the signal transmitter, the first signal receiver, the second signal receiver and the memory are coupled to the processor; the signal transmitter is used to transmit signals, and the first signal receiver and the second signal receiver are used to receive reflected signals; one or more computer program codes are stored in the memory, and the computer program codes include computer instructions; when the processor executes the computer instructions, the smart glasses implement the device interaction method described in any of the above embodiments.

[0225] Another embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor in the smart glasses, the smart glasses implement the device interaction method described in any of the above embodiments.

[0226] The present application also provides a computer program product, which, when executed on a computer, enables the computer to perform the functions or steps in the above method embodiments. The computer may be the smart glasses described in any of the above embodiments.

[0227] The present application also provides a chip system, such as Fig.16 As shown, the chip system 1600 includes at least one processor 1601 and at least one interface circuit 1602. The processor 1601 and the interface circuit 1602 can be interconnected through a line. For example, the interface circuit 1602 can be used to receive signals from other devices (such as a computer memory). For another example, the interface circuit 1602 can be used to send signals to other devices (such as the processor 1601).

[0228] Exemplarily, the interface circuit 1602 can read the instruction stored in the memory and send the instruction to the processor 1601. When the instruction is executed by the processor 1601, the computer can execute the various steps in the above embodiments. In some embodiments, the computer including the chip system can be the smart glasses described in any of the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiments of the present application.

[0229] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0230] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0231] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0232] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0233] If the function of the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0234] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A device interaction method, characterized in that: Applied to smart glasses, the smart glasses comprising at least one signal transmitter, at least one first signal receiver and at least one second signal receiver; The first signal receiver corresponds to the left eye area, and the second signal receiver corresponds to the right eye area; The method comprises: Controlling the at least one signal transmitter to transmit a signal, wherein the transmitted signal reaches the user's eye area to form a reflection signal; receiving at least one first reflected signal through the at least one first signal receiver, and receiving at least one second reflected signal through the at least one second signal receiver; In response to the detected target blinking action, executing a target control operation corresponding to the target blinking action; wherein the target blinking action is determined based on the at least one first reflection signal and the at least one second reflection signal; After filtering the low-frequency signals in the at least one first reflected signal and the at least one second reflected signal, Fourier transform processing is performed on the at least one first reflected signal and the at least one second reflected signal respectively to obtain at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence; wherein the first time-frequency spectrum sequence and the second time-frequency spectrum sequence each include N time-frequency spectrums, where N≥1, N is a positive integer, and N corresponds to the number of segments into which the signal is divided by the Fourier transform processing; performing static information elimination processing on the N time-spectrograms in the at least one first time-spectrogram sequence and the at least one second time-spectrogram sequence, respectively, and retaining dynamic information corresponding to the blinking action in the N time-spectrograms; After the static information elimination process, merging the at least one first time-spectrogram sequence and the at least one second time-spectrogram sequence to obtain a merged time-spectrogram; Detecting the user's blinking action based on the combined time spectrum graph, obtaining the target blinking action; Each of the N time-spectrograms includes M columns of data, M≥1, M is a positive integer, and M is determined based on the size of the window used for the Fourier transform processing, the length of the segment, and the moving step of the window; the window is used to move and select different signal segments, and each time the window moves, the signal segment included in the window undergoes a Fourier transform processing to form a corresponding column of data; The static information elimination is performed on the N time-frequency spectrum graphs in the at least one first time-frequency spectrum graph sequence and the at least one second time-frequency spectrum graph sequence, respectively, including: for each column of data of each time-frequency spectrum graph in the N time-frequency spectrum graphs, difference calculation is performed with the adjacent previous column of data, and the obtained difference time-frequency spectrum graph is the result of the static information elimination processing.

2. The method according to claim 1, characterized in that The smart glasses further include a frame, a left lens, and a right lens; wherein the frame is used to fix the left lens and the right lens; The at least one signal transmitter is arranged on the frame close to the left lens or the right lens; the at least one first signal receiver is arranged on the frame close to the left lens; and the at least one second signal receiver is arranged on the frame close to the right lens.

3. The method according to claim 2, characterized in that The smart glasses include four signal transmitters, two first signal receivers and two second signal receivers; Among them, two of the four signal transmitters are arranged on the frame close to the left lens and form a diagonal line, and the other two signal transmitters are arranged on the frame close to the right lens and form a diagonal line; the two first signal receivers are arranged on the frame close to the left lens and form a diagonal line, and the two second signal receivers are arranged on the frame close to the right lens and form a diagonal line.

4. The method according to claim 1, characterized in that: The signal transmitter is an ultrasonic transmitter, the first signal receiver and the second signal receiver are both ultrasonic receivers; and the first reflected signal and the second reflected signal are both ultrasonic signals.

5. The method according to claim 1, characterized in that The smart glasses are connected to an electronic device; and in response to the detected target blinking action, a target control operation corresponding to the target blinking action is performed, including: An operation instruction corresponding to the target control operation is sent to the electronic device, and the electronic device performs the target control operation in response to the operation instruction.

6. The method according to claim 1, characterized in that After filtering the low-frequency signal in the at least one first reflected signal and the at least one second reflected signal, respectively performing Fourier transform processing on the at least one first reflected signal and the at least one second reflected signal to obtain at least one first time-frequency spectrum sequence and at least one second time-frequency spectrum sequence, including: Respectively filtering low-frequency signals in the at least one first reflected signal and the at least one second reflected signal to obtain at least one first high-frequency signal and at least one second high-frequency signal; wherein the low-frequency signal is a signal less than a preset frequency, and the high-frequency signal is a signal greater than or equal to the preset frequency; Fourier transform processing is performed on the at least one first high-frequency signal and the at least one second high-frequency signal respectively to obtain the at least one first time-frequency spectrum sequence and the at least one second time-frequency spectrum sequence.

7. The method according to claim 6, characterized in that The filtering the low-frequency signals in the at least one first reflected signal and the at least one second reflected signal respectively to obtain at least one first high-frequency signal and at least one second high-frequency signal comprises: The at least one first reflected signal and the at least one second reflected signal are respectively subjected to high-pass filtering by using a high-pass filter to obtain the at least one first high-frequency signal and the at least one second high-frequency signal; wherein the preset frequency is the cutoff frequency of the high-pass filter.

8. The method according to claim 6, characterized in that The performing Fourier transform processing on the at least one first high-frequency signal and the at least one second high-frequency signal respectively to obtain the at least one first time-frequency spectrum sequence and the at least one second time-frequency spectrum sequence comprises: Perform signal screening on the at least one first high-frequency signal and the at least one second high-frequency signal based on the target frequency band to obtain the at least one first target signal and the at least one second target signal; Fourier transform processing is performed on the at least one first target signal and the at least one second target signal respectively to obtain the at least one first time-frequency spectrum sequence and the at least one second time-frequency spectrum sequence.

9. The method according to claim 1, characterized in that: The step of combining the at least one first time-spectrogram sequence and the at least one second time-spectrogram sequence to obtain a combined time-spectrogram comprises: splicing the N time-frequency spectrograms in each first time-frequency spectrogram sequence respectively to obtain at least one first spliced ​​time-frequency spectrogram corresponding to the at least one first time-frequency spectrogram sequence; splicing the N time-frequency spectrograms in each second time-frequency spectrogram sequence respectively to obtain at least one second spliced ​​time-frequency spectrogram corresponding one-to-one to the at least one second time-frequency spectrogram sequence; The at least one first spliced ​​time-frequency spectrum graph and the at least one second spliced ​​time-frequency spectrum graph are spliced ​​to obtain the merged time-frequency spectrum graph.

10. The method according to claim 9, characterized in that In the case where the first reflection signal is subjected to multiple Fourier transform processing based on windows of different sizes, windows of different sizes correspond to first time-frequency spectrum sequences of different resolutions, and the first time-frequency spectrum sequences of different resolutions respectively correspond to a spliced ​​time-frequency spectrum, and the first spliced ​​time-frequency spectrum is obtained by splicing the spliced ​​time-frequency spectrum corresponding to each resolution; wherein the window is used to move and select different signal segments in the first reflection signal, and the Fourier transform is processed based on the signal segments; When the second reflected signal is subjected to multiple Fourier transform processing based on windows of different sizes, there are second time-frequency spectrum sequences with different resolutions corresponding to windows of different sizes, and the second time-frequency spectrum sequences with different resolutions respectively correspond to a spliced ​​time-frequency spectrum, and the second spliced ​​time-frequency spectrum is obtained by splicing the spliced ​​time-frequency spectrum corresponding to each resolution.

11. The method according to claim 1, characterized in that: The detecting the user's blinking action based on the combined time spectrum graph to obtain the target blinking action includes: The combined time spectrum graph is input into a trained classification model, and the target blinking action is determined by the classification model; wherein, when the classification model is a multi-classification model and outputs multiple blinking action types, the blinking action type with the highest confidence is selected as the target blinking action.

12. The method according to claim 11, characterized in that The method further comprises: Acquire video data and signal data collected simultaneously; the video data includes different blinking movements, and the signal data is a reflected signal received by a signal receiver; After aligning the timestamps of the video data and the signal data, a corresponding signal segment is intercepted from the signal data as a positive sample based on a target time period, and a corresponding signal segment is intercepted from the signal data as a negative sample based on a non-target time period; wherein the target time period is a time period including a blinking action, and the non-target time period is a time period not including a blinking action; The positive samples and the negative samples are used to perform model training to obtain the classification model.

13. The method according to any one of claims 1 to 5, characterized in that The target blinking action includes a single blink and L consecutive blinks; the single blink includes a single blink of both eyes, a single blink of the left eye and a single blink of the right eye; the L consecutive blinks include L consecutive blinks of both eyes, L consecutive blinks of the left eye and L consecutive blinks of the right eye, L≥2, L is a positive integer; wherein different target blinking actions correspond to different or the same target control operations; the same target blinking action corresponds to one target control operation or to multiple different target control operations.

14. The method according to claim 13, characterized in that The target control operation corresponding to the single blink of both eyes includes a dry eye prompt; the target control operation corresponding to the execution of the target blink action includes: Counting the blink frequency of the two eyes in a single blink; When the blinking frequency is less than a preset frequency, the dry eye prompt is executed to prompt the user to blink.

15. The method according to any one of claims 1 to 5, characterized in that Different eye movements correspond to different orbicularis oculi muscle states, which form different reflection paths. Different reflection paths correspond to different reflection signals with different frequencies. The target blinking movement is determined based on changes in reflection signals of different frequencies.

16. A pair of smart glasses, characterized in that: include: At least one signal transmitter, at least one first signal receiver, at least one second signal receiver, one or more processors and memories, the signal transmitter, the first signal receiver, the second signal receiver and the memory are coupled to the processor; the signal transmitter is used to transmit signals, and the first signal receiver and the second signal receiver are used to receive reflected signals; one or more computer program codes are stored in the memory, and the computer program codes include computer instructions; when the processor executes the computer instructions, the smart glasses execute the device interaction method described in any one of claims 1-15.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor of the smart glasses, the smart glasses execute the device interaction method according to any one of claims 1 to 15.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor in the smart glasses, the smart glasses execute the device interaction method as described in any one of claims 1-15.

Images