Smart device for recognizing motion of user by using default speaker and microphone, and operating method thereof

Abstract

The present invention relates to a smart device for recognizing a hand motion of a user by recording, by a microphone, a reflected wave caused by the hand motion of the user when an ultrasonic wave is emitted from a speaker and analyzing the reflected wave, wherein the smart device according to an embodiment may comprise: a speaker unit for emitting an ultrasonic signal; a microphone unit for receiving a reflected wave returning after the emitted ultrasonic signal is reflected by a body part of the user; and a motion recognition unit for recognizing a motion of the body part of the user by analyzing the received reflected wave on the basis of the Doppler effect.

Description

A smart device and method for recognizing user actions using built-in speakers and microphones

[0001] The present invention relates to a smart device that emits ultrasonic waves from a speaker, records reflected waves from a user's hand movements through a microphone, analyzes the recorded waves, and recognizes the user's hand movements.

[0002] The Internet of Things (IoT) refers to technology that enables internet-connected devices to interact with each other. By connecting various devices around us to the internet, allowing them to exchange data and be controlled, IoT can improve the convenience and efficiency of our lives.

[0003] IoT devices typically consist of sensors, software, and communication technology. These devices can take many forms and are applied across a wide range of fields, including home appliances (smart TVs, smart refrigerators), consumer goods (smart light bulbs, smart locks), and healthcare devices (wearables, health monitoring devices).

[0004] User authentication for IoT devices can take various forms and methods. The following types of user authentication are commonly used:

[0005] The most common form of authentication is password-based, where users authenticate themselves by entering a pre-set password. However, using weak or vulnerable passwords can pose security vulnerabilities.

[0006] Biometric authentication utilizes a user's biological characteristics, including fingerprints, iris scans, and facial recognition. Biometric authentication eliminates the need for passwords and offers high security.

[0007] Security token-based authentication is a method of authenticating users using physical devices such as security tokens or smart cards. It is commonly used for two-factor authentication and is highly secure.

[0008] Multi-factor authentication is a method of authenticating users by combining two or more authentication factors. A typical example is using a password and biometrics together, and is effective in increasing security.

[0009] Remote authentication is a method of authenticating users remotely over the Internet, typically utilizing cloud-based authentication services.

[0010] Existing user authentication suffers from poor usability because it requires additional devices such as smartphones or remote controls.

[0011] Additionally, Voice Fingerprint technology is a method of authenticating users by learning human voice characteristics through deep learning, but it is vulnerable to attacks where an attacker records or plays the user's voice, or synthesizes the user's voice to disable authentication.

[0012] Most existing authentication methods required separate hardware equipment. Despite this, security vulnerabilities persisted.

[0013] The purpose of the present invention is to provide a technology for recognizing a user's actions by utilizing speakers and microphones that are basically installed in a smart device without adding separate hardware.

[0014] The purpose of the present invention is to easily and accurately recognize a user's movements by emitting ultrasonic waves from a speaker, collecting reflected waves caused by a user's hand movements by a microphone, and analyzing them based on the Doppler effect.

[0015] The purpose of the present invention is to implement interface diversity by recognizing simple user actions using a smart device.

[0016] A smart device according to one embodiment may include a speaker unit that emits an ultrasonic signal, a microphone unit that collects a reflected wave that is returned when the emitted ultrasonic signal is reflected by a part of the user's body, and a motion recognition unit that analyzes the collected reflected wave based on the Doppler effect to recognize a motion of the part of the user's body.

[0017] According to one embodiment, the operation of a smart device may include a step of emitting an ultrasonic signal through a speaker, a step of collecting a reflected wave of the emitted ultrasonic signal reflected by a part of a user's body through a microphone, and a step of analyzing the collected reflected wave based on the Doppler effect through a processor to recognize a motion of the part of the user's body.

[0018] According to one embodiment, a technology for recognizing a user's actions can be provided by utilizing speakers and microphones that are built into smart devices without the need for additional hardware.

[0019] In one embodiment, when an ultrasonic wave is emitted from a speaker, a microphone collects the reflected waves caused by the user's hand movements and analyzes them based on the Doppler effect, allowing the user's movements to be easily and accurately recognized.

[0020] According to one embodiment, interface diversity can be implemented by recognizing simple user actions using a smart device.

[0021] Figure 1 is a drawing illustrating a smart device according to an embodiment.

[0022] Figure 2 is a drawing illustrating a motion recognition unit according to an embodiment.

[0023] Figure 3 is a drawing illustrating an embodiment of recognizing a user's motion using a smart device.

[0024] Figures 4a to 4d are diagrams explaining the spectrum according to signal preprocessing.

[0025] Figures 5a and 5b are diagrams showing that many outliers occur in the altitude of the azimuth even in the same signal frame.

[0026] Figure 6 is a drawing explaining an operation method of a smart device according to an embodiment.

[0027] Figure 7 is a drawing specifically enforcing the process of recognizing the motion of a part of a user's body by analyzing the reflected waves collected based on the Doppler effect.

[0028] Figure 8 is a drawing specifically enforcing the process of extracting a specific motion by dynamically identifying a frame in which the Doppler effect exists.

[0029] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0030] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes modifications, equivalents, or alternatives that fall within the spirit and technical scope of the present invention.

[0031] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component," without departing from the scope of the invention.

[0032] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between," "immediately between," or "directly adjacent to," should be interpreted similarly.

[0033] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0034] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0035]

[0036] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same components.

[0037] FIG. 1 is a drawing illustrating a smart device (100) according to one embodiment.

[0038] The smart device (100) according to the present invention can provide technology for recognizing a user's movements by utilizing the speaker and microphone that are basically built into the smart device without additional hardware. In addition, when the speaker emits ultrasonic waves, the microphone collects the reflected waves caused by the user's hand movements and analyzes them based on the Doppler effect, thereby enabling simple and accurate recognition of the user's movements. Furthermore, by recognizing simple user movements using the smart device (100) according to the present invention, a variety of interfaces can be implemented.

[0039] To this end, a smart device (100) according to one embodiment may include a speaker unit (110), a microphone unit (120), a motion recognition unit (130), and a control unit (140).

[0040] According to one embodiment, the speaker unit (110) emits an ultrasonic signal using at least one speaker, and the microphone unit (120) collects a reflected wave that is returned by the emitted ultrasonic signal being reflected by a part of the user's body using at least one microphone.

[0041] For example, the speaker unit (110) is a speaker included in a generally widely used smart device and can emit an ultrasonic signal close to an audible frequency.

[0042] The speaker unit (110) can generate sound waves above a certain frequency using at least one speaker during a frequency conversion process. These ultrasonic waves are beyond the human auditory range, typically between 20 Hz and 20 kHz, and therefore cannot be heard. However, these ultrasonic waves can be utilized for various purposes.

[0043] Ultrasound emitted from home speakers can primarily accompany music or speech signals. These ultrasounds can be used to provide sound effects across a wider frequency range or to emphasize specific sound effects. Ultrasound can also be used with acoustic devices to transmit specific sound signals, detect the surrounding environment, or measure user behavior.

[0044] In this way, even ordinary household speakers can emit ultrasonic signals without a separate ultrasonic generator, and can be used for various purposes.

[0045] The microphone unit (120) collects the reflected waves that are returned when an ultrasonic signal is emitted using at least one microphone and is reflected by a part of the user's body.

[0046] By detecting ultrasound waves reflected from the user's body and re-collecting them with a microphone, the device can be used to track body movements or determine location. This information can be utilized in a variety of applications, such as recognizing user movements or enabling interaction with the user.

[0047] For example, these technologies can be used in interactive games, virtual reality (VR) systems, security systems, health monitoring devices, and more. They can also help improve user experiences or create more efficient situations by understanding a user's movements or location.

[0048] According to one embodiment, the motion recognition unit (130) analyzes the collected reflected waves based on the Doppler effect to recognize motions of a part of the user's body.

[0049] The Doppler effect refers to the frequency shift caused by a moving object. By analyzing the frequency shift of reflected waves, this effect can be used to detect and recognize user actions.

[0050] This gesture recognition analyzes ultrasonic signals reflected from parts of the user's body and detects the Doppler effect that occurs during movement. This allows the user's movements to be identified and interactions or controls performed accordingly.

[0051] For example, when a user performs a motion, such as waving their hand or moving their body, the Doppler effect of the reflected wave generated by the motion can be analyzed to detect such motion. This can be used to manipulate or interact with devices or systems.

[0052] A motion recognition unit (130) according to an embodiment can recognize at least one of a position or direction of a body part of the user from a time difference (TDoA, Time DiOerence of Arrival) in which the reflected wave reaches the microphone using a filter in which weights are reflected, and recognize a motion of a body part of the user based on at least one of the recognized positions or directions.

[0053] For example, the motion recognition unit (130) according to one embodiment can recognize at least one of the position or direction of a part of the user's body from the time difference of arrival (TDoA, Time Difference of Arrival) of the reflected wave reaching the microphone by reflecting the weight within the PHAT filter. In addition, the motion of a part of the user's body can be recognized based on at least one of the recognized positions or directions.

[0054] Here, "PHAT" stands for "Phase Transform." The PHAT filter is used to correct the wavelength and direction of reflected waves in the frequency domain. This filter adjusts the weighting factors considered when calculating the time difference between the reflected waves reaching the microphone, allowing for accurate location and direction determination.

[0055] Accordingly, the motion recognition unit (130) according to one embodiment analyzes the TDoA information of the reflected wave using a PHAT filter and determines the user's location or direction based on this. Furthermore, by utilizing this location or direction information, the user's motion can be recognized and interaction or control can be performed through this.

[0056] For example, TDoA of reflected waves collected from a microphone can be analyzed to determine the position or orientation of a body part, such as a user's hand or leg. This position or orientation information can then be used to recognize the user's actions and perform interaction or control based on this information.

[0057] In this way, the motion recognition unit that recognizes the user's motion can interact with the user more accurately and effectively by using location and direction information.

[0058] The control unit (140) according to one embodiment can be interpreted as a central processing unit (CPU) and can perform various operations and process data within the system.

[0059] In particular, the control unit (140) can read commands from memory, interpret and execute the commands, and can also perform arithmetic operations such as addition, subtraction, multiplication, and division.

[0060] In addition, the control unit (140) can handle data storage and retrieval, and can also perform the function of reading data from memory and storing the results of performing operations back into memory.

[0061] In addition, the control unit (140) can manage the execution flow of the program, and in particular, can control the flow of the program using commands such as conditional statements (if-else) or iterative statements (for, while). In addition, the control unit (140) can have a small and fast memory device called a register placed inside, and this register can be used to temporarily store data or perform operations.

[0062] The control unit (140) can process and take appropriate action when an external event or exceptional situation occurs, and can quickly access data and instructions by using cache memory that is faster than the main memory.

[0063] In addition, the control unit (140) can use a system bus to communicate with memory or input / output devices, and can provide various power management functions to minimize power consumption.

[0064] Figure 2 is a drawing illustrating a motion recognition unit according to an embodiment.

[0065] A motion recognition unit (130) according to an embodiment may include a preprocessing unit (210), a motion extraction unit (220), and an authentication unit (230) to recognize a user's motion using the Doppler effect.

[0066] A preprocessing unit (210) according to one embodiment can preprocess the collected reflected waves.

[0067]

[0068] The preprocessing unit (210) can preprocess the collected reflected waves by removing non-Doppler frames and environmental noise components and then emphasizing the signal.

[0069] The preprocessing unit (210) is a necessary step before analyzing the collected reflected wave signal. It prevents signal interference caused by external factors such as non-Doppler frames or environmental noise. To this end, the preprocessing unit (210) detects and removes non-Doppler frames and environmental noise to extract an accurate signal.

[0070] Additionally, the preprocessing unit (210) can enhance the signal to facilitate analysis. This is necessary to extract useful information from the reflected wave and accurately recognize the user's movements.

[0071] Non-Doppler frames and environmental noise are the primary environmental factors that affect signals, and can contribute to the inaccuracy of the reflected wave signal. For example, ambient noise or signal interference from other sources can distort the original signal, hindering accurate detection of the user's movements.

[0072] The preprocessing unit (210) detects and removes these external factors to accurately extract the original signal. Removing non-Doppler frames allows for better focus during motion recognition, and removing environmental noise enables accurate analysis of the user's motion.

[0073] For example, when a user waves their hand or moves their body, these movements can be reflected in a specific form in the reflected wave. However, interference from surrounding noise or other reflectors can distort the original signal and cause confusion in motion recognition.

[0074] Therefore, the preprocessing unit (210) detects and removes these external factors to extract accurate signals, thereby ensuring reliable motion recognition in subsequent steps.

[0075] The motion extraction unit (220) according to one embodiment can extract a specific motion corresponding to a preprocessed reflected wave.

[0076] The motion extraction unit (220) according to one embodiment can select the frequency size of the reference signal with the largest size within the frame of the preprocessed reflected wave, and generalize the size of other frequency signals according to the frequency size of the selected reference signal.

[0077] The motion extraction unit (220) can select the frequency size of the reference signal with the largest size within the frame of the preprocessed reflected wave, and generalize the size of other frequency signals according to the frequency size of the selected reference signal.

[0078] This is a process of extracting a motion signal from a preprocessed reflected wave and analyzing the frequency size of the motion signal. The motion extraction unit (220) can first select the signal with the largest size in each frame, i.e., the reference signal. This reference signal can generally represent the signal with the strongest movement.

[0079] Thereafter, the motion extraction unit (220) can normalize the amplitude of other frequency signals based on the frequency amplitude of the selected reference signal. This standardizes the frequency amplitudes, enabling comparisons based on the same standard. In other words, all frequency signals can be measured based on the same standard, enabling effective comparisons.

[0080] A standard deviation value for the frequency size values ​​of the corresponding frame is calculated, and after calculating the standard deviation value for all frames, the frame with the highest deviation is selected, and a specific motion can be extracted based on the selected frame.

[0081] The motion extraction unit (220) calculates the standard deviation for the frequency size values ​​in each frame, and through this, an indicator indicating how much the frequency values ​​of the frame are spread out can be confirmed.

[0082] Thereafter, the motion extraction unit (220) calculates the standard deviation value for all frames and selects the frame with the highest deviation among them. A frame with a high standard deviation means that the frequency values ​​have a high variability, indicating that movement is actively occurring in the frame.

[0083] The motion extraction unit (220) can extract specific motions based on the frame with the highest deviation. This allows the user's motions to be identified by analyzing the frequency patterns appearing in the corresponding frames, and motion recognition can be performed based on these.

[0084] According to an embodiment, a motion extraction unit (220) can extract the specific motion by dynamically identifying a frame in which a Doppler effect exists based on selected frames.

[0085] According to an embodiment, a motion extraction unit (220) can extract the specific motion by dynamically identifying a frame in which a Doppler effect exists based on selected frames.

[0086] The Doppler effect refers to a phenomenon in which waves are compressed or relaxed according to the movement of an object, and the motion extraction unit (220) utilizes this effect to identify frames in which the Doppler effect exists among the selected frames.

[0087] Typically, the Doppler effect is caused by the movement of an object, and the user's movements are also reflected by this movement. Therefore, the motion extraction unit (220) detects the Doppler effect in a selected frame and extracts a specific motion appearing in that frame.

[0088] Through this, the motion extraction unit (220) can dynamically track and analyze the user's motions to accurately identify specific motions.

[0089] An authentication unit (230) according to an embodiment can determine whether a specific extracted action matches a previously registered action.

[0090] The motion extraction unit (220) extracts input motions for a plurality of virtual buttons identified by a plurality of virtually divided regions for the preprocessed reflected wave, and the authentication unit (230) can determine whether to authenticate by considering whether a combination of the extracted input motions is a motion or information registered in advance.

[0091] For this purpose, frequency information, wavelength information, frequency or wavelength change information corresponding to a specific input action can be recorded in a database.

[0092] Figure 3 is a drawing illustrating an embodiment of recognizing a user's motion using a smart device.

[0093] In FIG. 3, a smart device according to an embodiment can divide a virtual area into a first quadrant (320), a second quadrant (310), a third quadrant (330), and a fourth quadrant (340) centered on the smart device.

[0094] Additionally, it can recognize user actions occurring in a virtual area.

[0095] Smart devices can detect when a user performs a virtual button-pushing action in the first quadrant.

[0096] Specifically, the smart device can generate ultrasonic signals for user input.

[0097] When a user who recognizes the operation of a smart device performs an action of pushing a virtual button in the first quadrant (320), a change may occur in the generated ultrasonic signal.

[0098] These changes are based on the Doppler effect, and the smart device's microphone can detect them by collecting the echoes of the emitted ultrasound signals that bounce back after being reflected by parts of the user's body.

[0099] The motion recognition unit of a smart device can recognize the motion of a part of the user's body by analyzing the reflected waves collected based on the Doppler effect.

[0100] If the motion for a body part is a motion to push a virtual button, the pushing motion of the button located in the first quadrant (320) can be recognized.

[0101] Assuming there are four buttons in the first quadrant (320) to the fourth quadrant (340), various passwords can be entered through the four buttons.

[0102] Figures 4a to 4d are diagrams explaining the spectrum according to signal preprocessing.

[0103] After identifying the frame, more noise needs to be removed to ensure accurate gesture recognition.

[0104] In the present invention, more noise can be removed through signal preprocessing.

[0105] Specifically, the present invention can apply a spectral subtraction method, which is a general method for removing noise in the frequency domain.

[0106] As shown in Fig. 4a, the smart device can take a non-Doppler spectrum (410) as an environmental noise spectrum.

[0107] Thereafter, as shown in Fig. 4c, the noise spectrum (420) can be subtracted from the Doppler presence spectrum (410) (430).

[0108] In the present invention, if the subtraction result of Fig. 4c is negative, the value of the original spectrum is set to be multiplied by 0.005.

[0109] For reference, the noise-subtracted Doppler signal (420) is as shown in Fig. 4b.

[0110] In FIG. 4d, the present invention can display the Doppler signal more clearly to the STATION by emphasizing the Doppler spectrum across the entire frame (440).

[0111] To determine the type of gesture (i.e. which virtual button was pressed), we need to know the direction the gesture is directed towards the smart device and whether the gesture is a PTP gesture.

[0112] To do this, two pieces of information need to be extracted from the highlighted Doppler spectrum: the direction of reflection (DOA) and the movement of the hand.

[0113] A smart device according to the present invention collects the direction in which a user's hand is pointing toward the smart device in the form of an altitude and azimuth tuple, which can use an acoustic positioning algorithm.

[0114] The smart device can compute the DoA from the sum of cross-correlation scores corresponding to the TDoA of fixed discrete points with respect to the microphone location.

[0115] Discrete points embedded in a virtual 3D sphere around a smart device can be expressed as direction vectors representing radians of the x, y, and z axis components from the sensing device, as in [Mathematical Formula 1].

[0116]

[0117] [Mathematical Formula 1]

[0118]

[0119]

[0120] In [Mathematical Formula 1], the TDoA from a discrete point x to microphones m1 and m2 is expressed as fs, where c is the sound propagation speed, and xm is the position of the mth microphone.

[0121] After calculating the TDoA of each discrete point, the smart device can calculate the cross-correlation score of the TDoA corresponding to the Doppler signal.

[0122] In the present invention, a weighting factor can be introduced into the PHAT filter to control the influence of spectral magnitude information. In the implementation of STATION, the experimental results are Setting the value to 0.5 helps to obtain more accurate cross-correlation scores from the Doppler signal.

[0123] After obtaining the cross-correlation scores, the smart device can run SRP-PHAT to calculate the DoA from the sum of the scores for each discrete point using [Mathematical Formula 2] below.

[0124]

[0125] [Equation 2]

[0126]

[0127]

[0128] As shown in [Mathematical Formula 2], the algorithm can calculate the direction point corresponding to the maximum value of the sum of cross-correlations.

[0129] In [Mathematical Formula 2], P (x) represents the result of the sum of the cross-correlations of the microphone pairs for each discrete point X, g represents all discrete points in the sphere, and Xs represents the point in the sphere corresponding to the maximum value.

[0130] Figures 5a and 5b are diagrams showing that many outliers occur in the altitude of the azimuth even in the same signal frame.

[0131] The results in Figures 5a and 5b represent the user's hand position in the x, y and z axes in a virtual 3D space around the smart device.

[0132] Converting these results into a specific gesture type is relatively difficult. Therefore, in the present invention, the above results are further converted into altitude and orientation tuples.

[0133] The problem of determining hand orientation is that, due to noise and variations in human gestures, there are many outliers in the elevation cross-sections for the same signal frame. Therefore, a smart device according to one embodiment can collect all tuples of elevation and orientation data for each frame and use the median, which is more resistant to outliers, to represent the actual elevation and orientation.

[0134] Figure 6 is a drawing explaining an operation method of a smart device according to an embodiment.

[0135] A method of operating a smart device according to one embodiment emits an ultrasonic signal through a speaker (step 601).

[0136] Ultrasonic signals can be used to gather information about the surrounding environment of a smart device or to detect a user's movements or location. These signals can be preprocessed to reduce noise and enhance the signal for accurate recognition.

[0137] Through the microphone, the reflected waves that are returned by the emitted ultrasonic signal being reflected by a part of the user's body are collected (step 602).

[0138] Reflected waves contain sound signals that are reflected by the user's body or objects in the surrounding environment. Microphones can collect these reflected waves and convert them into digital signals.

[0139] During this process, the microphone measures and collects the reflected ultrasonic signal's amplitude and time of reflection. Signals reflected from parts of the user's body can be modulated by various factors and influenced by environmental noise. Therefore, by collecting and analyzing these reflected waves, signal processing and filtering are performed to accurately recognize the user's movements and location. This allows smart devices to quickly detect user movements and provide the user with the services or functions they need.

[0140] In addition, through the processor, the collected reflected waves are analyzed based on the Doppler effect to recognize the motion of a part of the user's body (step 603).

[0141] In step 603, a processor is used to analyze the reflected waves collected above based on the Doppler effect to recognize the user's movements. The Doppler effect refers to the phenomenon in which the frequency of sound or light changes when reflected from a moving object. This allows for the detection and analysis of movements of parts of the user's body.

[0142] The processor analyzes the collected reflected waves using the Doppler effect to determine the user's body movements. During this process, the processor detects frequency shifts in the reflected waves due to the Doppler effect, and uses this to estimate the user's movement patterns. For example, the user's finger or arm movements can cause frequency shifts in the reflected waves.

[0143] Through this analysis, the processor can identify the user's actions and, based on this, control the smart device's functions or manipulate the user interface. Each time the user performs a specific action, the processor detects that action and can even provide a specific response to the user.

[0144] A more specific method of recognizing motions for parts of the user's body is described in more detail in Fig. 7.

[0145] A method of operating a smart device according to an embodiment may utilize a standard deviation value to extract a specific motion corresponding to a preprocessed reflected wave.

[0146] A specific method for extracting a specific motion corresponding to a preprocessed reflected wave by utilizing the standard deviation value is described in more detail through Fig. 8.

[0147] Figure 7 is a drawing specifically enforcing the process of recognizing the motion of a part of a user's body by analyzing the reflected waves collected based on the Doppler effect.

[0148] In the process of recognizing a motion for a part of a user's body, the method of operating a smart device removes non-Doppler frames and environmental noise components from the collected reflected waves (step 701), and then preprocesses the signals by emphasizing them (step 702).

[0149] In step 701, non-Doppler frames and environmental noise are identified and removed from the collected reflected waves. Non-Doppler frames represent signals without Doppler effect and may be data unrelated to movement of the user's body parts. Furthermore, environmental noise can degrade signal accuracy, so removing it can improve signal quality.

[0150] In step 702, the signal is preprocessed by removing the non-Doppler frames and environmental noise, and then enhancing the signal.

[0151] This process removes non-Doppler frames and environmental noise, then preprocesses the signal to enhance it. This process emphasizes important signal features, improving the reliability and efficiency of the signal used in recognition and analysis. Through this preprocessing process, smart devices can more accurately recognize and analyze the user's body movements.

[0152] A method of operating a smart device according to an embodiment extracts a specific motion corresponding to a preprocessed reflected wave, and extracts the specific motion by dynamically identifying a frame in which a Doppler effect exists based on selected frames (step 703).

[0153] In step 703, frames in which the Doppler effect exists are dynamically identified based on the selected frames.

[0154] At this stage, specific motions can be extracted based on the preprocessed reflected waves. To achieve this, frames with dynamic Doppler effects are identified from the preprocessed signals. Frames with Doppler effects are likely to contain signals indicating movement of parts of the user's body. By identifying these frames, smart devices can extract specific motions and connect them to appropriate services or functions based on the user's intent or needs.

[0155] Smart devices can more accurately understand the user's movements and provide appropriate responses by identifying these dynamic Doppler effects.

[0156] A method of operating a smart device according to an embodiment determines whether a specific extracted motion matches a motion registered in advance (step 704).

[0157] This step allows you to check whether the extracted specific motion matches a pre-registered motion. Pre-registered motions represent movement patterns that the user has preset or learned. If the extracted motion matches a pre-registered motion, the smart device can identify the intent of the motion and provide the user with the specified service or function.

[0158] Through this matching process, smart devices can identify a user's actions and provide customized services based on these actions. Each time a user performs a specific action, the smart device recognizes it, determines whether it matches a preset action, and provides an appropriate response.

[0159] Figure 8 is a drawing specifically enforcing the process of extracting a specific motion by dynamically identifying a frame in which the Doppler effect exists.

[0160] In particular, the method of operating a smart device according to one embodiment can select the frequency size of the reference signal with the largest size within the frame of the preprocessed reflected wave (step 801).

[0161] In step 801, each frame of the preprocessed reflected wave is analyzed to select the frequency of the largest reference signal within that frame. This reference signal can be used as a primary signal among the signals generated by motion. Therefore, various motions can be analyzed or distinguished based on this signal.

[0162] Additionally, the size of other frequency signals can be normalized to match the frequency size of the selected reference signal (step 802), and the standard deviation value for the frequency size values ​​of the corresponding frame can be calculated (step 803).

[0163] In step 802, the amplitudes of other frequency signals are adjusted to match the frequency amplitude of the selected reference signal, thereby normalizing the signal. This can be done to reduce amplitude differences between different frequency signals and maintain consistency in analysis. Normalizing the frequency amplitude allows for comparison and analysis of various signals within the same frequency range.

[0164] In step 803, the standard deviation of the frequency magnitude values ​​within the frame is calculated. The standard deviation represents the dispersion of the frequency magnitude values, allowing us to understand the variability of the frequency data within the frame. This helps us determine the degree of signal variation within the frame by measuring the distribution of the frequency data.

[0165] Through this process, the smart device can effectively analyze each frame of the preprocessed reflected wave and understand the characteristics of the frequency data to recognize and analyze the user's actions.

[0166] Next, after calculating the standard deviation value for all frames, the frame with the highest deviation is selected (step 804), and a specific motion can be extracted based on the selected frame (step 805).

[0167] In step 804, the standard deviation values ​​calculated for each frame are compared to select the frame with the highest deviation. A frame with a high deviation has high variability in frequency data, which means that there is a high probability that a significant action occurred in that frame.

[0168] Therefore, these frames can best represent the signal of the specific motion to be extracted.

[0169] Step 805 extracts specific actions occurring in the selected frames. The selected frames are those with the highest deviation, indicating high frequency data variability and a high probability of significant actions occurring. Therefore, specific actions can be extracted based on these frames, which can be used by smart devices to recognize and interpret user actions.

[0170] In one embodiment, a method of operating a smart device may recognize a motion of a user's body part by using a filter with weights reflected thereon, recognizing at least one of a position or direction of the user's body part from a time difference (TDoA, Time DiOerence of Arrival) at which the reflected wave reaches a microphone, and recognizing the motion of the user's body part based on at least one of the recognized positions or directions.

[0171] Ultimately, the present invention provides a technology for recognizing user actions by utilizing speakers and microphones inherently built into smart devices, without the need for additional hardware. Furthermore, when ultrasonic waves are emitted from the speaker, the microphone collects the reflected waves from the user's hand movements and analyzes them based on the Doppler effect, enabling simple and accurate recognition of the user's actions. Furthermore, by recognizing simple user actions using a smart device, a variety of interfaces can be implemented.

[0172]

[0173] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0174] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0175] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0176] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0177] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. For smart devices equipped with speakers and microphones, A speaker section that emits ultrasonic signals; A microphone unit that collects the reflected waves that are returned when the emitted ultrasonic signal is reflected by a part of the user's body; and A motion recognition unit that analyzes the collected reflected waves based on the Doppler effect to recognize motions of parts of the user's body. A smart device characterized by including:

2. In paragraph 1, The above motion recognition unit, A preprocessing unit that preprocesses the collected reflected waves; A motion extraction unit that extracts a specific motion corresponding to the above-mentioned preprocessed reflected wave; and An authentication unit that determines whether the specific action extracted above matches a previously registered action. Smart devices including:

3. In paragraph 2, The above preprocessing unit, A smart device that removes non-Doppler frames and environmental noise components from the collected reflected waves and then preprocesses the signals by emphasizing them.

4. In paragraph 2, A motion extraction unit that extracts a specific motion corresponding to the above-mentioned preprocessed reflected wave, Within the frame of the above preprocessed reflected wave, select the frequency size of the reference signal with the largest size, Generalize the size of other frequency signals according to the frequency size of the above-mentioned selected reference signal, Calculate the standard deviation value for the frequency magnitude values ​​of the corresponding frame, After calculating the standard deviation value for all frames, the frame with the highest deviation is selected. A smart device characterized in that it extracts a specific action based on the above-mentioned selected frame.

5. In paragraph 4, A motion extraction unit that extracts a specific motion corresponding to the above-mentioned preprocessed reflected wave, A smart device that extracts the specific motion by dynamically identifying frames in which a Doppler effect exists based on the above-mentioned selected frames.

6. In paragraph 1, The above motion recognition unit, A smart device characterized in that it recognizes at least one of a position or direction of a body part of the user from a time difference (TDoA, Time DiOerence of Arrival) in which the reflected wave reaches the microphone using a filter reflecting a weight, and recognizes an action of the body part of the user based on at least one of the recognized positions or directions.

7. In paragraph 2, The above motion extraction unit, Extract input actions for multiple virtual buttons identified by multiple virtually segmented regions for the above preprocessed reflected waves, The above authentication unit, A smart device characterized in that it determines whether authentication is performed by considering whether the combination of the extracted input actions is a previously registered action or information.

8. In the operation of smart devices equipped with speakers and microphones, A step of emitting an ultrasonic signal through a speaker; A step of collecting the reflected wave that is returned by the emitted ultrasonic signal being reflected by a part of the user's body through a microphone; and A step of recognizing a motion of a part of the user's body by analyzing the collected reflected waves based on the Doppler effect through a processor. A method of operating a smart device, characterized in that it includes:

9. In paragraph 8, The step of recognizing the motion of the user's body part is as follows: A step of preprocessing by removing non-Doppler frames and environmental noise components from the collected reflected waves and then emphasizing the signal; A step of extracting a specific motion corresponding to the preprocessed reflected wave, by dynamically identifying a frame in which a Doppler effect exists based on the selected frame; and A step for determining whether the above-mentioned extracted specific action matches a previously registered action. A method of operating a smart device, characterized in that it includes:

10. In paragraph 9, The step of extracting a specific motion corresponding to the above preprocessed reflected wave is: A step of selecting the frequency size of the reference signal with the largest size within the frame of the above preprocessed reflected wave; A step of generalizing the size of another frequency signal according to the frequency size of the selected reference signal; A step of calculating the standard deviation value for the frequency size values ​​of the corresponding frame; A step of calculating the standard deviation value for all frames and then selecting the frame with the highest deviation; and A step of extracting a specific action based on the above-mentioned selected frames. A method of operating a smart device, characterized in that it includes:

11. In paragraph 8, The step of recognizing the motion of the user's body part is as follows: A step of recognizing at least one of a position or direction of a body part of the user from a time difference (TDoA, Time DiOerence of Arrival) in which the reflected wave reaches the microphone using a filter with a weight, and recognizing an action of the body part of the user based on at least one of the recognized positions or directions. A method of operating a smart device, characterized in that it includes:

Images