Electronic device and gesture recognition method thereof
By integrating sensors and drive modules in electronic devices, adjusting distances in real time and selecting appropriate posture detection modes, the user posture recognition problem in narrow spaces is solved, and flexible posture input and low-power posture detection are achieved.
Patent Information
- Application Number
- CN201980065475.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-10-02
- Filing Date
- 2019-10-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-10-02
AI Technical Summary
Existing electronic devices are difficult to recognize user postures in narrow indoor spaces, especially at short distances, and cannot effectively adapt to different distance environments, resulting in limited posture input.
By integrating sensors and drive modules in the electronic device, the distance from the user is adjusted in real time and the appropriate posture detection mode is selected according to the distance, so as to achieve posture recognition and feedback.
It provides a more flexible posture input environment, reduces space limitations, and reduces power consumption by optimizing sensor usage, adapting to user posture detection at different distances.
Smart Images

Figure CN112805663B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an electronic device and a gesture recognition method thereof. Background Art
[0002] Typically, an electronic device includes an input device (such as a microphone, a mouse, a keyboard, a keypad, or a stylus) to receive an input command. In addition, the electronic device may recognize a user gesture as an input command and thereby manipulate a user interface.
[0003] In the latter case, the electronic device has a capability of recognizing a user's gesture through a motion sensor installed therein or by analyzing an image acquired through a camera installed therein.
[0004] When the electronic device is configured to recognize user gestures at a fixed or optimal distance from the user, it may be difficult for a user in a very narrow indoor space to input gestures to the electronic device. In addition, the electronic device may often fail to recognize user gestures made at a short distance. Summary of the invention
[0005] Technical issues
[0006] According to various embodiments of the present disclosure, there is provided an electronic device capable of performing a gesture recognition method while changing a distance from a user in real time.
[0007] In addition, the electronic device according to various exemplary embodiments of the present disclosure can perform a gesture recognition method by selecting a gesture detection mode suitable for a distance from a user.
[0008] Technical Solution
[0009] According to various embodiments of the present disclosure, there is provided an electronic device capable of performing a gesture recognition method while changing a distance from a user in real time.
[0010] In addition, the electronic device according to various exemplary embodiments of the present disclosure can perform a gesture recognition method by selecting a gesture detection mode suitable for a distance from a user.
[0011] According to various exemplary embodiments of the present disclosure, an electronic device may include: one or more sensors; a driving module including a driving circuit configured to move the electronic device; a memory and a processor. The processor may be configured to control the electronic device by executing instructions: using at least one of the one or more sensors to determine a first distance between the electronic device and a user; based on the first distance being within a first predetermined range, obtaining information corresponding to the user's posture through at least one of the one or more sensors; and based on the first distance being within a second predetermined range, using the driving module to move the electronic device so that the electronic device is within the first predetermined range, and based on the electronic device being within the first predetermined range, obtaining information corresponding to the user's posture through at least one of the one or more sensors.
[0012] According to various example embodiments of the present disclosure, a gesture recognition method for an electronic device may include: using at least one of one or more sensors to determine a first distance between the electronic device and a user; based on the first distance being within a first predetermined range, acquiring information corresponding to the user's gesture through at least one of the one or more sensors; and based on the first distance being within a second predetermined range, using a driving module to move the electronic device so that the electronic device is within the first predetermined range, and based on the electronic device being within the first predetermined range, acquiring information corresponding to the user's gesture through at least one of the one or more sensors.
[0013] According to various example embodiments of the present disclosure, an electronic device may include: one or more sensors; a driving module including a driving circuit configured to move the electronic device; a memory and a processor. The processor may be configured to control the electronic device by executing instructions: using a position detection sensor among the one or more sensors to determine the distance between the electronic device and a user, based on the distance satisfying a first distance range, using a motion detection sensor among the one or more sensors to detect the user's posture in a first predetermined detection mode; based on the distance satisfying a second distance range, using a motion detection sensor among the one or more sensors to detect the user's posture in a second predetermined detection mode; and performing a predetermined action linked to the detected posture.
[0014] According to the electronic device and the gesture recognition method thereof, by changing the distance between the electronic device and the user in real time, a gesture input environment with reduced space limitation can be provided to the user.
[0015] According to the electronic device and the gesture recognition method thereof, various feedback environments may be provided to the user by changing the distance between the electronic device and the user or changing a gesture detection mode in real time depending on the distance.
[0016] According to the electronic device and the gesture recognition method thereof, by changing the distance between the electronic device and the user or changing the gesture detection mode in real time depending on the distance, the user gesture can be detected through various sensors while reducing the power consumption of the electronic device.
[0017] Beneficial technical effects
[0018] According to the electronic device and the gesture recognition method thereof, by changing the distance between the electronic device and the user in real time, a gesture input environment with reduced space limitation can be provided to the user.
[0019] According to the electronic device and the gesture recognition method thereof, various feedback environments may be provided to the user by changing the distance between the electronic device and the user or changing a gesture detection mode in real time depending on the distance.
[0020] According to the electronic device and the gesture recognition method thereof, by changing the distance between the electronic device and the user or changing the gesture detection mode in real time depending on the distance, the user gesture can be detected through various sensors while reducing the power consumption of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following detailed description taken in conjunction with the accompanying drawings, in which:
[0022] Figure 1 is a block diagram illustrating an example electronic device in a network environment according to various embodiments of the present disclosure;
[0023] Figure 2 is a block diagram illustrating example software of an electronic device according to various embodiments of the present disclosure;
[0024] Figure 3 is a flowchart illustrating an example gesture recognition method of an electronic device according to various embodiments of the present disclosure;
[0025] Figure 4 is a diagram illustrating an example relocation operation of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0026] Figure 5 is a diagram illustrating an example repositioning and detection mode selection operation of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0027] Figure 6 is a flow chart illustrating an example repositioning and detection mode selection operation of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0028] Figure 7is a diagram illustrating an example change in a detection mode of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0029] Figure 8 is a diagram illustrating an example change in a detection mode of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0030] Fig. 9 is a diagram illustrating an example change in a detection mode of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0031] Fig.10 is a diagram illustrating an example repositioning and detection mode changing operation of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0032] Fig.11 is a diagram illustrating an example repositioning and detection mode changing operation of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0033] Fig.12 is a diagram illustrating an example of a gesture detection area according to various embodiments of the present disclosure.
[0034] Fig.13 is a diagram illustrating an example change in a detection mode of an electronic device for gesture recognition according to various embodiments of the present disclosure;
[0035] Fig.14 is a diagram illustrating an example gesture recognition method at a short distance between an electronic device and a user according to various embodiments of the present disclosure;
[0036] Fig.15 is a diagram illustrating an example gesture recognition method at a long distance between an electronic device and a user according to various embodiments of the present disclosure;
[0037] Fig.16 is a diagram illustrating example gesture detection areas depending on executed content according to various embodiments of the present disclosure; and
[0038] Fig.17 is a diagram illustrating example gesture detection areas depending on executed content according to various embodiments of the present disclosure. DETAILED DESCRIPTION
[0039] Hereinafter, various exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.
[0040] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.
[0041] Reference Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, a behavior module 163, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., display device 160 or camera module 180) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit. For example, sensor module 176 (e.g., fingerprint sensor, iris sensor, or illumination sensor) may be implemented to be embedded in display device 160 (e.g., display).
[0042] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may load a command or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a specified function. The auxiliary processor 123 may be implemented separately from the main processor 121 , or as part of the main processor 121 .
[0043] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 (not the main processor 121), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123.
[0044] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.
[0045] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .
[0046] The input device 150 may receive commands or data from outside the electronic device 101 (e.g., a user) to be used by other components of the electronic device 101 (e.g., the processor 120). The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus).
[0047] The sound output device 155 can output a sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as a part of the speaker.
[0048] The display device 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.
[0049] The behavior module 163 may perform facial expression changes, gesture expressions, or travel. According to an embodiment, the behavior module 163 may include a facial expression motor, a gesture expression motor, or a driver. The facial expression motor may visually provide, for example, the state of the electronic device 101 via the display device 160. The driver may, for example, be used to move the electronic device 101 or mechanically change other components. The driver may be in the form of being able to rotate up / down, left / right, or clockwise / counterclockwise around at least one axis. The driver may be implemented by controlling a drive motor (e.g., a wheel, a spherical wheel, a continuous track, or a propeller) in combination or independently.
[0050] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.
[0051] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a posture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.
[0052] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.
[0053] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0054] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or movement) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.
[0055] The camera module 180 may capture still images or moving images. Depending on the embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes. Depending on the embodiment, the camera module 180 may include a 2D (e.g., RGB) camera 182 or an infrared-based depth camera 184 (e.g., see FIG. Figure 2 ). The camera module 180 may include a lens assembly, a flash, an image sensor, an image stabilizer, a memory (eg, a buffer memory), or an image signal processor.
[0056] The lens assembly can collect light from an object that is a target for image capture. The lens assembly may include one or more lenses. According to an embodiment, the camera module 180 may include multiple lens assemblies. In this case, the camera module 180 may be, for example, a dual camera, a 360-degree camera, a spherical camera, a 2D camera 182, or an infrared-based depth camera 184. The multiple lens assemblies may have the same lens properties (e.g., viewing angle, focal length, autofocus, f-value, or optical zoom), or at least one lens assembly may have at least one lens property different from other lens assemblies. The lens assembly may include, for example, a wide-angle lens or a telephoto lens. A flash is a special light source for providing more light when taking photos. The flash may include at least one light emitting diode (e.g., a red, green, and blue (RGB) LED, a white light LED, an infrared LED, or an ultraviolet LED) or a xenon lamp.
[0057] The image sensor may convert light collected from an object through a lens assembly into an electrical signal, thereby acquiring an image corresponding to the object. According to an embodiment, the image sensor may include one image sensor selected from image sensors with different properties (such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor), a plurality of image sensors with the same properties, or a plurality of image sensors with different properties. Each image sensor included in the image sensor may be implemented, for example, by a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.
[0058] The image stabilizer may move at least one lens in the lens assembly or the image sensor in a specific direction or control (e.g., adjust the readout timing) so as to at least partially compensate for the negative effect of such movement on the captured image (e.g., image blur) when the camera module 180 or the electronic device 101 moves. According to an embodiment, the image stabilizer may be implemented as an optical image stabilizer, for example, and movement may be detected using a gyro sensor (not shown) or an acceleration sensor (not shown) arranged inside or outside the camera module 180.
[0059] The memory may at least temporarily store at least a portion of an image acquired by the image sensor for subsequent image processing operations. For example, if image acquisition is delayed due to a shutter, or if multiple images are acquired at high speed, the acquired original image (e.g., a high-resolution image) may be stored in the memory, and the corresponding copy image (e.g., a low-resolution image) may be previewed through the display device 160. Thereafter, when a specific condition (e.g., user input or system command) is met, at least a portion of the original image stored in the memory may be acquired and processed by the image signal processor. According to an embodiment, the memory may include a buffer memory as at least a portion thereof or operating independently.
[0060] The image signal processor is capable of performing image processing (e.g., depth map generation, three-dimensional modeling, panorama generation, feature point extraction, image synthesis, or image compensation (such as noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening)) on an image acquired by an image sensor or an image stored in a memory.
[0061] Additionally or alternatively, the image signal processor may perform control (e.g., exposure time control or extraction timing control) on at least one of the components (e.g., image sensor) included in the camera module 180. The image processed by the signal processor may be stored in the memory again for further processing, or may be delivered to an external component (e.g., memory 130, display device 160, electronic device 102 or 104, or server 108) outside the camera module. According to an embodiment, the image signal processor may be formed by at least a part of the processor 120 or by a separate processor that operates independently of the processor 120. In the latter case, the image processed by the image signal processor may be displayed on the display device 160 by the processor 120 without or after additional image processing.
[0062] According to an embodiment, the electronic device 101 may include two or more camera modules 180 having different properties or functions. In this case, for example, at least one camera module 180 may be a wide-angle camera or a front camera, and at least one other camera module may be a telephoto camera or a rear camera.
[0063] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).
[0064] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
[0065] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.
[0066] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiating element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a PCB). According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as a part of the antenna module 197.
[0067] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, general purpose input output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.
[0068] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 and the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute the requested at least part of the function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.
[0069] Figure 2 is a block diagram illustrating example software of an electronic device according to various embodiments of the present disclosure.
[0070] refer to Figure 2 , the software of the electronic device may include an operating system (OS) 142, middleware 144, internal memory 220, and / or smart framework 230. OS 142 controls one or more resources of the electronic device and may include, for example, Android TM 、iOS TM , Windows TM , Symbian TM 、Tizen TM or Bada TMFor example, at least some software programs may be preloaded into the electronic device at the time of manufacture, or downloaded (or updated) from an external electronic device (eg, electronic device 102 or 104 or server 108) when used by a user.
[0071] OS 142 can manage (e.g., allocate or restore) one or more system resources (e.g., processes, memory, or power) of the electronic device. In addition, OS 142 may include one or more device drivers 215 for driving other hardware devices of the electronic device 101, such as input device 150, sound output device 155, display device 160, behavior module 163, audio module 170, sensor module 176, interface 177, tactile module 179, camera module 180, power management module 188, battery 189, communication module 190, SIM 196, and / or antenna module 197.
[0072] The middleware 144 may perform authentication by detecting, tracking or recognizing the user's face through the signal processed data. In addition, the middleware 144 may recognize the user's 3D gesture, track the input position (or direction of arrival (DOA)) of the audio signal, recognize the voice and process the signals of various sensor data. The middleware 144 may include, for example, a gesture recognition manager 201, a face detection / tracking / recognition manager 203, a sensor information processing manager 205, a dialog engine manager 207, a speech synthesis manager 209, a sound source tracking manager 211, and / or a speech recognition manager 213.
[0073] The intelligent framework 230 may include various processing circuits and / or executable program elements, such as, but not limited to, a multi-mode fusion block 231, a user mode learning block 233, and / or a behavior controller 235. The multi-mode fusion block may collect and manage various types of information processed in the middleware 144. The user mode learning block 233 may use the information of the multi-mode fusion block 231 to extract and learn meaningful information, such as the user's life pattern and preferences. The behavior controller 235 may output feedback information of the electronic device to the user in the form of movement, graphics (UI / UX), lighting, voice, or audio.
[0074] The internal memory 220 may include, for example, a user model DB 221, a behavior model DB 223, or a voice model DB 225. The user model DB 221 may store information learned by the intelligent framework 230 for each user. The behavior model DB 223 may store information for behavior control of the electronic device. Such information stored in each DB may be stored in or shared with the wireless network DB 210 (e.g., cloud).
[0075] Figure 3 is a flowchart illustrating an example gesture recognition method of an electronic device according to various embodiments of the present disclosure.
[0076] According to various embodiments, in operation 301 , the electronic device 101 may determine a first distance between the electronic device 101 and a user using one or more sensors under the control of the processor 120 .
[0077] In various embodiments, one or more sensors may include, for example but not limited to, a proximity light sensor, a three-dimensional (3D) depth sensor, a posture sensor, an accelerometer, a gyroscope sensor, a geomagnetic sensor, an infrared (IR) sensor, an ultrasonic sensor, a microphone, a camera sensor, a radio frequency (RF) sensor, and / or a touch sensor (or display), etc.
[0078] The proximity light sensor may be a proximity sensor and / or an illumination sensor, and may be a sensor included in the sensor module 176. The 3D depth sensor may be a depth camera 184. The camera sensor may be one of a dual camera, a 360-degree camera, a spherical camera, a 2D camera 182, or an infrared-based depth camera 184. The 3D depth sensor and the camera sensor may include a camera module 180.
[0079] The electronic device 101 may use one or more sensors to measure and determine the distance between the electronic device 101 and the user.
[0080] For example, the 3D depth sensor may include an IR transmitter and a depth image CMOS. Using the time difference when the IR signal emitted from the IR transmitter is reflected by an object (eg, user) and returns, the distance between the electronic device 101 and the object (eg, user) may be measured.
[0081] In various embodiments, the camera sensor may acquire and process an image containing an object (eg, a user), determine the size of the object contained in the image, and thereby measure the distance between the electronic device 101 and the object (eg, the user).
[0082] In various embodiments, when a user wears a communication accessory (e.g., a wearable device such as Figure 1 When an external electronic device 102 or 104 is connected to the user, the RF sensor can measure the received signal strength indication (RSSI) of the signal received from the accessory to determine the distance between the electronic device 101 and the user. The RF sensor can be capable of communicating with Figure 1 The communication module 190 is used to communicate with the external electronic device 102 or 104 shown in FIG.
[0083] In various embodiments, a sound signal (eg, voice, clapping, stomping, etc.) generated by the user may be detected, and the transmission time of the sound signal may be calculated to measure the distance between the electronic device 101 and the user.
[0084] According to various embodiments, at operation 303 , under the control of the processor 120 , the electronic device 101 may detect a user gesture in a gesture area corresponding to the determined first distance using a first detection mode.
[0085] In various embodiments, the electronic device 101 may store a gesture recognition algorithm in the memory 130. In addition, based on the gesture recognition algorithm, the electronic device 101 may process a user gesture detected in the gesture area.
[0086] In various embodiments, based on a gesture recognition algorithm, the electronic device 101 can track the movement or shape of the joints connecting the user's head, arms, shoulders, hands, etc. (e.g., moving the hand to the left or right, drawing a circle with the hand, moving the hand back and forth, moving the hand closer or farther, etc.), thereby recognizing the user's gesture. The recognized gesture can be used as an input command.
[0087] In various embodiments, the priority of gesture recognition depending on distance can be defined differently according to the user's selection. In various embodiments, according to the gesture recognition algorithm, the electronic device 101 can be configured to first recognize gestures made by the movement of relatively large joints (such as arms or heads) based on distance (for example, when the distance between the electronic device 101 and the user is far). In various embodiments, according to the gesture recognition algorithm, the electronic device 101 can be configured to first recognize finger gestures with smaller movements based on distance (for example, when the distance is close). In various embodiments, according to the gesture recognition algorithm, the electronic device 101 can be configured to recognize various gestures from the movement or shape of the finger joints based on distance (for example, when the distance is close).
[0088] According to various embodiments, at operation 305 , under the control of the processor 120 , the electronic device 101 may determine whether it is necessary to move (eg, relocate) the electronic device 101 .
[0089] In various embodiments, operation 305 may depend on the distance between the electronic device 101 and the user, and on specific content that is primarily executed at the distance.
[0090] In various embodiments, operation 305 may depend on the type of application associated with electronic device 101 .
[0091] In various embodiments, when an application requiring facial recognition or an application requiring depth measurement is being executed in an external electronic device (eg, 102, 104, or 108) or the electronic device 101, the electronic device 101 may determine whether to move based on the type of application being executed.
[0092] For example, in the case where the user's face or upper body is recognized as a gesture at a first distance, it may be necessary to sense the movement of the entire area in order to determine the user's full body gesture. In this example, the electronic device 101 may determine that it needs to move (ie, reposition).
[0093] For example, when any game content requiring large movements is executed at close range, the electronic device 101 may determine whether to move so that the effective gesture area may include the whole body. In another example, when any content requiring relatively small movements (such as a browser or video player) is executed, the electronic device 101 may determine whether to move so that the effective gesture area may only include the hands or the head.
[0094] In various embodiments, operation 305 may be performed based on a user's request.
[0095] In various embodiments, in the event that a gesture cannot be recognized even after receiving the gesture, the electronic device 101 may determine that it is required to move to receive the gesture again.
[0096] When it is determined that movement is required, the electronic device 101 may then perform operation 307 under the control of the processor 120. When it is determined that movement is not required, the electronic device 101 may then perform operation 313 under the control of the processor 120.
[0097] According to various embodiments, at operation 307, under the control of the processor 120, the electronic device 101 may determine a second distance between the user and the electronic device 101 so that the gesture area corresponds to the detection range of the one or more motion detection sensors. The second distance is a distance required to identify the user's body area required for gesture recognition. The second distance may be longer or closer than the first distance.
[0098] According to various embodiments, at operation 309 , under the control of the processor 120 , the electronic device 101 may operate a driver by controlling the behavior module 163 to move to a position corresponding to the determined second distance.
[0099] According to various embodiments, at operation 311 , under the control of the processor 120 , the electronic device 101 may determine whether a user gesture may be detected in a gesture area corresponding to the determined second distance using the first detection mode.
[0100] According to various embodiments, when it is determined at operation 311 that the user gesture may be detected in the gesture area corresponding to the second distance, the electronic device 101 may detect the user gesture using the first detection mode and perform operation 313 .
[0101] According to various embodiments, when it is determined at operation 311 that the user gesture cannot be detected in the gesture area corresponding to the second distance, the electronic device 101 may perform operation 315 .
[0102] According to various embodiments, at operation 315 , under the control of the processor 120 , the electronic device 101 may change from the first detection mode to the second detection mode and detect a user gesture using the second detection mode.
[0103] For example, the gesture area corresponding to the first distance described in operation 303 and the gesture area corresponding to the second distance described in operation 311 may have different area values depending on the first distance and the second distance.
[0104] According to various embodiments, at operation 313, under the control of the processor 120, the electronic device 101 may perform a predetermined action linked to the gesture detected in the gesture area. For example, as described above, at operation 303, the user gesture may be detected in the gesture area corresponding to the first distance using the first detection mode, or after operation 311, the user gesture may be detected in the gesture area corresponding to the second distance using the first detection mode, or the user gesture may be detected in the gesture area corresponding to the second distance using the second detection mode in operation 315. The predetermined action linked to the detected gesture may be performed at operation 313.
[0105] According to various embodiments, the predetermined action linked to the detected gesture may be, for example, but not limited to, turning on or off lighting, inputting letters, controlling content being displayed (e.g., moving content, controlling playback of content), connecting to an external device, controlling an external device, etc.
[0106] According to various embodiments, the first detection mode and the second detection mode may have the following example differences, but the present disclosure is not limited thereto.
[0107] In various embodiments, the electronic device 101 may independently control each photodiode input of the pixel array of the camera module 180, and may also independently control (turn on or off) pixels according to a predetermined (based on factors such as distance) resolution necessary to recognize user gesture input.
[0108] For example, when a user makes a gesture, the image sensor of the camera module 180 can recognize the gesture in a first detection mode (e.g., a low resolution mode such as 8x8 pixels) or in a second detection mode (e.g., a medium resolution mode such as 64x64 pixels or a high resolution mode such as full pixels).
[0109] In various embodiments, the image sensor of the camera module 180 may include a color filter layer for filtering the light incident on the photodiode by wavelength. Typically, four filters of RGBG comprise a group corresponding to one pixel. Such a group of four filters may be arranged in a square or quadrilateral.
[0110] According to various embodiments, when operating in a first detection mode (e.g., a low-resolution mode), the electronic device 101 may recognize data of one pixel corresponding to the RGBG filter group as a single pixel itself. When operating in a second detection mode (e.g., a monochrome mode), the electronic device 101 may operate the color filter in a form of recognizing only one color (e.g., black and white) other than RGBG, thereby improving resolution.
[0111] According to various embodiments, when the group of four filters of RGBG is arranged in a square or quadrilateral, the electronic device 101 operating in the first detection mode (e.g., low-resolution mode) can recognize the interpolated pixel data corresponding to the RGBG filter group as a single pixel itself. The RGBG filter group can be interpolated using, for example, the Bayer interpolation technique. The electronic device 101 operating in the second detection mode (e.g., high-resolution mode) can operate all pixels to increase resolution.
[0112] According to various embodiments, in the first detection mode, the electronic device 101 may enable the IR emitter of the IR sensor to scan light sparsely to a target area where a user is present. In addition, in the second detection mode, the electronic device 101 may enable the IR emitter of the IR sensor to scan light densely to a target area where a user is present. The electronic device 101 may change the density of light scanned by the IR emitter by changing a mask or lens provided in front of the IR emitter.
[0113] Figure 4 is a diagram illustrating an example relocation operation of an electronic device for gesture recognition according to various embodiments of the present disclosure.
[0114] According to various embodiments, the distance 410 between the electronic device 101 and the user 401 may be changed by movement of the electronic device 101. Using one or more sensors under the control of the processor 120, the electronic device 101 may determine a distance 411 between the first electronic device 101 and the user 401. Under the control of the processor 120, the electronic device 101 may detect a user gesture in a gesture area corresponding to the determined first distance 411. In addition, under the control of the processor 120, the electronic device 101 may determine whether any movement thereof is required. If so, the electronic device 101 may determine a second distance 413 between the electronic device 101 and the user 401 using one or more sensors under the control of the processor 120, and move to a position corresponding to the determined second distance 413 by operating the driver included therein by controlling the behavior module 163.
[0115] According to various embodiments, determining whether any movement of the electronic device 101 is required may be performed based on a user's request. When it is determined that the movement is required, the electronic device 101 may run a driver in the behavior module 163 to move from a position corresponding to the first distance 411 to another position corresponding to the second distance 413.
[0116] According to various embodiments, if a user gesture is not recognized even after moving from a position corresponding to the first distance 411 to another position corresponding to the second distance 413 , the electronic device 101 may change from the first detection mode to the second detection mode.
[0117] Figure 5 is a diagram illustrating example repositioning and detection mode selection operations of an electronic device for gesture recognition according to various embodiments of the present disclosure.
[0118] According to various embodiments, the electronic device 101 may determine a first distance between the electronic device 101 and the user 500 using one or more sensors under the control of the processor 120 .
[0119] When the user 500 does not move, the distance between the electronic device 101 located at the first position 510A and the user 500 will be referred to as a first distance, and the distance between the electronic device 101 located at the second position 510B and the user 500 will be referred to as a second distance.
[0120] Under the control of the processor 120 , the electronic device 101 may detect the gesture of the user 500 in the gesture area corresponding to the determined first distance using the first detection mode.
[0121] Under the control of the processor 120, the electronic device 101 may determine whether movement of the electronic device 101 is necessary. This determination operation may depend on, for example, the distance between the electronic device 101 and the user 500, and on specific content that is mainly executed at the distance.
[0122] The external electronic device 501 and the electronic device 101 may be in a communication connection state. The electronic device 101 may recognize content being displayed (or being executed) in the external electronic device 501 .
[0123] For example, when yoga content is being displayed in the external electronic device 501, and when the user 500 follows a yoga pose while watching the yoga content, the electronic device 101 may move to a position (e.g., the second position 510B) that allows detection of the whole body pose of the user 500. For example, the electronic device 101 may control the behavior module 163 to operate a driver under the control of the processor 120, thereby moving to the second position 510B. At the second movement position 510B, the electronic device 101 may detect the pose of the user 500 in the first detection mode under the control of the processor 120.
[0124] In various embodiments, if the electronic device 101 fails to recognize the gesture of the user 500 even when trying to detect the user gesture at the second position 510B in the first detection mode under the control of the processor 120, the electronic device 101 may detect the user gesture in the second detection mode.
[0125] For example, for gesture recognition, the electronic device 101 may first change position (eg, move) under the control of the processor 120 and change the detection mode.
[0126] According to various embodiments, under the control of the processor 120, the electronic device 101 may determine whether to move based on the type of application associated with the electronic device 101. Based on the determination result, the electronic device 101 may move from the first position 510A to the second position 510B by executing the driver of the behavior module 163.
[0127] For example, when an application requiring facial recognition or an application requiring depth measurement is being executed in an external electronic device (e.g., 102, 104, or 108) or in the electronic device 101, the electronic device 101 may determine that movement is required and therefore move from the first position 510A to the second position 510B by running a driver of the behavior module 163.
[0128] Figure 6 is a flowchart illustrating an example repositioning and detection mode selection operation of an electronic device for gesture recognition according to various embodiments of the present disclosure.
[0129] According to various embodiments, at operation 601 , the electronic device 101 may determine a distance between the electronic device 101 and a user using one or more sensors under the control of the processor 120 .
[0130] In various embodiments, one or more sensors may include, for example, but not limited to, a proximity light sensor, a three-dimensional (3D) depth sensor, a posture sensor, an accelerometer, a gyroscope sensor, a geomagnetic sensor, an infrared (IR) sensor, an ultrasonic sensor, a microphone, a camera sensor, a radio frequency (RF) sensor, and / or a touch sensor (or display), etc.
[0131] The proximity light sensor may be a proximity sensor and / or an illumination sensor, and may be a sensor included in the sensor module 176. The 3D depth sensor may be a depth camera 184. The camera sensor may be one of a dual camera, a 360-degree camera, a spherical camera, a 2D camera 182, or an infrared-based depth camera 184. The 3D depth sensor and the camera sensor may include a camera module 180.
[0132] Using one or more sensors, the electronic device 101 may measure and determine the distance between the electronic device 101 and the user.
[0133] For example, the 3D depth sensor includes an IR transmitter and a depth image CMOS. Using the time difference when an IR signal emitted from the IR transmitter is reflected by a certain object (eg, user) and returns, the distance between the electronic device 101 and the object (eg, user) can be measured.
[0134] In various embodiments, the camera sensor may acquire and process an image containing a specific object (eg, a user), determine the size of the object contained in the image, and thereby measure the distance between the electronic device 101 and the object (eg, the user).
[0135] In various embodiments, when a user wears a communication accessory (e.g., a wearable device such as Figure 1 When an external electronic device 102 or 104 is connected to the user, the RF sensor can measure the received signal strength indication (RSSI) of the signal received from the accessory to determine the distance between the electronic device 101 and the user. The RF sensor can be capable of communicating with Figure 1 The communication module 190 is used to communicate with the external electronic device 102 or 104 shown in FIG.
[0136] In various embodiments, a sound signal (eg, voice, clapping, stomping, etc.) generated by the user may be detected, and the transmission time of the sound signal may be calculated to measure the distance between the electronic device 101 and the user.
[0137] According to various embodiments, at operation 603 , under the control of the processor 120 , the electronic device 101 may determine whether the determined distance is within the first distance.
[0138] When the determined distance is within the first distance, the electronic device may perform operation 605 under the control of the processor 120. When the determined distance is not within the first distance, the electronic device may perform operation 609 under the control of the processor 120.
[0139] In operation 605 , under the control of the processor 120 , the electronic device 101 may detect a user gesture in a first detection mode within a first distance.
[0140] In various embodiments, the electronic device 101 may store a gesture recognition algorithm in the memory 130. In addition, based on the gesture recognition algorithm, the electronic device 101 may process a user gesture detected in the gesture area.
[0141] In various embodiments, based on a gesture recognition algorithm, the electronic device 101 can track the movement or shape of the joints connecting the user's head, arms, shoulders, hands, etc. (e.g., moving the hand to the left or right, drawing a circle with the hand, moving the hand back and forth, moving the hand closer or farther, etc.), thereby recognizing the user's gesture. The recognized gesture can be used as an input command.
[0142] In various embodiments, the priority of gesture recognition depending on distance can be defined differently according to the user's selection. In various embodiments, according to the gesture recognition algorithm, the electronic device 101 can be configured to first recognize gestures made by movement of relatively large joints (such as arms or heads) based on distance (for example, when the distance between the electronic device 101 and the user is far). In various embodiments, according to the gesture recognition algorithm, the electronic device 101 can be configured to first recognize finger gestures with smaller movements based on distance (for example, when the distance is close). In various embodiments, according to the gesture recognition algorithm, the electronic device 101 can be configured to recognize various gestures from the movement or shape of the finger joints based on distance (for example, when the distance is close).
[0143] According to various embodiments, at operation 607 , under the control of the processor 120 , the electronic device 101 may perform a predetermined action linked to the gesture detected in the first detection mode.
[0144] In various embodiments, under the control of the processor 120 , the electronic device 101 may determine its distance and change the detection mode based on the type of application associated with the electronic device 101 .
[0145] In various embodiments, when an application requiring facial recognition or an application requiring depth measurement is being executed in an external electronic device (eg, 102, 104, or 108) or the electronic device 101, the electronic device 101 may determine whether to change the detection mode based on the type of application being executed.
[0146] For example, in the case where the user's face or upper body is recognized as a gesture at a first distance, the motion of the entire area may need to be sensed in order to determine the user's full body gesture. In this case, the electronic device 101 may determine that the detection mode needs to be changed.
[0147] In various embodiments, based on a user's request, the electronic device 101 may determine whether to change the detection mode.
[0148] Meanwhile, when it is determined at operation 603 that the determined distance is not within the first distance, under the control of the processor 120, the electronic device 101 may determine whether the determined distance is within the second distance at operation 609. In various embodiments, the first distance may be closer or farther than the second distance.
[0149] When the determined distance is within the second distance, the electronic device 101 may perform operation 611 under the control of the processor 120. When the determined distance is not within the second distance, the electronic device 101 may terminate the gesture recognition process under the control of the processor 120.
[0150] In various embodiments, at operation 611 , under the control of the processor 120 , the electronic device 101 may determine whether a user gesture may be detected in the second detection mode.
[0151] When the user gesture can be detected in the second detection mode, the electronic device 101 may perform operation 613 under the control of the processor 120. When the user gesture cannot be detected in the second detection mode, the electronic device 101 may perform operation 615 under the control of the processor 120.
[0152] At operation 613, the electronic device 101 may detect a user gesture in the second detection mode under the control of the processor 120. At operation 607, under the control of the processor 120, the electronic device 101 may perform a predetermined action linked to the gesture detected in the second detection mode.
[0153] At operation 615 , the electronic device 101 may move to another location to change the distance from the user under the control of the processor 120 .
[0154] In various embodiments, at operation 615 , under the control of the processor 120 , the electronic device 101 may move to a position corresponding to a distance closer than the distance determined at operation 601 .
[0155] In various embodiments, after completing the movement for changing the distance from the user at operation 615, the electronic device 101 may detect the user gesture in the first or second detection mode at operation 617 under the control of the processor 120. At operation 607, under the control of the processor 120, the electronic device 101 may perform a predetermined action linked to the gesture detected in the first or second detection mode.
[0156] The first detection mode and the second detection mode may have, for example but not limited to, the following differences.
[0157] In various embodiments, the electronic device 101 may independently control each photodiode input of the pixel array of the camera module 180, and may also independently control (turn on or off) pixels according to a predetermined (based on factors such as distance) resolution necessary to recognize user gesture input.
[0158] For example, when the user makes a gesture at a first distance (e.g., a close distance such as 30 cm), the image sensor of the camera module 180 may recognize the gesture in a first detection mode (e.g., a low-resolution mode such as 8×8 pixels). However, when the user makes a gesture at a second distance (e.g., a medium distance such as 1 m or a long distance such as 3 m), the image sensor of the camera module 180 may recognize the gesture in a second detection mode (e.g., a medium-resolution mode such as 64×64 pixels or a high-resolution mode such as full pixels). Therefore, regardless of the distance, the same recognition rate (e.g., recognition of 20 spots) can be provided.
[0159] In various embodiments, the image sensor of the camera module 180 may include a color filter layer for filtering the light incident on the photodiode by wavelength. Typically, four filters of RGBG comprise a group corresponding to one pixel. Such a group of four filters may be arranged in a square or quadrilateral.
[0160] According to various embodiments, when operating in a first detection mode (e.g., low-resolution mode) at a first distance (e.g., close distance), the electronic device 101 may recognize data of one pixel corresponding to the RGBG filter group as a single pixel itself. When operating in a second detection mode (e.g., monochrome mode) at a second distance (e.g., long distance), the electronic device 101 may operate the color filter in a form of recognizing only one color (e.g., black and white) other than RGBG, thereby improving resolution.
[0161] According to various embodiments, when the group of four filters of RGBG is arranged in a square or quadrilateral, the electronic device 101 operating in a first detection mode (e.g., low-resolution mode) at a first distance (e.g., close distance) can recognize the interpolated pixel data corresponding to the RGBG filter group as a single pixel itself. The RGBG filter group can be interpolated using the Bayer interpolation technique. The electronic device 101 operating in a second detection mode (e.g., high-resolution mode) at a second distance (e.g., long distance) can operate all pixels to improve resolution.
[0162] According to various embodiments, the electronic device 101 may be configured to enable a specific portion of the color filter to pass IR wavelengths when the distance is farther. This can improve the recognition rate of gestures at a long distance (eg, the second distance) by recognizing IR light reflected from the outside.
[0163] According to various embodiments, in the first detection mode, the electronic device 101 may enable the IR emitter of the IR sensor to scan light sparsely to a target area at a first distance (e.g., a short distance) where a user exists. In addition, in the second detection mode, the electronic device 101 may enable the IR emitter of the IR sensor to scan light densely to a target area at a second distance (e.g., a long distance) where a user exists. The electronic device 101 may change the density of light scanned by the IR emitter by changing a mask or lens provided in front of the IR emitter.
[0164] According to various embodiments, when the distance between the electronic device 101 and the user is a second distance that exceeds the distance at which the 3D sensor normally operates (e.g., the first distance), the electronic device 101 may recognize the movement of the external electronic device through a motion / gesture detection module installed in an external electronic device (e.g., electronic device 102 or 104, or a wearable device) of the hand making the gesture, rather than recognizing the movement of the hand through a distance image sensor (e.g., time of flight (ToF)). For example, the motion / gesture detection module of the external electronic device may detect the movement of the external electronic device and send the motion / gesture information to the electronic device 101 to recognize the gesture input.
[0165] According to various embodiments, the electronic device 101 may change the user's gesture input area depending on the distance between the user and the electronic device 101. When the user is at a first distance from the electronic device (e.g., a close distance), the electronic device 101 may operate in a first detection mode (e.g., a mode for finger gesture input). In this mode, the electronic device 101 may recognize each individual finger gesture, thereby enabling detailed gesture input, such as character input. In addition, when the user is at a second distance (e.g., a long distance) or farther from the electronic device 101, the electronic device 101 may operate in a second detection mode (e.g., a mode for upper body gesture input). In this mode, the electronic device 101 may mainly recognize the user's arm movements. In addition, when the user is farther from the electronic device 101, the electronic device 101 may recognize gestures through the user's entire body.
[0166] According to various embodiments, the motion / posture detection module of the electronic device 101 obtains information about the distance between the electronic device and the user. In the case of a relatively close distance (e.g., 1 to 3 cm), the electronic device may track the position of the gesture input object (e.g., hand) only in a vicinity including the user's head / chest. In addition, in the case of a relatively medium distance (e.g., 1 m), the electronic device may track the position of the gesture input object (e.g., arm) in an area including the user's upper body. In the case of a relatively long distance (e.g., 3 m), the electronic device may track the position of the gesture input object (e.g., body) in an area including the user's entire body.
[0167] According to various embodiments, the same gesture may be recognized at different gesture sizes depending on the distance between the electronic device 101 and the gesture input object. To compensate for this, a scale factor may be determined and used. For example, when the hand moves 10 cm from left to right at a first distance, this hand motion may have a size of 100 pixels in the entire image for recognition. When the same motion is performed at a second distance, the hand motion may have a size of 25 pixels in the entire image for recognition. In such a case, the depth sensor may identify the ratio of the second distance between the electronic device and the user to the first distance, and thus may scale the motion of the hand to generate a scaled gesture. Therefore, using the scale factor, the electronic device 101 may detect a gesture scaled at a first distance in a first detection mode, and may also detect a gesture scaled at a second distance in a second detection mode.
[0168] Figure 7 is a diagram illustrating an example change in a detection mode for gesture recognition of an electronic device according to various embodiments of the present disclosure.
[0169] In various embodiments, the electronic device 101 may independently control each photodiode input of the pixel array of the camera module 180 and may independently control (turn on or off) the pixels according to a predetermined (based on factors such as distance) resolution necessary to recognize user gesture input.
[0170] For example, when the user 700 makes a gesture at a first distance (e.g., a close distance 711), the image sensor of the camera module 180 may recognize the gesture in a first detection mode (e.g., a low-resolution mode 701 such as 8×8 pixels). However, when the user 700 makes a gesture at a second distance (e.g., a medium-resolution mode 703 such as 64×64 pixels, or a high-resolution mode 705 such as full pixels), the image sensor of the camera module 180 may recognize the gesture in a second detection mode (e.g., a medium-resolution mode 703 such as 64×64 pixels, or a high-resolution mode 705 such as full pixels). Therefore, regardless of the distance, the same recognition rate (e.g., recognition of 20 spots) can be provided.
[0171] Figure 8 is a diagram illustrating an example change in a detection mode for gesture recognition of an electronic device according to various embodiments of the present disclosure.
[0172] In various embodiments, the image sensor of the camera module 180 may include a color filter layer for filtering light incident on the photodiode by wavelength. In addition, the electronic device 101 may control the color filter layer based on the detection mode.
[0173] Generally, four filters of RGBG include a group corresponding to one pixel. Such a group of four filters can be arranged in a square or a quadrilateral.
[0174] According to various embodiments, when the user 800 makes a gesture at a first distance (e.g., close distance 811) and when the electronic device 101 operates in a first detection mode (e.g., low resolution mode 801), the electronic device 101 may identify data of a pixel corresponding to the RGBG filter group as a single pixel itself.
[0175] In addition, when the user 800 makes a gesture at a second distance (e.g., medium distance 813) and when the electronic device 101 operates in a second detection mode (e.g., monochrome mode 803), the electronic device 101 can operate the color filter in the form of recognizing only one color (e.g., black and white) other than RGBG, thereby improving resolution.
[0176] According to various embodiments, when the user 800 makes a gesture at a second distance (e.g., long distance 815), the electronic device 101 may be configured to enable a specific portion of the color filter to further pass IR wavelengths in the second detection mode 805. This can improve the recognition rate of gestures at a long distance by recognizing IR light reflected from the outside.
[0177] According to various embodiments, when a set of four filters of RGBG are arranged in a square or quadrilateral, and when the user 800 makes a gesture at a first distance 811, the electronic device 101 operating in the first detection mode 801 (e.g., low-resolution mode) may recognize the interpolated pixel data corresponding to the RGBG filter group as a single pixel itself. The RGBG filter group may be interpolated using the Bayer interpolation technique. When the user 800 makes a gesture at a second distance 813 or 815, the electronic device 101 operating in the second detection mode 803 or 805 may operate all pixels to increase resolution.
[0178] Fig. 9 is a diagram illustrating an example change in a detection mode for gesture recognition of an electronic device according to various embodiments of the present disclosure.
[0179] According to various embodiments, depending on the distance from the user, the electronic device 101 may change a mask or lens provided in front of the IR emitter of the IR sensor under the control of the processor 120 to change the density of light scanning at the IR emitter of the IR sensor.
[0180] For example, for a user located at a first distance 910 , the electronic device 101 may enable the IR transmitter to scan light with a first gap 930 in a first detection mode 901 .
[0181] In addition, for a user at the second distance 920, the electronic device 101 may enable the IR transmitter to scan light with a second gap 940 in the second detection mode 903. When the second distance 920 is greater than the first distance, the second gap 940 may be smaller than the first gap 930.
[0182] Fig.10 is a diagram illustrating an example repositioning and detection mode changing operation of an electronic device for gesture recognition according to various embodiments of the present disclosure.
[0183] According to various embodiments, under the control of the processor 120 , the electronic device 101 may compensate for the gesture input using a scaling factor according to the distance.
[0184] In various embodiments, when the distance between the electronic device 101 and the user 1000 making the gesture changes, even if the same gesture is recognized, the electronic device 101 can recognize the gesture as having a different gesture size. In order to compensate for this, a scaling factor can be determined and used. For example, when the hand moves 10 cm from left to right as a rolling gesture at a first distance 1001, this hand movement can have a size of 100 pixels in the entire image for recognition, as shown in reference numeral 1010. When the same movement is performed at a second distance 1003, the hand movement can have a size of 25 pixels in the entire image for recognition, as shown in reference numeral 1010. In this case, the depth sensor can identify the ratio of the second distance between the electronic device 101 and the user 1000 to the first distance, so that the hand movement can be scaled to generate a scaled gesture 1020.
[0185] Therefore, using the zoom factor, the electronic device 101 can detect the zoomed gesture in the first detection mode at the first distance, and detect the zoomed gesture in the second detection mode at the second distance. In the event that the zoomed gesture cannot be recognized in the second detection mode at the second distance, the electronic device 101 can move to another location to find a recognizable distance, as described above. Figure 6 As described in operation 615 in .
[0186] Fig.11 is a diagram illustrating an example repositioning and detection mode changing operation of an electronic device for gesture recognition according to various embodiments of the present disclosure.
[0187] According to various embodiments, the electronic device 101 may set the gesture input area of the user 1100 differently depending on the distance between the user 1100 and the electronic device 101. When the user 1100 is at a first distance 1101 from the electronic device 101, the electronic device 101 may operate in a first detection mode 1110 (e.g., a mode for finger gesture input). In this mode, the electronic device 101 may recognize each individual finger gesture 1111, thereby enabling detailed gesture input, such as character input. When the user 1100 is at a second distance 1103 farther from the electronic device 101 than the first distance 1101, the electronic device 101 may operate in a second detection mode 1120 while setting the upper body 1113 of the user as the gesture input area. In this mode, the electronic device 101 may mainly recognize the arm movement of the user. When the user is at another second distance 1105 from the electronic device 101, the electronic device 101 may recognize gestures through the whole body 1115 of the user.
[0188] According to various embodiments, the motion / posture detection module of the electronic device 101 obtains information about the distance between the electronic device 101 and the user 1100. When the distance is determined to be a first distance 1101, the electronic device may track the position of the gesture input object (e.g., hand) only in a proximate area including the user's head / chest. In addition, when the distance is determined to be a second distance 1103, the electronic device may track the position of the gesture input object (e.g., arm) in an area including the user's upper body. When the distance is determined to be another second distance 1105, the electronic device may track the position of the gesture input object (e.g., body) in an area including the user's entire body. In the event that the user's gesture at the second distance 1105 is not recognized, as described above in Figure 6 As described at operation 615 in , the electronic device 101 may move to another location to find a recognizable distance.
[0189] Fig.12 is a diagram illustrating an example of a gesture detection area according to various embodiments of the present disclosure.
[0190] According to various embodiments, when the recognition range is changed while the distance between the user 1200 and the electronic device 101 is unchanged, the processed data size and power consumption may be changed when recognizing the user gesture. Fig.12 Three examples of recognition ranges (e.g., gesture detection areas) are shown. In different gesture detection areas, the number of active pixels, power consumption, and processing speed of the camera module 180 are different. As shown in Table 1 below, full body tracking 1210 may require more active pixels (e.g., 300 x 300) and more power consumption than upper body tracking 1220 (e.g., 150 x 150) and hand tracking 1230.
[0191] [Table 1]
[0192] Posture detection area Active Pixels Power consumption Processing speed Full body tracking 300*300 20mW More than 1ms Upper body tracking 150*150 15mW More than 1ms Hand Tracking 50*50 7mW More than 0.5ms
[0193] When a gesture detection area is designated, the electronic device 101 may turn on only pixels corresponding to the gesture detection area in the camera module 180 and turn off the remaining pixels.
[0194] Active pixels refer to specific pixels that correspond to the gesture detection area and are turned on. However, in an exemplary embodiment, the remaining pixels except the gesture detection area may not necessarily be turned off. In this case, the gesture detection area can be densely identified, while other areas can be sparsely identified.
[0195] Fig.13 is a diagram illustrating an example change in a detection mode for gesture recognition of an electronic device according to various embodiments of the present disclosure.
[0196] According to various embodiments, when the user is located at a second distance 1303 corresponding to the second gesture detection area 1320 outside the first distance 1301 corresponding to the first gesture detection area 1310, the electronic device 101 may recognize the movement of the external electronic device 103 through a motion / gesture detection module installed in the external electronic device (e.g., wearable device) 103 of the hand making the gesture, instead of recognizing the movement of the hand through a distance image sensor (e.g., time of flight (ToF)). For example, the motion / gesture detection module of the external electronic device 103 may detect the movement of the external electronic device and send the motion / gesture information to the electronic device 101 to recognize the gesture input.
[0197] Fig.14 is a diagram illustrating an example gesture recognition method at a close distance between an electronic device and a user according to various embodiments of the present disclosure.
[0198] As shown by reference numeral 1401 , based on distance (eg, when the distance between the electronic device 101 and the user is close), the electronic device 101 may be configured to first recognize finger gestures with small movements according to a gesture recognition algorithm under the control of the processor 120 .
[0199] In addition, as shown in reference number 1403, based on distance (for example, when the distance between the electronic device 101 and the user is short), the electronic device 101 can be configured to recognize various hand gestures from the movement or shape of finger joints according to a gesture recognition algorithm under the control of the processor 120.
[0200] Reference numeral 1405 indicates various examples of finger / hand gestures recognizable at a short distance.
[0201] Fig.15 is a diagram illustrating an example gesture recognition method at a long distance between an electronic device and a user according to various embodiments of the present disclosure.
[0202] As shown by reference number 1501, based on distance (for example, when the distance between the electronic device 101 and the user is far), the electronic device 101 can be configured to first recognize gestures made through movements of relatively large joints (such as arms or heads) according to a gesture recognition algorithm under the control of the processor 120.
[0203] In addition, as shown in reference number 1503, the electronic device 101 can track the movement or shape of the joints connecting the user's head, arms, shoulders, hands, etc. (for example, moving the hands to the left or right, moving the hands back and forth, moving the hands closer or farther, etc.), thereby identifying the user's gestures according to the gesture recognition algorithm under the control of the processor 120.
[0204] Reference numeral 1505 indicates various examples of recognized gestures to be used as input commands.
[0205] Fig.16 is a diagram illustrating example gesture detection areas depending on executed content according to various embodiments of the present disclosure.
[0206] When game content requiring large motions is executed in the external electronic device 102 connected to the electronic device 101 , the electronic device 101 may move to another location or change the gesture detection area so that the effective gesture area includes the entire body 1600 of the user.
[0207] Fig.17 is a diagram illustrating example gesture detection areas depending on executed content according to various embodiments of the present disclosure.
[0208] When content requiring relatively small motions, such as a browser or a video player, is executed in the external electronic device 102 connected to the electronic device 101 , the electronic device 101 may determine whether to move such that the valid gesture area includes only the upper body 1701 of the user.
[0209] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a household appliance, etc. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.
[0210] It should be understood that the various exemplary embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, and include various changes, equivalents or replacements for corresponding embodiments. With respect to the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the singular form of the noun corresponding to the project may include one or more things, unless the relevant context explicitly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any one or all possible combinations of the items listed together in the corresponding one of these phrases. As used herein, terms such as "first" and "second", or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component (e.g., importance or order) in other aspects. It will be understood that if an element (e.g., a first element) is referred to as being "coupled with," "coupled to," "connected with," or "connected to" another element (e.g., a second element) (with or without the terms "operably" or "communicatively"), the element may be coupled to the other element directly (e.g., by wire), wirelessly, or via a third element.
[0211] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms, such as "logic," "logic block," "component," or "circuit." A module may be a single integral component, or its smallest unit or portion, adapted to perform one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).
[0212] Various embodiments as described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, a processor (e.g., processor 120) of a machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium, and execute the instruction with or without one or more other components under the control of the processor. This allows the machine to be operated to perform at least one function according to at least one of the called instructions. The one or more instructions may include code generated by a compiler or code executable by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, a "non-transitory" storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being permanently stored in a storage medium and data being temporarily stored in a storage medium.
[0213] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or via an application store (e.g., PlayStore). TM ), or distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server.
[0214] According to various embodiments, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as performed by the corresponding components in the multiple components before integration. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of these operations may be performed or omitted in different orders, or one or more other operations may be added.
[0215] While the present disclosure has been shown and described with reference to various example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the subject matter defined, for example, by the appended claims and their equivalents.
Claims
1. A gesture recognition method for an electronic device, the method include: determining, using at least one of the one or more sensors, a first distance between the electronic device and a user; acquiring, by at least one of the one or more sensors, information corresponding to a gesture of the user detected in a gesture area corresponding to the determined first distance; as well as Using a drive module to move the electronic device so that the electronic device is located at a position corresponding to a second distance, the second distance depending on a type of application associated with the electronic device; When the distance between the electronic device and the user is the second distance in response to the movement, acquiring information corresponding to the gesture of the user through at least one of the one or more sensors in a first detection mode of the one or more sensors; When the user's gesture in the gesture area corresponding to the second distance is not recognized, the one or more sensors are changed from a first detection mode to a second detection mode, and the information is acquired in the second detection mode of the one or more sensors.
2. The method according to claim 1, in, Determining whether the electronic device needs to be moved is based on the type of the application executed in the electronic device or executed in an external electronic device connected to the electronic device.
3. The method according to claim 2, in, The one or more sensors include at least one of: a proximity light sensor, a three-dimensional (3D) depth sensor, a posture sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared (IR) sensor, an ultrasonic sensor, a microphone, a camera sensor, a radio frequency (RF) sensor, or a touch sensor.
4. An electronic device, include: one or more sensors; A driving module including a driving circuit configured as a mobile electronic device; Memory; as well as A processor configured to execute instructions to control the electronic device: determining a distance between the electronic device and a user using a position detection sensor among the one or more sensors; When the determined distance is within a first distance, detecting a gesture of the user within the first distance in a first detection mode using a motion detection sensor among the one or more sensors; When the determined distance is within a second distance and the gesture of the user is detectable in a second detection mode, changing the one or more sensors from the first detection mode to the second detection mode and detecting the gesture of the user in the second detection mode of the one or more sensors different from the first detection mode using the motion detection sensor among the one or more sensors; when the determined distance is within a second distance and the gesture of the user is undetectable after switching from the first detection mode of the one or more sensors to the second detection mode, controlling the electronic device to move the electronic device and detecting the gesture of the user at a moved position using the motion detection sensor of the one or more sensors in the second detection mode of the one or more sensors; as well as A predetermined action linked to the detected gesture is performed.
5. The electronic device as claimed in claim 4, in, The processor is further configured to control the electronic device to independently control photodiode inputs of a pixel array of a camera module based on corresponding ones of the first detection mode and the second detection mode.
6. The electronic device as claimed in claim 5, in, The processor is further configured to control the electronic device to change resolution by turning on or off the photodiodes of the pixel array of the camera module based on the distance.
7. The electronic device as claimed in claim 4, in, The processor is further configured to control the electronic device to control a color filter layer configured to filter light incident on a photodiode by a wavelength in an image sensor module of a camera module based on a corresponding detection mode of the first detection mode and the second detection mode.
8. The electronic device as claimed in claim 4, in, The processor is further configured to control the electronic device to change a mask or a lens disposed in front of the IR emitter based on a corresponding detection mode of the first detection mode and the second detection mode.
9. The electronic device as claimed in claim 4, in, The processor is further configured to control the electronic device to compensate for gesture input using a scaling factor based on a corresponding detection mode of the first detection mode and the second detection mode.
10. The electronic device as claimed in claim 4, in, The processor is further configured to control the electronic device to differently set a gesture input area of the user based on a corresponding detection mode of the first detection mode and the second detection mode.
11. The electronic device as claimed in claim 4, in, The processor is further configured to control the electronic device to receive a gesture input from an external electronic device based on a corresponding detection mode of the first detection mode and the second detection mode.
12. The electronic device as claimed in claim 4, in, The processor is further configured to control the electronic device to recognize a finger gesture or a gesture made by movement of a large joint based on a corresponding detection mode in the first detection mode and the second detection mode.
