Apparatus and method for authenticating an object in an electronic device

Through the wireless communication module and antenna array combined with camera and millimeter wave equipment, wide and narrow beam technology is used to perform object authentication under different brightness conditions, solving the problem of object authentication in the prior art that is susceptible to fraud and difficult to use in low-light environments, realizing the problem of object authentication in the existing technology. Efficient and safe certification effect.

CN112424776BActive Publication Date: 2024-11-22SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980046524.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-07-11
Filing Date
2019-07-08
Publication Date
2024-11-22
Estimated Expiration
2039-07-08

AI Technical Summary

Technical Problem

When authenticating objects in existing electronic devices, facial recognition methods are susceptible to malicious deception and are difficult to use in low-light environments. They often require multiple camera modules, which increases the complexity and cost of the equipment.

Method used

Through wireless communication modules and antenna arrays, a wide beam is used to quickly search for object locations and convert to narrow beams when the object is determined to exist for high resolution authentication, combining cameras and millimeter wave devices for authentication operations under different brightness conditions.

Benefits of technology

It realizes efficient and secure object authentication in different environments, reduces equipment complexity and cost, and improves the ability to resist malicious attacks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112424776B_ABST
    Figure CN112424776B_ABST
Patent Text Reader

Abstract

An electronic device and a method for authenticating an object by the electronic device are provided. The method comprises: transmitting a first directional beam sequence having a first beam width through the antenna array to scan a first area having a first size; receiving a first reflected wave sequence generated by reflection of the first directional beam sequence from an object through the antenna array; transmitting a second directional beam sequence having a second beam width narrower than the first beam width through the antenna array based on at least a portion of the received first reflected wave sequence to scan a second area included in the first area and having a second size smaller than the first size; receiving a second reflected wave sequence generated by reflection of the second directional beam sequence from an object through the antenna array; and authenticating the object based on at least a portion of the second reflected wave sequence.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an apparatus and method for authenticating an object in an electronic device. Background Art

[0002] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure.

[0003] As electronic device performance has improved, various services and additional functions provided by the electronic devices have gradually expanded. In order to increase the value of electronic devices and meet various needs of users, various applications that can be run by electronic devices have been developed.

[0004] Due to the high amount of personal data that electronic devices may carry, it becomes important to verify the user's authorization before providing access to the electronic device. Summary of the invention

[0005] Solution to the problem

[0006] According to aspects of the present disclosure, an electronic device is provided. The electronic device includes an antenna array, a wireless communication module electrically connected to the antenna array and configured to form a directional beam through the antenna array, at least one processor operably connected to the wireless communication module, and a memory operably connected to the at least one processor. The memory stores instructions that cause the at least one processor to perform multiple operations, the operations including: sending a first directional beam sequence with a first beam width through the antenna array to scan a first area with a first size, receiving a first reflection wave sequence generated by reflection of the first directional beam sequence from an object through the antenna array, sending a second directional beam sequence with a second beam width narrower than the first beam width through the antenna array based on at least a portion of the received first reflection wave sequence to scan a second area included in the first area and having a second size smaller than the first size, receiving a second reflection wave sequence generated by reflection of the second directional beam sequence from the object through the antenna array, and authenticating the object based on at least a portion of the second reflection wave sequence.

[0007] According to another aspect of the present disclosure, an electronic device is provided. The electronic device includes a camera, an antenna array, a wireless communication module electrically connected to the antenna array and configured to form a directional beam through the antenna array, a sensor module configured to sense illumination intensity, and at least one processor operably connected to the camera, the wireless communication module, and the sensor module. The at least one processor is configured to identify the brightness around the electronic device through the sensor module, when the sensed brightness is equal to or lower than a set level, perform a first object authentication operation through the antenna array and the wireless communication module, and when the sensed brightness is higher than the set level, perform a second object authentication operation through the camera, the antenna array, and the wireless communication module. The first object authentication operation sends a first directional beam sequence having a first beam width through an antenna array to scan a first area having a first size, receives a first reflected wave sequence generated by reflection of the first directional beam sequence from an object through the antenna array, sends a second directional beam sequence having a second beam width narrower than the first beam width through the antenna array based on at least a portion of the received first reflected wave sequence to scan a second area included in the first area and having a second size smaller than the first size, receives a second reflected wave sequence generated by reflection of the second directional beam sequence from the object through the antenna array, and authenticates the object based on at least a portion of the second reflected wave sequence.

[0008] According to another aspect of the present disclosure, a method for authenticating an object by an electronic device is presented. The method includes sending a first directional beam sequence with a first beam width through an antenna array to scan a first area with a first size, receiving a first reflected wave sequence generated by reflection of the first directional beam sequence from an object through the antenna array, sending a second directional beam sequence with a second beam width narrower than the first beam width through the antenna array based on at least a portion of the received first reflected wave sequence to scan a second area included in the first area and having a second size smaller than the first size, receiving a second reflected wave sequence generated by reflection of the second directional beam sequence from the object through the antenna array, and authenticating the object based on at least a portion of the second reflected wave sequence.

[0009] According to another aspect of the present disclosure, a method for authenticating an object by an electronic device is provided. The method includes: sensing the brightness around the electronic device by a sensor module; when the sensed brightness is lower than a set level, performing a first object authentication operation by an antenna array and a wireless communication module; and performing a second object authentication operation when the sensed brightness is higher than a set level, for authenticating the object by an image acquired by a camera and a signal received by the antenna array and the wireless communication module. The first object authentication operation can send a first directional beam sequence with a first beam width by an antenna array to scan a first area with a first size, receive a first reflection wave sequence generated by reflection of the first directional beam sequence from an object by the antenna array, send a second directional beam sequence with a second beam width narrower than the first beam width by the antenna array based on at least a portion of the received first reflection wave sequence to scan a second area included in the first area and having a second size smaller than the first size, receive a second reflection wave sequence generated by reflection of the second directional beam sequence from the object by the antenna array, and authenticate the object based on at least a portion of the second reflection wave sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a block diagram illustrating an electronic device in a network environment according to some embodiments;

[0012] Figure 2 shows a configuration of an electronic device according to some embodiments;

[0013] Figure 3A and Figure 3B A method for an electronic device to determine a first beam and a second beam according to some embodiments is shown;

[0014] FIG. 4A to FIG. 4E A method for an electronic device to determine a beam according to some embodiments is shown;

[0015] Figure 5 The operation of an electronic device determining acquired object information as a template through machine learning according to some embodiments is shown;

[0016] Figure 6 shows a configuration of an electronic device according to some embodiments;

[0017] Figure 7 is a flow chart illustrating object authentication operations of an electronic device according to some embodiments;

[0018] Figure 8A method for an electronic device to control a beam for object authentication according to certain embodiments is shown;

[0019] Fig. 9 A method for an electronic device to authenticate an object by determining the liveness of the object according to some embodiments is shown;

[0020] Fig.10 shows a configuration of an electronic device according to some embodiments;

[0021] Fig.11 shows a configuration of an electronic device according to some embodiments;

[0022] Fig.12 shows a configuration of an electronic device for performing a second object authentication operation according to some embodiments; and

[0023] Fig.13 is a flow chart illustrating a method for an electronic device to authenticate an object based on a camera image and a beam according to some embodiments. DETAILED DESCRIPTION

[0024] Hereinafter, certain embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0025] As the performance of electronic devices has improved, various services and additional functions provided by the electronic devices have gradually expanded. In order to increase the value of electronic devices and meet various needs of users, various application programs executable by the electronic devices have been developed.

[0026] There are applications related to the camera function among the applications, and users can capture themselves or the background through the camera module installed in the electronic device. For example, the electronic device can perform an authentication function based on an image of an object captured using the camera module. For example, the object can be a face or an iris.

[0027] The facial recognition mechanism mainly used by electronic devices may use a method of using only an image of an object acquired by an RGB camera, or a method of authenticating an object by combining an image of an object acquired by an RGB camera and depth information acquired by a depth camera. The method of using only an image of an object acquired by an RGB camera may be susceptible to manipulation by a third party (e.g., a malicious spoofing attack using a photo or image stored in a smartphone). For the method of using both an RGB camera and a depth camera, the electronic device is required to include multiple camera modules, and when the ambient brightness is low, it is difficult for the electronic device to use the method of authenticating an object by a camera.

[0028] An electronic device according to some embodiments may provide an apparatus and method for authenticating an object through a wireless communication module.

[0029] An electronic device according to some embodiments may provide an apparatus and method for authenticating an object through a millimeter wave device when ambient brightness is outside a specific range.

[0030] A method of authenticating an object through an electronic device according to certain embodiments may quickly search a portion where the object exists by estimating the position of the object through beamforming with a wide beam, and authenticate the object through beamforming with a narrow beam in the portion where the object exists.

[0031] According to certain embodiments, the electronic device includes a camera and a wireless communication module, and is capable of acquiring object information through the camera and the wireless communication module when the brightness around the subject is brighter than or equal to the set brightness, and acquiring object information through the wireless communication module when the brightness around the subject is darker than the set brightness, so as to authenticate the object.

[0032] An electronic device according to some embodiments may perform an object authentication operation through a millimeter wave device without information about an image of an object.

[0033] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.

[0034] refer to Figure 1 , the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120 (hereinafter, "processor" may include both singular contexts and plural contexts), a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit.For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented to be embedded in the display device 160 (eg, a display).

[0035] The processor 120 may run instructions (e.g., program 140) stored in the memory, for example, to control at least one other component (e.g., hardware component or software component) of the electronic device 101 coupled to the processor 120, and may perform various data processing or calculations. According to an embodiment, as at least part of the data processing or calculation, the processor 120 may load a command or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a designated function. The auxiliary processor 123 may be implemented separately from the main processor 121, or implemented as a part of the main processor 121.

[0036] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 (not the main processor 121), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as a part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123.

[0037] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0038] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or an application program 146 .

[0039] The input device 150 may receive commands or data from outside the electronic device 101 (eg, a user) to be used by other components (eg, the processor 120) of the electronic device 101. The input device 150 may include, for example, a microphone, a mouse, or a keyboard.

[0040] The sound output device 155 can output a sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented as a part of the speaker or as a part of the speaker.

[0041] The display device 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0042] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly coupled to the electronic device 101.

[0043] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0044] The interface 177 may support one or more designated protocols to be used to directly (e.g., wired) or wirelessly couple the electronic device 101 to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0045] The connection end 178 may include a connector, wherein the electronic device 101 may be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0046] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0047] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0048] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0049] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0050] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors that can operate independently from the processor 120 (e.g., an application processor (AP)) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0051] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include one or more antennas, and at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the one or more antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). Signals or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna.

[0052] At least some of the above components may be coupled to each other via an inter-peripheral communication scheme (e.g., a bus, general purpose input output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and communicate signals (e.g., commands or data) therebetween.

[0053] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 coupled to the second network 199. Each of the electronic device 102 and the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be run on the electronic device 101 may be run on one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically perform a function or service or should perform a function or service in response to a request from a user or another device, the electronic device 101 may request one or more external electronic devices to perform at least a part of the function or service instead of running the function or service, or the electronic device 101 may request one or more external electronic devices to perform at least a part of the function or service in addition to running the function or service. The one or more external electronic devices that receive the request may perform at least a part of the requested function or service, or perform additional functions or additional services related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a part of the reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology or client-server computing technology may be used, for example.

[0054] The electronic device according to some embodiments disclosed herein may be various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0055] It should be understood that some embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalents or alternative forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of items listed together in a corresponding phrase in a plurality of phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish a corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being "coupled with," "connected with," with or without the terms "operably" or "communicatively," it means that the element can be coupled to the other elements directly (e.g., by wire), wirelessly, or via a third element.

[0056] As used herein, the term "module" may include a unit implemented in hardware or hardware storing instructions, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or the smallest unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0057] The various embodiments described herein may be implemented as a memory storing executable instructions that electrically, chemically, or magnetically change a memory (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a computer. For example, a processor (e.g., processor 120) of a machine (e.g., electronic device 101) calls at least one of the one or more instructions stored in the storage medium and runs at least one instruction. This enables the machine to operate to perform at least one function according to at least one of the called instructions. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0058] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be downloaded through an application store (e.g., Play Store TM ) The computer program product may be published (e.g., downloaded or uploaded) online, or may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily generated, or at least a portion of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a forwarding server).

[0059] According to certain embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. According to certain embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to certain embodiments, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performs one or more functions before integration. According to certain embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

[0060] Figure 2 An electronic device (eg, Figure 1 Configuration of electronic device 101).

[0061] refer to Figure 2 , the electronic device 101 may include a processor 200 , a memory 210 , a wireless communication module 220 , an antenna array 230 , and a display 240 .

[0062] In some embodiments, the electronic device 101 is configured to authenticate an object by transmitting a directional beam through the antenna array 230 and receiving a reflected signal. The antenna array 230 may be controlled by the processor 200. The processor 200 may execute instructions stored in the memory 210.

[0063] In some embodiments, the processor 200 may control the antenna array 230 to transmit a first directional beam having a first beam width through the antenna array. The antenna array may then receive a first reflected wave sequence generated by reflection of the first directional beam. The processor may then cause the antenna array to transmit a narrower beam and authenticate the object based on at least a portion of the reflected wave sequence from the narrower beam.

[0064] The electronic device 101 may include a housing (not shown). The housing may include a first plate facing in a first direction, a second plate facing in a direction opposite to the first plate, and a side member surrounding a space between the first plate and the second plate.

[0065] Wireless communication module 220 (eg, Figure 1The wireless communication module 192) can be electrically connected to the antenna array 230 and can be configured to form a directional beam through the antenna array. For example, the wireless communication module 220 can be a millimeter (mm) wave device and can be configured to send and / or receive frequencies from 3 GHz to 100 GHz. The antenna array 230 can be arranged inside the housing and / or on a portion of the housing. The antenna array 230 can send a sequence of at least one directional beam in the second direction through the selected antenna array 230.

[0066] Display 240 (eg, Figure 1 The display device 160 of the housing may visually provide information (eg, an image captured by the camera module) to the outside of the electronic device 101. The display 240 may be exposed through a first portion of the first plate of the housing.

[0067] Memory 210 (eg, Figure 1 The memory 130 of the processor 200 may store various data used by the processor 200. The memory 210 may store executable instructions (e.g., Figure 1 140) to drive a learning engine for identifying reflected waves received by the wireless communication module 220.

[0068] According to some embodiments, the memory 210 may store instructions that cause the processor 200 to perform operations of configuring a beam width and authenticating an object. The instructions stored in the memory 210 may cause the processor 200 to send a first directional beam sequence having a first beam width so as to scan a first area outside the housing having a first area through the antenna array 230. The instructions stored in the memory 210 may cause the processor 200 to receive a first reflected wave sequence generated by reflection of the first directional beam sequence from an object through the antenna array 230, and send a second directional beam sequence having a second beam width smaller than the first beam width so as to scan a second area included in the first area and having a second area smaller than the first area through the antenna array 230 based on at least a portion of the received first reflected wave sequence. The instructions stored in the memory 210 may cause the processor 200 to receive a second reflected wave sequence generated by reflection of the second directional beam sequence from an object through the antenna array 230, and authenticate the object based on at least a portion of the second reflected beam sequence.

[0069] According to some embodiments, the memory 210 may store at least one template generated by learning. For example, the template may include at least one of a template for recognizing the presence of an object, a template for determining the activity of an object (the possibility that the object is part of a human body), or a template for authenticating an object. According to some embodiments, the memory 210 may store information about a location for presence, activity, and / or object authentication in an image of a recognized object.

[0070] The processor 200 (eg, Figure 1 The processor 120 of the memory 210 may be located within the housing and may be operably connected to the wireless communication module 220, the antenna array 230, and the display 240. Through the instructions stored in the memory 210, the processor 200 may send a first directional beam (hereinafter referred to as the first beam) sequence having a first beam width for determining the distance and range of an object located in the scanning area, determine and send a second directional beam (hereinafter referred to as the second beam) sequence having a second beam width according to the distance and range determined based on the reflected wave of the first beam, acquire object information (3D depth information) based on the reflected wave of the second beam, and match the acquired object information with the template of the memory 210 to authenticate the object.

[0071] According to some embodiments, the electronic device 101 may determine a range for estimating the depth information of a subject, which is an object to be authenticated. The electronic device 101 may authenticate an object using a depth sensor capable of estimating the depth information of an object. The depth sensor may be a device (e.g., a millimeter wave device) including an antenna array 230 and a wireless communication module 220. For example, the millimeter wave device may send a wireless signal to the object for estimating the depth information corresponding to each pixel acquired by the image sensor of the camera, and receive a signal reflected from the object. The millimeter wave device may estimate the distance between the electronic device 101 and the object based on the time-of-flight (TOF) value of the beam. The electronic device 101 may calculate the range of the object so as to limit the range of beamforming as much as possible, and control the direction of the beam according to the distance between the object and the electronic device 101. The electronic device 101 may send a first beam for calculating the distance to the object and the range of the object in the first direction where the object is located, determine a second beam for accurately scanning the object based on the range and distance determined according to the first beam, send the determined second beam in the second direction where the object is located, and receive a reflection signal including object information according to the reflection of the second beam.

[0072] Figure 3A and Figure 3B An electronic device (eg, Figure 1 An electronic device 101) determines a method for a first beam and a second beam.

[0073] According to some embodiments, the electronic device 101 may send a beam to a scanning area where an object exists through a millimeter wave device, and receive a signal reflected from an object in the scanning area to extract depth information, and the object may be authenticated based on the extracted depth information. The scanning area may be in a first direction of the electronic device (e.g., the direction in which the camera is pointing), and may be an area including an object. For example, the scanning area may be an area in which an image of an object within the field of view of an image sensor from a camera may be acquired. In some embodiments, the first beam may be used to determine the distance of an object with a higher granularity, while the second beam may determine the distance of a feature on an object with a finer granularity and a higher resolution (such as by having a narrower beam). The electronic device 101 may determine the distance to an object and the range of the object through a first beam, form a second beam based on the distance and range determined using a reflection signal of the first beam (first reflection wave) reflected from the object, send a second beam in the direction of the object, and acquire object information (3D depth information) based on a reflection signal of the second beam (second reflection wave) reflected from the object.

[0074] refer to Figure 3A According to some embodiments, the electronic device 101 may divide the scanning area 300 into a plurality of first areas (m*n areas) having a first area (size), and generate a first beam 340 having a first beam width so as to scan the m*n divided first areas. In some embodiments, the scanning area 300 may include a rectangular area located at a certain distance from the electronic device and substantially (or within 3 degrees) parallel to the display. For example, the distance from the display may be 1 foot / 30 cm (the average distance between the user's face and the display when using the electronic device).

[0075] Figure 3A An example of segmenting the scan region 300 into 16 (m=4, n=4) first regions 311 to 326 is shown. For example, the scan region 300 may include 16 first regions 311 to 326 having the same size (area). The object 330 may be located inside the scan region 300 (being located inside may include “splice” the object from the scan region 300). Figure 3A An example is shown in which the object 330 is located at the center of the scanning area 300. For example, the scanning area 300 may include a plurality of first areas 311 to 315, 318, 319, and 322 to 326 in which the object 330 does not exist, and at least one first area which is an adjacent area formed by areas 316, 317, 320, and 321 in which the object 330 exists.

[0076] The electronic device 101 according to some embodiments may transmit the first beam 340 to a set position (e.g., a center position) of the first area corresponding to each of the first beams 340 by controlling the azimuth and elevation of one or more first beams 340. The electronic device 101 may transmit the first beam 340 to the first areas 311 to 326 of the scanning area 300 sequentially or based on a set order by controlling the azimuth and elevation of the first beam 340. According to an embodiment, the size of the first beam 340 transmitted to the first areas 311 to 326 may be changed so that the first beam 340 may be applied to the entire first area. According to an embodiment, the size of the first beam 340 transmitted to the first areas 311 to 326 may be changed so that the first beam 340 may be applied to a portion of the corresponding first area. When the size of the first beam 340 is changed so that the first beam 340 may be applied to a portion of the corresponding first area, the first beam 340 may be transmitted to the center position or the set position of the corresponding first area.

[0077] The electronic device 101 according to some embodiments may identify or detect the object 330 by receiving a reflected wave signal of the first beam 340 reflected from the first areas 311 to 326 of the scanning area 300. For example, the first beam 340 may be transmitted to one of the first areas 311 in which the object 330 does not exist, and may be transmitted to another one of the first areas 316 in which the object 330 exists. The electronic device 101 may not receive the reflected wave signal of the first beam 340 reflected from the first area 311, or may receive a reflected wave that does not have a set value. The electronic device 101 may receive the reflected wave signal of the first beam 340 reflected from the first area 316, and identify the presence of the object 330 based on the received reflected wave signal.

[0078] The electronic device 101 can identify the presence of an object in the first area based on the first reflected wave signal reflected from the object 330 by the antenna array 230. For example, since the object does not exist, the reflected waves reflected from the first areas 311 to 315, 318, 319 and 322 to 326 are not received. The reflected waves from the first areas 316, 317, 320 and 321 may have different time of flight (TOF) values. For example, the first reflected wave of the first beam 340 reflected from the first area 311 may not have a set TOF value, but the first reflected wave of the first beam 340 reflected from the first area 316 may meet the set TOF value. The electronic device 101 can estimate the first area (e.g., area 316, 317, 320 or 321) that meets the set TOF value as an area where the object 330 exists based on the first beam 340 and the first reflected wave.

[0079] refer to Figure 3B, the electronic device 101 may scan a first area (eg, Figure 3A The precise scanning operation is performed on the first regions 316, 317, 320, and 321 of the object region 310, but not on the first regions not included in the object region 310 (eg, Figure 3A The electronic device 101 may perform a precise scanning operation on the first areas 311 to 315, 318, 319, and 322 to 326 of the present object 300. The electronic device 101 may divide the first areas 316, 317, 320, and 321 where the object 300 is present into a plurality of second areas (M*N areas) having a second area smaller than the first area, and generate a second beam 370 having a second beam width smaller than the first beam width, so as to precisely scan each of the M*N second areas. Figure 3B An example is shown in which a first region in which an object exists is divided into 16 (M=4, N=4) second regions 351 to 366. For example, in order to accurately scan the first regions 316, 317, 320, and 321 in which the object 330 exists, the electronic device 101 may divide each of the first regions into second regions 351 to 366 having a size smaller than each of the first regions.

[0080] The electronic device 101 may transmit the second beam to the second areas 351 to 366 sequentially or based on a set order by controlling the azimuth and elevation of the second beam. A beam width of the second beam may have a smaller range than a beam width of the first beam. Figure 3B An example is shown in which the second beam 370 is applied to the second area 351. According to an embodiment, the size of the second beam transmitted to the second areas 351 to 366 may be changed so that the second beam can be applied to the entire area of ​​the second area. According to an embodiment, the size of the second beam transmitted to the second areas 351 to 366 may be changed so that the second beam can be applied to a portion of the second area (e.g., area 351). When the size of the second beam 370 is changed so as to be able to be applied to a portion of the second area (e.g., area 351), the second beam 370 may be transmitted to a central position or a set position of the second area (e.g., area 351).

[0081] In some embodiments, the distance of the object from the device in each of the second regions 351 to 366 within the first regions 316, 317, 320, and 321. The set of distances can be used to determine what the object 330 is, such as a user's face.

[0082] The electronic device 101 according to some embodiments may authenticate the object 330 by receiving a reflected wave signal of the second beam 370 reflected from the second area 351 to 366 of the first area (eg, the first area 316 , 317 , 320 , or 321 ) in which the object 330 exists.

[0083] According to certain embodiments, the first beam may use a beam having a relatively wider beam width than the second beam, and the second beam may acquire detailed object information (high-resolution object information) using a beam having a narrow beam width. According to an embodiment, in order to reduce the time taken to estimate the range of the object, the electronic device 101 may first estimate the range of the object by sending the first beam having a wider beam width to the first areas 311 to 326 of the scanning area 300. The electronic device 101 may estimate the first areas 316, 317, 320, and 321 in which the object exists based on the first reflected wave. In order to acquire high-resolution information in the first areas 316, 317, 320, and 321 in which the presence of the object is estimated, the electronic device 101 may estimate the range of the object by sending the second beam having a narrower beam width to the first areas 311 to 326 corresponding to the scanning area 300. Figure 3B The electronic device 101 receives the second reflected wave from the second regions 351 to 366 of the first region (eg, region 317) shown in FIG. 1. The electronic device 101 may authenticate the object based on the second reflected high-resolution wave.

[0084] According to some embodiments, the electronic device 101 may transmit the second beam to only the selected first area (eg, Figure 3B The method can reduce the beam processing time by precisely manufacturing the beam for the area where the object exists (at least one of the first areas 316, 317, 320) and ensure reflection of a high-resolution signal.

[0085] FIG. 4A to FIG. 4E An electronic device (eg, Figure 1 A method for determining a beam using an electronic device 101).

[0086] Figure 4A An example is shown in which the electronic device 101 determines the distance to an object and the range of the object through the first beam. Figure 4A , the electronic device 101 may determine a scanning area 300 for identifying the object 330. For example, the scanning area 300 may be a viewing angle of a camera. The electronic device 101 may separate the scanning area 300 into a first area having a first area (eg, Figure 3AThe electronic device 101 detects the position of the object in the scanning area 300 based on the second beam having a narrower beam width for authenticating the object, which may take a longer time. When the position of the object 330 is unknown, the electronic device 101 may divide the scanning area 300 into larger areas to identify the position of the object 330, and may accurately scan the area 310 in which the position of the object 330 is identified in order to authenticate the object. For example, in order to quickly detect an object, the electronic device 101 may divide the scanning area 300 into predetermined areas (e.g., the first area (e.g., Figure 3A The electronic device 101 estimates the first area 311 to 326 of the present invention, transmits a beam having a wider beam width (e.g., the first beam 340) to the first area through beamforming, and estimates the area 310 in which the object 330 exists based on the signal reflected from the first area. The electronic device 101 can authenticate the object 330 by accurately scanning the area 310 in which the object exists.

[0087] Figure 4A FIG. 2 shows an example in which the scanning area 300 is divided into 16 first areas. Figure 4A As shown, the user face corresponding to the object 330 may be located in a portion of the scanning area 300. For example, the scanning area 300 may be the viewing angle of the camera (an area of ​​a*b). Here, a may be the horizontal size of the screen, and b may be the vertical size of the screen. The electronic device 101 may send the first beam 340 to the first area (e.g., Figure 3A and identifying the object location area 310 (eg, the first area 311 to 326) in which the object 330 exists by the signal reflected from the first area 311 to 326. Figure 3A and Figure 3B The object location area 310 may include at least one first area (e.g., Figure 3A and Figure 3B According to an embodiment, the electronic device 101 may send the first beam 340 to the 16 first areas 311 to 326 according to the set order, and receive the reflected wave of the first beam 340 reflected from the 16 first areas 311 to 326. The electronic device 101 may estimate the object position area 310 based on the reflected wave information (e.g., TOF, phase, and amplitude). For example, the reflected wave reception time (e.g., TOF) of the object position area 310 may be shorter than the reflected wave reception time of the area in which the object does not exist. The object position area 310 may be estimated as at least one first area value.

[0088] According to some embodiments, the electronic device 101 may quickly find the range of the target by controlling the size of the first beam 340 and transmitting the controlled first beam to a larger area. Figure 4A The transmission of the beam to a total of 16 parts is shown, and the beam can be changed to be larger and sent to 8 parts generated by grouping the 16 parts by 2. This operation can be repeated several times, and the number of times is unlimited. When the transmission is performed according to IEEE 802.11ad, the electronic device 101 can send a first beam 340 at a time, thereby repeating the transmission a total of 16 times. When the number of antenna array groups is 2 or more, the electronic device 101 can simultaneously calculate the reflected signal by dividing the area into the first area. According to an embodiment, the electronic device 101 can use a transmission band of several GHz or higher (e.g., 3.1GHz to 10.6GHz) and send the beam by providing an ultra-wideband (UWB) with a speed of 100Mbps or faster. According to some embodiments, the electronic device 101 can use another protocol that can support beamforming other than IEEE 802.11ad or UWB to perform object authentication and receive a reflected wave generated by the reflection of the beam from the object.

[0089] Figure 4B shows the relationship between aspect ratio and viewing angle, and Figure 4C and Figure 4D A method of determining the azimuth and elevation of a particular point based on aspect ratio and viewing angle is shown.

[0090] According to some embodiments, in order to transmit a beam to a segmented area (e.g., the first area or the second area) through a millimeter wave device, the electronic device 101 may control the angle of the beam. The electronic device 101 may control the angle of the beam using the aspect ratio and viewing angle of the camera. Figure 4B In the figure, a and b may indicate the ratio of the width and length of the screen, c may indicate the diagonal length of the screen, and α may be the viewing angle of the camera. The viewing angle of the camera may be an angle at a diagonal, and the electronic device 101 may calculate the camera angle in the direction of line a and the camera angle in the direction of line b based on the viewing angle. The distance to the object cannot be known initially, and therefore the viewing angle may be calculated based on the maximum viewing angle of the camera.

[0091] When an object is mapped at a distance x from the electronic device 101 with a viewing angle α, the center point of the object may be (x, 0, 0), as shown in FIG. Figure 4C As shown. x can be expressed as equation (1), and the azimuth angle β and elevation angle γ of a specific point (x, y, z) can be calculated as equation (2).

[0092]

[0093] According to some embodiments, Figure 4D As shown, the electronic device 101 can calculate the azimuth angle β (horizontal angle) and the elevation angle γ (vertical angle) corresponding to the specific point (y, z) relative to the camera aspect ratios a and b and the camera viewing angle α as shown in equation (2). When the area for scanning the object is divided into a specific number of first areas and the point (y, z) at which the first beam is transmitted to each area in the first area is calculated, the electronic device 101 calculates the same number of beam angles β and γ as the number of areas, and transmits the beam in the direction according to the part. For example, when Figure 4A When the scan area is divided into 16 parts as shown, the first area can be divided as shown in [Table 1].

[0094] [Table 1]

[0095] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

[0096] The electronic device 101 can perform beamforming for each of the first areas corresponding to the 16 parts by calculating the angles β and γ of the beam for the first area as shown in equation (2). For example, when it is assumed that the aspect ratio is 16:9 and the viewing angle is 90 degrees, the range of the transmission angle of the beam for the first area corresponding to the first part can be calculated. The range of the first part calculated by Cartesian coordinates can be (-8, 4.5) to (-4, 2.25), and the azimuth and elevation angles can be calculated as (-41°~-23°, 13°~26°) by equation (2). According to certain embodiments, when the distance between the camera and antenna array of the electronic device 101 and the millimeter wave device is different, the calculation can be performed after moving the distance corresponding to the difference from (0,0). For example, when the beam transmission position is (y1, z1) (when the camera position is assumed to be (0, 0) and the RF position is (y1, z1) in order to compensate for the difference between the RF position and the camera position), the electronic device 101 can calculate the azimuth angle β and the elevation angle γ as shown in the following equation (3).

[0097]

[0098] According to some embodiments, the electronic device 101 may control and send the first beam width based on the object type. For example, the electronic device 101 may identify the object type based on information input from the user or the camera. For example, when the object type is a landscape, an elongated first beam may be sent. When the object type is facial authentication, the electronic device may predict the distance to the face of a typical user and control the width of the first beam sent to each first area. When the part of the face is measured by the above method, the electronic device 101 may estimate the distance to the face based on the time value of the area where the reflected signal is received. The electronic device 101 may form a first beam to be sent far away, and make the width of the first beam inversely proportional to the distance according to the object type. For example, when the out-of-focus function of the electronic device 101 is activated when capturing a picture or video, the electronic device 101 may form a first beam to be sent over a long distance, and determine the parameter value related to out-of-focus based on the TOF value of the received reflected signal of the first beam. For example, when capturing a picture or a video, when the electronic device 101 receives input values ​​having different resolutions for a facial area and a landscape area from the camera, the electronic device 101 can apply different beams to the facial area and the landscape area, and can measure the information requested by the camera (e.g., TOF of each of the segmented areas) and transmit the results to another module.

[0099] Figure 4E An example is shown in which the electronic device 101 acquires object information through the second beam according to some embodiments.

[0100] refer to Figure 4E According to some embodiments, the electronic device 101 divides each of the first areas in the area 310 (four first areas) in which the object 330 exists into 16 second areas. The electronic device 101 can further divide the area in which the object 330 exists and send the second beam 370 to each of the divided second areas 351 to 366 to obtain a more accurate beam reflection signal. The azimuth and elevation angles of the second beam 370 can be calculated using equation (2).

[0101] According to certain embodiments, the electronic device 101 may acquire object information based on the reflected wave of the second beam 370. The reflected signal of the second beam 370 may include object information (e.g., depth information). The reflected signal of the second beam 370 may be a reflected signal from an object to which the beam is sent in a specific area, rather than information about pixels. For example, the bandwidth in the beamforming scheme of IEEE 802.11ad 60GHz may be approximately 3 to 3.56GHz, and the resolution that can be measured for the reflection of the second beam 370 may be approximately 100μm. Therefore, a reflected signal for a very short distance of approximately 100μm may be sent to a specific area and measured. For example, when the activity of the user's face is to be determined or when security is required, the electronic device 101 may increase the resolution. When only the object needs to be identified, the electronic device 101 may reduce the resolution.

[0102] According to an embodiment, the electronic device 101 may transmit a beam and receive a reflected wave through ultra-wideband (UWB) to acquire object information. UWB uses a transmission frequency band higher than or equal to several GHz (eg, 3.1 GHz to 10.6 GHz) and provides a speed faster than or equal to 100 Mbps.

[0103] Figure 5 is a diagram showing an electronic device (eg, Figure 1 Flowchart 500 of the operation of determining the acquired object information as a template by machine learning of the electronic device 101. In some embodiments, flowchart 500 can be executed when the user initializes the electronic device, such as during the first or early use.

[0104] According to some embodiments, the electronic device 101 may estimate an area in which an object exists, learn a beam including object information in the object estimation area through machine learning, and store the learning result as a template in a memory (e.g., Figure 1 The electronic device 101 may generate and store a template (comparison data) for object authentication. In order to register a template, the electronic device 101 may place an object (e.g., a subject) at a predetermined distance (e.g., 20 cm, 25 cm, or 30 cm), collect data, and perform machine learning based on the collected data. For example, when the object is a person (e.g., a face), the user may perform an authentication operation at a location spaced a predetermined distance from the electronic device 101. The electronic device 101 may collect user facial information according to each distance, learn the data collected at the corresponding distance based on a machine learning scheme, and then register the result as a template. Thereafter, for example, when the user is at a distance of 21 cm / 8.4 in, the electronic device 101 may also identify whether to perform authentication based on machine learning, and update the corresponding information as a template (comparison data).

[0105] Figure 5 An operation is shown in which the electronic device 101 recognizes an object spaced a certain distance away therefrom and registers the object as a template.

[0106] According to some embodiments, in operation 511, the electronic device 101 may transmit the second beam to a second area (an area where the object is predicted to exist) of the first area. For example, the second area may be a portion of the first area, and the first area is an area where the object can be captured by the camera module (e.g., Figure 1 The electronic device 101 may receive a second reflected wave reflected from an object in the second area and obtain object information based on the received second reflected wave. In operation 515, the electronic device 101 may extract and compare feature points (feature extraction and matching). The electronic device 101 may learn and update the compared feature points in operation 517, and in operation 519, store the updated feature points (feature vectors) as a template.

[0107] According to certain embodiments, the electronic device 101 may determine a template for authenticating an object by learning feature points based on a received second reflected wave generated by reflection of a second beam from the object. For example, the electronic device 101 may generate a template for authenticating an object based on all or part of TOF, phase, and amplitude based on the second reflected wave. The electronic device 101 may learn a template for identifying the presence of an object, a template for determining the activity of an object, and / or a template for authenticating an object. For example, a template for identifying the presence and a template for determining the activity may be generated based on the first reflected wave of the first beam, and a template for authenticating an object may be generated based on the second reflected wave of the second beam.

[0108] According to some embodiments, the electronic device 101 or the external server may generate a template based on the first reflected wave and the second reflected wave through deep learning (or a deep neural network or an artificial neural network). For example, the deep learning method may use a convolutional neural network (CNN) scheme. When learning is performed through the CNN scheme, learning is performed by inputting the object information to be learned into the CNN, and features based on the learning results may be extracted and stored as templates. For example, when the object is the face of the user, information about a specific part that may be a facial feature point (e.g., at least one of the eyes, nose, and mouth actually obtained from the user's face), and information about each specific part received through the wireless communication module (e.g., phase, TOF, and amplitude values ​​obtained from the reflected wave of the 60GHz beam) may be input to the CNN, and the output value corresponding thereto may be determined to be true. In addition, the same parameters of a typical user may be input, and the output value corresponding thereto may be determined to be false. In the learning method using CNN, the above-mentioned calculation setting number of times may be repeated, and the error generated during the calculation process may be updated (through a back propagation scheme), and a model (template) for making the user parameters true may be generated and stored in a memory. The template may include a presence template, an activity template, and / or a face authentication template. The presence template and liveness template may be used for all users, and the face authentication template may be a unique template generated by learning each user image and reflection wave.

[0109] The approach can be trained via transfer learning, where the model is pre-configured with parameters for typical users and fine-tuned with parameter values ​​for new users.

[0110] The electronic device 101 can perform facial authentication by comparing the information of the facial reflected wave received during the operation of the facial authentication application with the corresponding information stored in the memory. When the actual authentication is performed using the training model, the electronic device 101 can perform an authentication operation with parameters based on the reflected wave received as input to the CNN in real time. When the output of the CNN is true, the electronic device 101 can authenticate the user as a legitimate user, and when the output is false, it is determined that the user is not a legitimate user.

[0111] Figure 6 An electronic device (e.g., Figure 1 Configuration of electronic device 101).

[0112] refer to Figure 6 , the electronic device 101 may include a first beam determination module 610 , a first beam processing module 620 , a second beam determination module 630 , a second beam processing module 640 and an object authentication module 650 . Figure 6 The configuration in can be the configuration of the processor (for example, Figure 1 The processor 120 or Figure 2 processor 200).

[0113] The first beam determination module 610 may determine a first beam for calculating a distance between an object and the electronic device 101 and / or a range of the object. The first beam determination module 610 may divide a scanning area into regions having a set area and determine a first beam to be sent to the divided region in order to quickly detect an object. The scanning area may be an area including an object to be authenticated. For example, the scanning area may be an area in which a camera module (e.g., Figure 1 The first beam determination module 610 can divide the scanning area into a plurality of first areas having a first area, and calculate the directivity of the first beam to be sent to the divided first areas. The directivity of the first beam can be calculated using equation (2) or equation (3) based on the coordinate information of the corresponding first area.

[0114] The first beam processing module 620 may perform control to transmit the beam to the wireless communication module (eg, Figure 2 The wireless communication module 220 and the antenna array (eg, Figure 2 The antenna array 230 of the wireless communication module 220 can send a sequence of first beams to the corresponding first area. For example, the sequence of first beams can be beamformed to be sent to the center position of the corresponding first area. The first beam processing module 620 can receive a first reflected wave sequence generated by the reflection of the first beam sequence from the first area through the wireless communication module 220. The first beam processing module 620 can identify the object information of the first area based on the received first reflected wave sequence. The first reflected wave may include information about the TOF, phase and amplitude of the corresponding first area. The first beam processing module 620 can identify whether the object is located in the corresponding first area based on the first reflected wave information. For example, when the first reflected wave is received within the reference time set by calculating the TOF from the first reflected wave, the first beam processing module 620 can determine that the object is located in the corresponding first area. The first beam processing module 220 can determine whether the object is located in the corresponding first area by scanning the first area of ​​the scanning area.

[0115] The second beam determination module 630 may determine a sequence of second beams to be sent to a first area determined as an area in which an object exists. The second beam determination module 630 may divide the first area in which the object exists into a second area having a second area smaller than the first area, and determine the directivity of the second beam to be sent to the divided second area. The second beam may have a beam width narrower than the beam width of the first beam. The directivity of the second beam may be calculated using equation (2) or equation (3) based on the coordinate information of the corresponding second area. According to an embodiment, the second beam determination module 630 may divide at least one first area including an object and having a first area into a second area having a second area. For example, when the first area is divided into 16 second areas and the object is determined to be located in four first areas by the first beam, the second beam determination module 640 may determine 16*4 second beams.

[0116] The second beam processing module 640 may perform control to output the determined sequence of the second beam through the wireless communication module 220. The wireless communication module 220 and the antenna array 230 may send the sequence of the second beam to the corresponding second area. For example, the sequence of the second beam may be beamformed to be sent to the center position of the corresponding second area. The second beam processing module 640 may receive the second reflected wave sequence reflected from the corresponding second area through the wireless communication module 220. The second beam processing module 640 may identify the object information of the second area based on the received second reflected wave sequence. The second reflected wave may include information about the TOF, phase, and amplitude of the corresponding second area. The second beam processing module 620 may obtain information about the object located in the corresponding second area based on the second reflected wave information.

[0117] The object authentication module 650 may authenticate the object by performing machine learning on the object information acquired by the second beam processing module 640. The object authentication module 650 may authenticate the object based on the information stored in the memory (e.g., Figure 2 The object is authenticated by machine learning that matches the object authentication template in the memory 210) with the acquired object information.

[0118] According to some embodiments, an electronic device (eg, Figure 1The electronic device 101) may include a housing, the housing including a first plate facing a first direction, a second plate facing a direction opposite to the direction facing the first plate, and a side member surrounding the space between the first plate and the second plate, a display 240 exposed through a first portion of the first plate, an antenna array 230 arranged in the housing and / or on a portion of the housing, a wireless communication module 220 electrically connected to the antenna array 230 and configured to form a directional beam through the antenna array 230; a processor 200 is located in the housing and is operably connected to the display 240 and the wireless communication module 240; and a memory 210 is operably connected to the processor 200. The memory 210 may store instructions which, when executed, cause the processor 200 to send a first directional beam sequence having a first beam width through the antenna array 230 to scan a first region having a first area outside the housing, receive a first reflected wave sequence generated by reflection of the first directional beam sequence from an object through the antenna array 230, send a second directional beam sequence having a second beam width narrower than the first beam width through the antenna array 230 based on at least a portion of the received first reflected wave sequence to scan a second region, the second region being included in the first region and having a second area smaller than the first area, receive a second reflected wave sequence generated by reflection of the second directional beam sequence from the object through the antenna array 230, and authenticate the object based on at least a portion of the second reflected wave sequence.

[0119] According to some embodiments, the instructions may cause the processor 200 to activate an antenna selected from the antenna array 230 in order to determine the first beamwidth and / or the second beamwidth.

[0120] According to some embodiments, the instructions may cause the processor 200 to determine a relative distance and / or direction between the object and the electronic device 101 based on the sequence of first reflected waves.

[0121] According to some embodiments, the electronic device 101 may further include an image sensor exposed through a portion of the first board and arranged to face the first direction, and the instructions may cause the processor to include the first area in a viewing angle of the image sensor.

[0122] According to some embodiments, the object may include the user's face.

[0123] In some embodiments, templates for the user's face may be determined for various distances. Each template may then be reduced to only a subset (shortened template) of second regions 351 ... 366 having first regions (e.g., 316, 317, 320, and 321) of the object. Specific second regions 351 ... 366 may be used based on past frequency of occurrence. If the reflected wave pattern at a specific second region has a sufficiently high correlation with the shortened template, the object may be considered authenticated, thereby bypassing a more extensive comparison using all second regions.

[0124] Figure 7 is a diagram showing an electronic device (eg, Figure 1 Flow chart of object authentication operation 700 of electronic device 101).

[0125] refer to Figure 7 , before the object authentication operation, the electronic device 101 may guide the object (e.g., the user) to be positioned within the transmission range of the first beam 340. For example, when the object 330 is authenticated, the object 330 may not be located within the scanning area 300 of the electronic device 101. Before the object authentication operation, the electronic device 101 may guide the object 330 to be positioned within the scanning area 300. For example, the electronic device 101 may display a guidance message in the form of an audio and / or video message, instructing the user to look at the camera while being spaced a predetermined distance from the electronic device 101 in order to authenticate the object. In another example, when the user executes an object authentication related application, the electronic device 101 may display guidance on the display 240 instructing the user to position himself within the field of view of the camera module 180.

[0126] In operation 711, when a request is made to perform an object authentication procedure, the electronic device 101 according to some embodiments may determine a first beam 340 for identifying the distance between the electronic device 101 and the object 330 and the range of the object. The object 330 may be located in a portion of the scanning area 300. The electronic device 101 may divide the scanning area into a plurality of first areas 311...326 having a first size, and determine a first beam to be sent to the divided first areas in order to quickly detect the object. The first beam 340 may include a beam having a wide beam width, and may be formed to be sent to the center of the first area. According to an embodiment, the electronic device 101 may determine information about the first area to which the first beam is sent based on the viewing angle of a camera located in the front surface of the electronic device 101. In operation 713, the electronic device 101 may communicate with the wireless communication module ( Figure 2 The wireless communication module 220) and the antenna array (e.g., Figure 2The electronic device 101 may transmit the first beam 340 to the center position of the first area through the antenna array 230 and the wireless communication module 220 when performing object authentication of the user. For example, when performing object authentication of the user, the electronic device 101 may transmit the first beam 340 to the center position of the first area through the wireless communication module 220 and the antenna array 230. The transmitted first beam 340 may be reflected by an object located in the first area of ​​the scanning area 300, and no reflected wave is generated in the first area where no object exists. In operation 715, the electronic device 101 may receive the first reflected wave reflected from the first area through the antenna array 230 and the wireless communication module 220.

[0127] In operation 717, the electronic device 101 may identify the object information in the received first reflected wave and determine the first area 316, 317, 320, 321 in which the object exists. In the first area 316, 317, 320, 321 in which the object exists, it may take a short time to send, reflect and receive the first beam. The electronic device 101 may identify the first area 316, 317, 320, 321 in which the object exists and the first area 311-315, 318, 319, 322-326 in which the object does not exist based on the first reflected wave. The electronic device 101 may divide the first area in which the object exists into a plurality of second areas 351 ... 366 having a second size and determine a second beam to be sent to the divided second area. The second beam may have a beam width narrower than the beam width of the first beam.

[0128] In operation 719, the electronic device 101 may transmit the corresponding second beam to the second area of ​​the first area where the object exists through the wireless communication module 220 and the antenna array 230. In operation 721, the electronic device may receive the second reflected wave reflected from the second area through the antenna array 230 and the wireless communication module 220. The electronic device 101 may obtain the object information from the received second reflected wave. In operation 723, the electronic device 101 may compare the object information obtained through the second reflected wave with the object information stored in the memory (e.g., Figure 2 The object authentication operation is performed by matching the template in the memory 210).

[0129] According to certain embodiments, the electronic device 101 can reduce the time used to estimate the range of an object (for example, the location where the object exists) by first estimating the range of the object through a first beam having a wide beam width, and can acquire high-resolution object information in an area estimated to be the location of the object through a second beam having a narrow beam width.

[0130] Figure 8 An electronic device (eg, Figure 1Method 800 of controlling a beam for object authentication using an electronic device 101).

[0131] refer to Figure 8 In operation 811, the electronic device 101 may divide the scanning area into a specific number of parts. The scanning area 300 is an area in which the object 330 is scanned, and may include an area in which the object exists (e.g., first areas 316, 317, 320, 321) and an area in which the object does not exist (e.g., first areas 311-315, 318, 319, and 322-326). For example, the scanning area may be a region that can be based on a camera module (e.g., Figure 1 The electronic device 101 may divide the scanning area into a plurality of first areas 311 ... 326 having a first size. In operation 813, the electronic device 101 may form a first beam having a beam width suitable for the first areas 311 ... 326. In operation 815, the electronic device 101 may form a first beam so that the first beam may be sent to a set position of the first areas 311 ... 326. For example, the electronic device 101 may calculate the azimuth angle β and the elevation angle γ of the center coordinates of the first area using equation (2) or equation (3). In operation 817, the electronic device 101 may communicate with the wireless communication module (e.g., Figure 2 The wireless communication module 220) and the antenna array (e.g., Figure 2 The antenna array 230 of the embodiment of the present invention performs beamforming of the first beam with the calculated azimuth and elevation angles. The sequence of the beamformed first beams can be sent from the antenna array 230 to the center position of the corresponding first area.

[0132] In operation 819, the electronic device 101 may receive a sequence of reflected waves (first reflected waves) of the first beam reflected from the first area. For example, the sequence of the first reflected waves may be a signal reflected from the center coordinate position of the first area of ​​the scanning area. In operation 821, the electronic device 101 may estimate the distance to the corresponding first areas 316, 317, 320, and 321 and the range of the object 330 based on the received first reflected waves. The first reflected wave reflected from the first area where the object 330 exists may be received by the electronic device 101 within a specific time, and the first reflected wave reflected from the first area where no object exists (e.g., 311-315, 318, 319, and 322-326) may not be received by the electronic device 101 within a specific time. The electronic device 101 may estimate the distance to the object 330 based on the first reflected wave reflected from the first area (e.g., by calculating the TOF of the first reflected wave). In operation 823, the electronic device 101 may determine the first regions 316, 317, 320, and 321 in which the object 330 exists based on the estimated distance. For example, the electronic device 101 may recognize that the first region located within a specific range based on the first reflected wave is the region in which the object exists.

[0133] In operation 825, the electronic device 101 may divide the first area identified as the area in which the object exists into second areas 351 ... 366 having a second area. The electronic device 101 may divide the corresponding first area into smaller parts so as to accurately obtain the object information of the first area in which the object exists. The resolution of the acquired object information may be high depending on the number of second areas 351 ... 366 into which the first area is divided. In operation 827, the electronic device 101 may form a second beam 370 having a beam width suitable for the second area 351 ... 366. The second beam 370 may have a second beam width, and the second beam width may be narrower than the first beam width. In operation 829, the electronic device 101 may calculate the azimuth and elevation of the corresponding second area 351 ... 366. For example, the electronic device 101 may calculate the azimuth β and elevation γ of the center coordinates or the coordinates of the set position of the second area 351 ... 366 using equation (2) or equation (3). In operation 831, the electronic device 101 may perform beamforming of the second beam having the calculated azimuth and elevation angles through the wireless communication module 220 and the antenna array 230. A sequence of beamformed second beams 370 may be transmitted to the setting positions of the corresponding second areas 351...366 through the antenna array 230.

[0134] In operation 833, the electronic device 101 may receive a sequence of reflected waves (second reflected waves) of a second beam reflected from a second area 351 ... 366. For example, the sequence of second reflected waves may be a signal reflected from a central position of a second area 351 ... 366, and the second area 351 ... 366 originates from a segmentation of a first area in which an object exists. In operation 835, the electronic device 101 may acquire object information based on the received second reflected waves. The object information obtained in the second area 351 ... 366 may be information about TOF, phase and / or amplitude. The resolution of the object 330 information may be determined by the number of second areas 351 ... 366 originating from the segmentation of the first area and the sequence of the second beam 370.

[0135] In operation 837, the electronic device 101 can store the acquired object information in a memory (eg, Figure 2 The matching score is calculated by comparing the object information with the learning data (template) in the memory 210 of the electronic device 101. The learning data stored in the memory 210 may be data learned by machine learning and data generated by learning information about the object to be authenticated. The electronic device 101 may acquire object information from the second reflected wave information based on the second beam sequence, and match the acquired object information with the learning data that has been pre-learned and stored in the memory 210 to calculate the matching score. In operation 839, the electronic device 101 may perform an object authentication operation based on the calculated matching score. For example, the electronic device 101 may compare the calculated matching score with a reference value, and may determine that the object authentication is successful (true) when the matching score is greater than the reference value, and determine that the object authentication fails (false) when the matching score is less than the reference value.

[0136] Fig. 9 An electronic device 101 (eg, Figure 1 Method 900 of electronic device 101) determining the activity of object 330 and authenticating object 330.

[0137] According to some embodiments, the electronic device 101 may determine the activity of the first area 316, 317, 320, 321 in which the object 330 exists and authenticate the object 330. In order to identify the distance to the object 330 located in the scanning area and the range of the object 330, in operation 911, the electronic device 101 may divide the scanning area into first areas 311...326 having a first area. In operation 920, the electronic device 101 may send a sequence of first beams 340 having a first beam width to the first areas 311...326, and identify the position of the object 330 in the first areas 311...326 based on the first reflected waves reflected from the first areas. Figure 8Operation 920 is performed in the same way as operations 811 to 823 .

[0138] In order to determine the activity of the object 330 (the possibility that the object is not a photo), in operation 931, the electronic device 101 may perform beamforming to send a beam to one or more first areas of the first areas 311 ... 326 in which the position of the object 330 is identified. For example, the first area 311 ... 326 in which the beamforming is performed may be a first area located at the center of the scanning area. For example, the beam formed in the first area 311 ... 326 may be a beam having a first beam width or a second beam width. In operation 933, the electronic device 101 may receive a reflected wave reflected from the corresponding first area 316, 317, 320, 321. In operation 935, the electronic device 101 may acquire object information based on the received reflected wave. The acquired object information may be a TOF value, a phase value and / or an amplitude value of the reflected wave. In operation 937, the electronic device 101 may determine the activity of the object 330 based on the acquired object information. For example, when the object 330 is the user's face, the electronic device 101 may compare the phase of the reflected wave and / or the TOF pattern with the motion pattern (micro-motion) of the face stored by learning (training) to determine the activity, and may determine the activity by comparing the amplitude reduction pattern according to the frequency of the reflected wave with the pre-stored human skin pattern. If there is no activity of the determined object 330 in the first area in which the object 330 exists, the electronic device 101 may end the object authentication operation.

[0139] If there is activity of the determined object 330 in the first area 311 ... 326 in which the object 330 exists, in operation 940, the electronic device 101 may divide the first area in which the object 330 exists into second areas 351 ... 366, and transmit a second beam having a second beam width to the second areas 351 ... 366. The electronic device 101 may receive reflected waves reflected from the second areas 351 ... 366 and acquire object information. Operation 940 may be performed by Figure 8 Operations 825 to 835 are performed in the same manner.

[0140] In operation 950, the electronic device 101 may perform object authentication based on the acquired object information of the second area 351 . . . 366. Figure 8 Operation 950 is performed in the same manner as operations 837 to 839.

[0141] According to certain embodiments, a method for authenticating an object 330 through an electronic device 101 may include an operation of sending a first directional beam sequence having a first beam width through an antenna array 230 arranged inside and / or on a portion of the shell to scan a first region 311 ... 326 having a first area outside the shell, an operation of receiving a first reflected wave sequence generated by reflection of the first directional beam sequence from the object 330 through the antenna array 230, an operation of sending a second directional beam sequence having a second beam width narrower than the first beam width through the antenna array 230 based on at least a portion of the received first reflected wave sequence to scan a second region 351 ... 366 included in the first region 316, 317, 320, 321 and having a second area smaller than the first area, an operation of receiving a second reflected wave sequence generated by reflection of the second directional beam sequence from the object 330 through the antenna array 230, and an operation of authenticating the object 330 based on at least a portion of the second reflected wave sequence.

[0142] According to some embodiments, the operation of transmitting the first directional beam sequence may include activating a certain number of antennas selected from antenna array 230 to determine the first beam width. The operation of transmitting the second directional beam sequence may include activating a greater number of antennas than the certain number of antennas selected from antenna array 230 to determine the first beam width to determine the second beam width.

[0143] According to certain embodiments, the operation of sending a second directional beam sequence may include an operation of determining a relative distance and / or direction between an object and the electronic device 101 based on a received first reflected wave sequence, an operation of generating a second directional beam sequence to be sent to a second area 351...366 included in the first area based on the determined distance and / or direction, and an operation of sending the generated second directional beam sequence.

[0144] According to some embodiments, the method may further include an operation of including the first area 311 . . . 326 within the viewing angle of the camera 1010 arranged to face the first direction, and guiding the object to be positioned in the first area 311 . . . 326.

[0145] According to some embodiments, the object may include the user's face.

[0146] Fig.10 An electronic device (eg, Figure 1 Configuration of electronic device 101).

[0147] refer to Fig.10 , electronic devices (e.g. Figure 1The electronic device 101) may include a processor 200, a memory 210, a wireless communication module 220, an antenna array 230, a display 240, a camera module 1010 and a sensor module 1020.

[0148] Camera module 1010 (eg, Figure 8 The camera module 180 of the embodiment may include an image sensor (not shown) capable of receiving light in a visible light band reflected from an object 330 (e.g., a face or an iris) and generating pixel data. The camera module 1010 may be exposed through a second portion of the first plate of the housing adjacent to the display 240 and may be arranged to face the first direction.

[0149] Sensor module 1020 (eg, Figure 1 The sensor module 176 of the embodiment may detect the brightness of the surrounding environment where the object 330 is located. For example, the sensor module 1020 may include an illumination sensor.

[0150] Display 240 (eg, Figure 1 The display device 160 of the housing 101 may visually provide information (eg, an image captured by the camera module 1010) to the outside of the electronic device 101. The display 240 may be exposed through a first portion of the first plate of the housing.

[0151] The memory 210, the antenna array 230, the wireless communication module 220 and the display 240 may have Figure 2 The processor 200 (eg, Figure 1 The processor 120 of the electronic device 101 may be located within the housing of the electronic device 101 and may be operably connected to the camera module 1010, the display 240, the wireless communication module 220, and the antenna array 230. The processor 200 may perform object authentication operations according to instructions stored in the memory 210.

[0152] When a request for authenticating an object is made, the processor 200 may recognize the brightness of the surrounding environment in which the object 330 exists through the sensor module 1020. When the ambient brightness becomes low, it may be difficult for the processor 200 to recognize the object 330 from the image acquired by the camera module 1010. According to some embodiments, if the ambient brightness is lower than the set brightness, the processor 200 may recognize the object 330 through the antenna array 230 and the wireless communication module 220. The processor 200 may beamform a first beam having a wide beam width through the wireless communication module 220 and the antenna array 230, and may transmit the first beam to an area in which the object 330 exists, so as to determine the activity of the object 330. When the activity is determined, the processor 200 may beamform a second beam having a narrower beam width than the first beam width to transmit the second beam to an area in which the object 330 exists, receive a reflected wave of the second beam to acquire object information, and match the acquired object information with information learned through machine learning, so as to authenticate the object 330.

[0153] When the ambient brightness is bright enough to identify the object 330, the processor 200 may acquire and receive at least one image through the camera module 1010, and identify the object 330 in the received image. The processor 200 may send a directional beam sequence in at least one second direction set based on the image through the wireless communication module 220 and the antenna array 230, and may receive a reflected wave sequence generated by reflection of the beam sequence from the object 330. The processor 200 may authenticate the object 330 based on the identified object 330 and at least a portion of the reflected wave sequence.

[0154] According to some embodiments, the wireless communication module 220 may be a millimeter wave device. The millimeter wave device may transmit a millimeter wave signal in the direction of the object 330 by a beamforming scheme in the direction corresponding to a specific pixel of the image data, and receive a signal reflected from the object 330 to identify the characteristics of the reflected signal. In order to authenticate the object 330 using the electronic device 101 including the millimeter wave device, the electronic device 101 needs to extract only the reflected object information by selecting a necessary part of the object 330 and transmitting a millimeter wave in the direction of the selected part of the object 330. The electronic device 101 may add the unique signal characteristics of the millimeter wave device to the object authentication operation, thereby further enhancing the security level of the object authentication. The electronic device 101 according to some embodiments may reduce the time spent on generating millimeter wave image data by optimizing the selection of the object 330 portion (e.g., a specific position of the object 330 based on the image data) to be beamformed by the millimeter wave device based on the image data of the RGB camera, and as a result, the total processing time of the object authentication system may be reduced.

[0155] The electronic device 101 may perform an authentication operation for acquiring an image of an object (e.g., a face) and performing a setting function. The user may capture an object by driving the camera module 1010. The processor 200 may acquire an image including an object from the camera module 1010 and identify the object portion in the acquired image. The processor 200 may extract features of a main portion of the object in the image of the identified object. The main portion of the object may be a portion for identifying the presence of the object or the activity of the object or authenticating (identifying) the object. The processor 200 may generate a beam sequence in a direction corresponding to the extracted object portion through the wireless communication module 220 and the antenna array 230 and transmit millimeter waves. The processor 200 may receive a beam sequence reflected from the object through the wireless communication module 220 and the antenna array 230. The processor 200 may learn information about the main portion of the object through deep learning, an artificial neural network, or a deep neural network. The memory 210 may store the learned information about the main portion of the object. When reflection information about a main part of an object is received from the wireless communication module 220, the processor 220 can recognize the output of the deep learning system and determine whether the output is a result of an object corresponding to the user, wherein the deep learning system matches the characteristics of the main part of the object stored in the memory 210 with the characteristics based on the received reflection wave information.

[0156] According to some embodiments, the electronic device 101 may use the machine learning engines for both object image recognition and reflected wave recognition as one machine learning engine.According to an embodiment, the electronic device 101 may use the machine learning engine for object image recognition and the machine learning engine for reflected wave recognition separately.

[0157] According to one embodiment, the processor 200 may identify an object portion in the image data acquired and received by the camera module 1010, and set an authentication identification position for authenticating the object in the image of the identified object. The processor 200 may use the antenna array 230 to send a millimeter wave to the authentication identification position through beamforming, and receive the millimeter wave reflected from the object, so as to authenticate whether the object corresponds to the user's set object. The object authentication operation may be performed based on a deep learning algorithm.

[0158] According to an embodiment, the processor 200 may identify a portion of an object in the image data acquired and received by the camera module 1010, and set an activity determination position for determining the activity of the object and an authentication identification position for authenticating the object in the image of the identified object. For example, the identification position may be a specific portion such as an eye, a nose, or a mouth, or may be a portion that can effectively represent a user's features. In addition, the predetermined position may vary according to information of the image data (e.g., the angle of the face). For example, the activity determination position may be a portion such as an eye, a nose, and a mouth, in which a slight movement of the user may be detected. For example, the authentication identification position may be the entire facial area or the position of the eyes, nose, or mouth that defines the user's face.

[0159] The processor 200 may send millimeter waves to the identified activity determination position through beamforming, and receive millimeter waves reflected from the object to determine the activity of the object. For example, the processor 200 may send millimeter waves to the authentication identification position through beamforming, and receive millimeter waves reflected from the object to authenticate whether the object corresponds to the user's set object. The activity determination and object authentication operations may be performed based on a deep learning algorithm.

[0160] Fig.11 A configuration of an electronic device according to some embodiments is shown.

[0161] refer to Fig.11 , electronic equipment ( Figure 1 The electronic device 101) may include a selection module 1110, a first object authentication module 1120 and a second object authentication module 1130. Figure 1 The electronic device 101 may be a processor (eg, Figure 1 The processor 120 or Fig.10 Configuration of processor 200).

[0162] The selection module 1110 can be configured by a sensor module (e.g., Fig.10 The sensor module 1020 of the object 330) acquires information about brightness near the object 330. The selection module 1110 may identify whether the intensity of light acquired by the sensor module 1020 is outside a specific range, and select the first object authentication module 1120 when the intensity of light is outside the specific range, and select the second authentication module 1130 when the intensity of light has a value within the specific range.

[0163] The first object authentication module 1120 may perform the following operations: authenticating the object 330 by recognizing the distance to the object 330 and the range of the object 330 through beamforming of the first beam, acquiring object information of the area in which the object 330 exists through beamforming of the second beam, and matching the acquired object information with the learned data. The first object authentication module 1120 may have Figure 6 Same configuration as shown.

[0164] The second object authentication module 1130 can perform the following operations: authenticate the object 330 by identifying the image acquired via the camera module 1010, extract the object 330 area and the feature points of the object 330, form a beam to the identified object 330 area through the wireless communication module 220 and the antenna array 230, acquire the activity and object information of the object 330, and match the acquired object information with the learned data.

[0165] According to an embodiment, the electronic device 101 may receive a signal of the intensity of light around the object 330 through the sensor module 1020, and identify whether the intensity of the received light is equal to or lower than a predetermined brightness. When the intensity of the light is equal to or lower than the predetermined brightness, it may be difficult to recognize the image acquired by the camera module 1010. When it is difficult to recognize the image acquired by the camera module 1020, the processor 200 may activate the first object authentication module 1120 to perform an object authentication operation through the wireless communication module 220 and the antenna array 230 (e.g., a millimeter wave device). When the object authentication operation is performed using only the millimeter wave device, the processor 200 may display a message for guiding the object to be positioned within the viewing angle of the camera module 1010 through the display 240.

[0166] The configuration of the first object authentication module 1120 of the electronic device 101 can be Figure 6 When the electronic device 101 performs an object authentication operation through the first object authentication module 1120, the object authentication method can be the same as Figures 7 to 9 The method is the same.

[0167] When it is determined that the intensity of the light is sufficient to recognize the image acquired through the camera module 1020 , the processor 200 may perform an object authentication operation by activating the second object authentication module 1130 .

[0168] Fig.12 An electronic device (eg, Figure 1 Configuration of electronic device 101).

[0169] refer to Fig.12 , electronic devices (e.g. Figure 1The electronic device 101 may include an object recognition module 1210, a position recognition module 1220, an active position setting module 1230, an active determination module 1240, an authentication position setting module 1250, and an object authentication module 1260. Fig.12 The configuration of the electronic device 100 may be Fig.11 Configuration of the second object authentication module 1130.

[0170] The object recognition module 1210 can obtain the information from the camera module (e.g., Figure 1 Camera module 180 or Fig.10 The object recognition module 1210 may recognize an image of the object 330 in the acquired image (face detection). For example, in order to recognize objects of various sizes, the object recognition module 1210 may generate a pyramid image in the acquired image and determine whether a region of a specific size is the object 330 by a classifier (e.g., AdaBoost (adaptive boosting)) while shifting the image by one pixel each time.

[0171] The position recognition module 1220 can recognize the main features (facial features or facial feature points) of the object in the image of the recognized object. According to some embodiments, when the object is the face of the user, the position recognition module 1220 can specify the eyes, nose or mouth and also the facial area. For example, the position recognition module 1220 can detect (feature detection or feature point detection) a position for recognizing the existence of the object in the image of the object (e.g., the center position of the image), a position for determining the activity of the object (e.g., a specific position of the face where the movement of the eyes, nose or mouth can be detected), and a position for authenticating the face (e.g., the entire facial area where facial features can be recognized or multiple positions of the eyes, nose or mouth).

[0172] The active position setting module 1230 may select an object position (e.g., a specific pixel position of an image of an object) for determining the activity (activity detection) of an object, and perform beamforming through the wireless communication module 220 and the antenna array 230 to transmit a beam sequence to the selected object position. The wireless communication module 220 and the antenna array 230 may transmit a beam sequence in a corresponding direction based on the active beamforming information (azimuth) set by the active position setting module 1230, and receive a signal reflected from the subject.

[0173] The activity determination module 1240 may determine the activity of the object based on the signal reflected from the subject received by the antenna array 230. The activity determination module 1240 may calculate the TOF, phase and / or amplitude of the received reflected signal and determine the activity based on the activity template having the calculated information stored in the memory 210.

[0174] The authentication position setting module 1250 can select an object position for authenticating an object (e.g., the entire area of ​​a facial image or a set of feature points including a plurality of feature points) and perform beamforming to send a beam sequence to the selected position. The wireless communication module and the antenna array can send a beam sequence in a corresponding direction based on the beamforming information (azimuth and elevation) for object authentication set by the authentication position setting module 1250 and receive a signal reflected from the object.

[0175] The object authentication module 1260 can authenticate the image of the object based on the received signal reflected from the subject. The object authentication module 1260 can calculate the TOF, phase and / or amplitude of the received reflected signal, and authenticate the user's object in the image based on the object authentication template with the calculation information stored in the memory. The active position setting module 1230 can determine the position for measuring the micro-motion of the object, and the active determination position can be used to measure the distance from the object. For example, the active determination position can be an object area or a portion of an object area. The object authentication module 1260 can perform object authentication based on object information (e.g., TOF of information about the distance to the object) through deep learning information.

[0176] The electronic device 101 according to some embodiments may determine the activity of an object in an acquired image through a millimeter wave device, and when there is the activity of the object, authenticate the image of the object. For example, an object that is part of a person may continuously generate motion at the micron level, even if the object remains stationary. Such motion may continuously vibrate a phase value or a TOF value for the reflected wave of the millimeter wave device. The electronic device 101 may pre-store motion features in a specific pattern in a storage unit (e.g., memory 210) through a deep learning scheme (or a machine learning scheme). In addition, the reflected wave from human skin may have characteristics different from those of an object having a different reduction pattern in the reflected wave amplitude. The electronic device 101 may also have pre-stored the reflected wave amplitude reduction pattern in the storage unit. The active position setting module 1230 may perform a beamforming operation to send a beam sequence to a setting position for determining activity (e.g., the position of the eyes, nose, or mouth of the object image), and may receive a signal reflected from the subject's object. The activity determination module 1240 can determine activity by learning (training) by comparing the phase or TOF pattern of the millimeter wave image data with the micro-motion pattern of the object stored in the memory or with the human skin pattern (which is pre-stored according to the amplitude reduction pattern of the frequency of the reflected wave).

[0177] When the electronic device 101 recognizes activity (when the conditions for activity determination (activity detection) are met, or when the characteristics of the reflected wave are identified as a reflected wave signal similar to that of an object that is part of the human body), the authentication position setting module 1250 can perform additional beamforming operations in the direction of the position of the object used for object authentication (e.g., an object area or a portion of an area of ​​the object including multiple feature points in the object area). The electronic device 101 can additionally send a beam sequence to the position of the object of the subject for object authentication, and receive a signal reflected from the object position of the corresponding subject. The object authentication module 1260 can authenticate the user's object based on the image of the identified object, the sequence of reflected waves, and the object authentication template stored in the memory.

[0178] According to some embodiments, when the millimeter wave image data is generated, the electronic device 101 may perform an operation for comparing the millimeter wave image data with data (template) that has been generated by image learning (feature extraction operation by a deep learning (e.g., CNN) algorithm) and determining the success or failure of object authentication by calculating a matching score according to a previously determined algorithm. When the object authentication succeeds or fails, the electronic device 101 may use a pre-stored object template as additional data for continuous updating.

[0179] According to some embodiments, the electronic device 101 may include a housing, the housing including a first plate facing a first direction, a second plate facing a direction opposite to the direction facing the first plate, and a side member surrounding a space between the first plate and the second plate, a display 240 exposed through a first portion of the first plate, a camera module 1010 exposed through a second portion of the first plate adjacent to the display 240 and arranged to face the first direction, an antenna array 230 arranged within the housing and / or on a portion of the housing, a wireless communication module 220 electrically connected to the antenna array 230 and configured to form a directional beam through the antenna array 230, a sensor module 1020 configured to sense illumination intensity, and a processor 200 located within the housing and operably connected to the display 240, the camera module 1010, the wireless communication module 220, and the sensor module 1020. The processor 200 may identify the brightness near the electronic device through the sensor module, perform a first object authentication operation through the antenna array 230 and the wireless communication module 220 when the sensed brightness is equal to or lower than the set level, and perform a second object authentication operation through the camera module 1010, the antenna array 230 and the wireless communication module 220 when the sensed brightness is higher than the set level. The first object authentication operation may include sending a first directional beam sequence having a first beam width through the antenna array 230 to scan a first area 311 ... 326 having a first area, receiving a first reflected wave sequence generated by reflection of the first directional beam sequence from an object 330 through the antenna array 230, sending a second directional beam sequence having a second beam width narrower than the first beam width through the antenna array 230 based on at least a portion of the received first reflected wave sequence to scan a second area 351 ... 366 included in the first area and having a second area smaller than the first area, receiving a sequence of second reflected waves generated by reflection of the second directional beam sequence from the object 330 through the antenna array 230, and authenticating the object 330 based on at least a portion of the second reflected wave sequence.

[0180] According to certain embodiments, the second object authentication operation of the processor 200 may acquire at least one image through the camera module 1010, identify the object 330 in the image, send a sequence of at least one directional beam in the direction of the object 330 through the antenna array 230, receive a reflected wave sequence generated by reflection of the beam sequence from the object 330 through the antenna array 230, and authenticate the object 330 based on the identified object 330 and at least a portion of the reflected wave sequence.

[0181] According to some embodiments, the processor 200 may activate an antenna selected from the antenna array in order to determine the first beam width and / or the second beam width.

[0182] According to some embodiments, the processor 200 may determine a relative distance and / or direction between the object and the electronic device based on the sequence of first reflected waves.

[0183] According to some embodiments, the camera module 1010 may include an image sensor arranged to face a first direction, and the processor 200 may include the first area 311 . . . 326 in the field of view of the image sensor. Fig.13 is a diagram showing an electronic device (eg, Figure 1 Flowchart of method 1300 of electronic device 101) authenticating object 330 based on camera image and beam.

[0184] Fig.13 is a flowchart illustrating a second object authentication method of the electronic device 101 according to some embodiments.

[0185] refer to Fig.13 In operation 1311, the electronic device 101 may use a camera module (eg, Figure 1 Camera module 180 or Fig.10 The electronic device 101 may obtain an image including the object 330 by using the camera module 1010 of the electronic device 101. In operation 1313, the electronic device 101 may identify the object 330 region and the main part of the object 330 in the obtained image. In operation 1315, the electronic device 101 may identify feature points for authenticating the object 330 in the image of the recognized object. The electronic device 101 may calculate the coordinates of the position of the object 330 region or the part that may be the feature points of the object 330 in the image obtained by the camera module 1010. The electronic device 101 may select a position for determining the activity of the object 330 and a position for authenticating the object 330. For example, when the object 330 is the face of the user, the position for determining the activity may be the pixel coordinates of a specific part, which may be the feature points of the face, such as the eyes, nose, or mouth in the face image, and the position for authenticating the face may be a set of pixel coordinates that may be collected in the entire area of ​​the face image within a limited time.

[0186] In operation 1317, the electronic device 101 may determine a directional beam sequence for determining the activity of the object 330. For example, the electronic device 101 may calculate the azimuth and elevation of the beam for determining the activity by applying the activity determination location information to equation (2) or equation (3). In operation 1319, the electronic device 101 may send the beam sequence to the activity determination location based on the calculated azimuth and elevation. In operation 1321, the electronic device 101 may receive a reflected wave of the beam reflected from the subject. For example, the electronic device 101 may receive a reflected wave of the beam reflected from the subject through the antenna array 230. In operation 1323, the electronic device 101 may calculate a phase, TOF and / or amplitude value from the received reflected wave, and determine the activity of the object 330 based on the calculated values. According to an embodiment, the memory (e.g., Fig.10 The memory 210 of the device 101 may store an activity template for determining the activity of the object 330. For example, the activity template may be a specific value generated by learning the changes in the phase, TOF, and / or amplitude of the millimeter wave device based on the tiny movements (vibrations) of the object 330 through a deep learning scheme. The electronic device 101 may identify a matching score by matching the phase, TOF, and / or amplitude values ​​of the reflected wave received by the millimeter wave device with the learned activity template, and determine the activity of the object 330 based on the identified matching score. When there is no activity of the object 330 (false), the electronic device 101 may determine that the object authentication has failed and end the authentication operation.

[0187] When there is activity (true) of the object 330, in operation 1325, the electronic device 101 may identify whether there is additional position information for object authentication (position information (pixel coordinates) for object authentication). For example, the memory may store an activity template and an object authentication template, and the electronic device 101 may determine feature points for activity determination and object authentication based on operation 1315. In operation 1325, the electronic device 101 may determine a directional beam sequence for object authentication. For example, the electronic device 101 may calculate the azimuth and elevation of the beam for a predetermined position of the image of the object 330 by applying the object authentication position information to equation (2) or equation (3) in order to authenticate the object 330. In operation 1327, the electronic device 101 may send the beam sequence to the object authentication position of the subject based on the calculated azimuth and elevation. In operation 1329, the electronic device 101 may receive a reflected wave of the directional beam sequence reflected from the subject. The electronic device 101 may calculate phase, TOF, and amplitude values ​​from the reflected waves received through the antenna array 230 and match the calculated values ​​with the object authentication template to perform an object authentication operation.

[0188] According to certain embodiments, a method for authenticating an object 330 by an electronic device 101 may include an operation of sensing brightness around the electronic device 101 through a sensor module 1020, an operation of performing a first object 330 authentication operation using a signal received through an antenna array 230 and a wireless communication module 220 arranged inside a housing of the electronic device 101 and / or on a portion of the housing when the sensed brightness is equal to or lower than a set level, and an operation of performing a second object 330 authentication operation for authenticating the object 330 through an image acquired by a camera 1010 and a signal received through the antenna array 230 and the wireless communication module 220 when the sensed brightness is higher than or equal to the set level. The first object 330 authentication operation can be performed by sending a first directional beam sequence with a first beam width through the antenna array 230 to scan a first area with a first area, receiving a first reflected wave sequence generated by reflection of the first directional beam sequence from the object 330 through the antenna array 230, sending a second directional beam sequence with a second beam width narrower than the first beam width through the antenna array 230 based on at least a portion of the received first reflected wave sequence to scan a second area 351...366 included in the first area and having a second area smaller than the first area, receiving a second reflected wave sequence generated by reflection of the second directional beam sequence from the object 330 through the antenna array 230, and authenticating the object 330 based on at least a portion of the second reflected wave sequence.

[0189] According to certain embodiments, the second object authentication operation may include an operation of acquiring at least one image through the camera 1010, an operation of identifying an object in the image, an operation of sending at least one directional beam sequence in the direction of the object through the antenna array 230, an operation of receiving a reflected wave sequence generated by reflection of the beam sequence from the object through the antenna array 230, and an operation of authenticating the object based on the identified object and at least a portion of the reflected wave sequence.

[0190] According to some embodiments, the operation of transmitting the first directional beam sequence may include activating a certain number of antennas selected from the antenna array to determine the first beam width. The operation of transmitting the second directional beam sequence may include activating a greater number of antennas than the certain number of antennas selected from the antenna array to determine the first beam width to determine the second beam width.

[0191] According to certain embodiments, the operation of sending a second directional beam sequence may also include an operation of determining a relative distance and / or direction between an object and an electronic device based on the first reflected wave sequence, an operation of generating a second directional beam sequence to be sent to a second area 351…366 included in the first area based on the determined distance and / or direction, and an operation of sending the generated second directional beam sequence.

[0192] According to some embodiments, the first object authentication operation may further include an operation of setting the first area to be included in the viewing angle of the camera 1010 and guiding the object to be positioned in the first area.

[0193] Although the present disclosure has been shown and described with reference to certain embodiments thereof, it will be apparent to those skilled in the art that the electronic device according to the present disclosure is not limited to these embodiments and that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. An electronic device include: Antenna arrays; a wireless communication module electrically connected to the antenna array and configured to form a directional beam through the antenna array; at least one processor operably connected to the wireless communication module; as well as a memory operably connected to the at least one processor, The memory stores instructions, and when the instructions are executed by the at least one processor, the electronic device performs multiple operations, and the multiple operations include: identifying a type of object based on information received from a user interface or a camera; identifying a first beam width based on the identified type of object; transmitting a first directional beam sequence having the first beam width through the antenna array to scan a first area having a first size, receiving, by the antenna array, from the object a first reflected wave sequence generated by reflections of the first directional beam sequence, Based on at least a portion of the received first reflected wave sequence, a second directional beam sequence having a second beam width narrower than the first beam width is transmitted through the antenna array to scan a second area included in the first area and having a second size smaller than the first size, receiving, by the antenna array, from the object a second reflected wave sequence generated by reflections of the second directional beam sequence, and The object is authenticated based on at least a portion of the second reflected wave sequence. 2 . The electronic device of claim 1 , wherein the plurality of operations further comprises activating a certain number of antennas selected from the antenna array to determine the first beamwidth or the second beamwidth. 3 . The electronic device of claim 1 , wherein the plurality of operations further comprises determining a relative distance or direction between the object and the electronic device based on the first reflected wave sequence.

4. The electronic device according to claim 1, further comprising: include: a housing including a first plate facing a first direction, a second plate facing a direction opposite to the direction faced by the first plate, and a side member surrounding a space between the first plate and the second plate; a display exposed through the first portion of the first panel; as well as an image sensor exposed through the second portion of the first board and arranged to face the first direction, and The multiple operations further include: The first region in the field of view of the image sensor is identified. The electronic device of claim 1 , wherein the object comprises a face of a user.

6. The electronic device according to claim 1, further comprising: include: a sensor module configured to sense illumination intensity; as well as The multiple operations further include: Identify the brightness around the electronic device by using the sensor module, When the identified brightness exceeds a predetermined value, acquiring at least one image by the camera, and The object in the image is identified.

7. A method for authenticating an object by an electronic device, the method include: identifying the type of the object based on information received from a user interface or a camera; identifying a first beam width based on the identified type of object; transmitting, via the antenna array, a first sequence of directional beams having the first beam width to scan a first area having a first size; receiving, by the antenna array, from an object a first reflected wave sequence generated by reflections of the first directional beam sequence; Based on at least a portion of the received first reflected wave sequence, transmitting, through the antenna array, a second directional beam sequence having a second beam width narrower than the first beam width to scan a second area included in the first area and having a second size smaller than the first size; receiving, by the antenna array, from the object a second reflected wave sequence generated by reflections of the second directional beam sequence; as well as The object is authenticated based on at least a portion of the second reflected wave sequence.

8. A method according to claim 7, wherein the sending of the first directional beam sequence includes activating a certain number of antennas selected from an antenna array to determine a first beam width, and the sending of the second directional beam sequence includes activating a greater number of antennas than the certain number of antennas selected from the antenna array to determine the first beam width in order to determine a second beam width.

9. The method according to claim 7, wherein the second directional beam sequence is transmitted include: determining a relative distance or direction between the object and the electronic device based on the received first reflected wave sequence; generating the second directional beam sequence to be transmitted to a second area included in the first area based on the determined distance and / or direction; and The generated second directional beam sequence is transmitted. 10 . The method of claim 7 , further comprising identifying the first area in a field of view of a camera arranged to face a first direction, and directing the object to be positioned in the first area. The method of claim 7 , wherein the object comprises a face of a user.

12. The method according to claim 7, further comprising: include: Identify the brightness around the electronic device through a sensor module, When the identified brightness exceeds a predetermined value, acquiring at least one image by a camera, and The object in the image is identified.