Biometric authentication system and electronic device thereof

Abstract

A foldable electronic device is disclosed, including a first housing, a second housing, at least one sensor, a display, a camera operable in a first mode or a second mode, a processor, and a memory operably connected to the processor. The foldable electronic device displays notification information about an event on the display, acquires at least one first image using the camera in the first mode if a folding state of the foldable electronic device sensed using the at least one sensor satisfies a specified condition, determines whether a face is present based on the at least one first image acquired using the camera, acquires at least one second image using the camera in the second mode if it is determined that the face is present in the at least one first image, performs a biometric authentication based on face recognition using at least a portion of the at least one second image, and displays content of the event on the display if the biometric authentication is successful. Various embodiments inferred from the specification are also possible.

Description

Technical Field

[0001] The various embodiments disclosed in this specification relate to a method for performing biometric authentication and an electronic device for biometric authentication. Background Technology

[0002] While the importance of security increases with the proliferation of digital devices, various authentication methods, such as biometric authentication, are used to ensure the security of user information. Biometric authentication refers to the extraction of personally unique biological information, such as fingerprints, irises, sweat gland structures, facial features, and / or blood vessels, and the use of this extracted biometric information to authenticate the user. Biometric authentication is used in electronic devices for a variety of purposes. For example, it is applied in various fields, such as unlocking the home screen to use an electronic device and authenticating users when making e-commerce payments.

[0003] In addition, technologies are being developed to increase the display surface of electronic devices. This could alter the form of electronic devices to improve portability and display size in mobile devices. For example, a portion of the electronic device could be folded or extended. Summary of the Invention

[0004] Technical issues

[0005] With the diversification of electronic devices, there is a need for a convenient means of providing services based on existing technologies. For example, a user could employ a foldable electronic device in a semi-folded state (e.g., desktop display mode). By placing a portion of the electronic device on a flat surface such as a table, the user can view content without needing to grasp the device. When the electronic device receives an event in its semi-folded state, the user may need to perform biometric authentication to identify the content of the event. To perform biometric authentication, the user needs to grasp the electronic device in desktop display mode.

[0006] To perform biometric authentication without physically touching the electronic device, the device can monitor the user's biometric information. For example, the device can determine whether a camera has detected the user's biometric information. The device can monitor biometric information by activating the camera and using it to acquire an image. For example, the device can determine if a specific shape is present in the acquired image. In this process, the device's power consumption may increase due to camera activation.

[0007] Problem Solution

[0008] According to one embodiment disclosed in this specification, an electronic device may include: a first housing; a second housing; a foldable structure located between the first housing and the second housing; at least one sensor; a display; a camera capable of operating in a first mode for user identification or a second mode for user authentication with higher operating power than the first mode; a processor; and a memory operatively connected to the processor. The memory may store one or more instructions that, when executed, cause the processor to: display notification information of an event of the foldable electronic device on the display; acquire at least one first image in the first mode using the camera when the folding state of the foldable electronic device detected by the at least one sensor meets the specified conditions; determine the presence of a face based on the at least one first image acquired using the camera; acquire at least one second image in the second mode using the camera when the presence of the face is determined in the at least one first image; perform facial recognition-based biometric authentication using at least a portion of the at least one second image; and display the content of the event of the foldable electronic device on the display when the biometric authentication is successful.

[0009] According to one embodiment disclosed in this specification, a method for providing notification functionality to a foldable electronic device may include: displaying notification information of an event of the foldable electronic device on a display; determining whether a folded state of the foldable electronic device, detected by using at least one sensor, meets the specified conditions; when the specified conditions are met, acquiring at least one first image using a camera in a first mode provided for user identification; determining the presence of a face based on the at least one first image acquired using the camera; when the presence of the face in the at least one first image is determined, acquiring at least one second image using the camera in a second mode for user authentication; performing biometric authentication based on face recognition using at least a portion of the at least one second image; and when the biometric authentication is successful, displaying the content of the event of the foldable electronic device on the display.

[0010] Beneficial effects

[0011] According to the various embodiments disclosed in this specification, electronic devices can perform biometric authentication functions optimized for users in response to changes in shape.

[0012] Electronic devices can perform effective biometric authentication based on folding angles and information from images obtained through cameras.

[0013] According to the various embodiments disclosed in the specification, the electronic device can allow pixels to be selectively used to perform biometric authentication functions, thereby reducing power consumption and efficiently performing data processing or calculations by the processor.

[0014] In addition, various effects can be provided, either directly or indirectly, through this instruction manual. Attached Figure Description

[0015] Figure 1 This is a block diagram of an electronic device in a network environment according to various embodiments.

[0016] Figure 2 This is a block diagram illustrating a camera module according to various embodiments.

[0017] Figure 3 These are block diagrams of electronic devices according to various embodiments.

[0018] Figure 4 Block diagrams of electronic devices according to various embodiments are shown.

[0019] Figure 5 Electronic devices whose shape has been altered according to various embodiments are shown.

[0020] Figure 6a Electronic devices whose shape has been altered according to various embodiments are shown.

[0021] Figure 6b Electronic devices whose shape has been altered according to various embodiments are shown.

[0022] Figure 7a This illustrates the distance between a user and an electronic device, which varies depending on the shape of the electronic device used by the user, according to an embodiment.

[0023] Figure 7b The illustration shows a user's facial image obtained by a camera during a face recognition-based biometric authentication process, according to an embodiment.

[0024] Figure 8 This is a block diagram illustrating the structure of an image sensor according to an embodiment.

[0025] Figure 9 The state of the image sensor according to the operating state of the electronic device is shown in the embodiment.

[0026] Figure 10 An electronic device for selectively reading out images according to an embodiment is shown.

[0027] Figure 11 The notification operation of the electronic device is illustrated according to an embodiment when multiple faces are detected.

[0028] Figure 12The electronic device according to an embodiment is shown to operate according to various notifications from the application.

[0029] Figure 13 A flowchart illustrating the biometric authentication operation of an electronic device according to various embodiments is shown.

[0030] Figure 14 The diagram illustrates, conceptually, an operational flowchart of an authentication method performed by an electronic device according to various embodiments.

[0031] Regarding the description of the accompanying drawings, identical or similar parts will be labeled with identical or similar reference numerals. Detailed Implementation

[0032] In the following description, various embodiments of the present disclosure may be referenced to the accompanying drawings. Therefore, those skilled in the art will recognize that various modifications, equivalents, and / or substitutions can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure.

[0033] Figure 1 This is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. (Refer to...) Figure 1 In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input device 150, sound output device 155, display device 160, audio module 170, sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module (SIM) 196, or antenna module 197. In some embodiments, at least one of the components (e.g., display device 160 or camera module 180) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit. For example, the sensor module 176 (e.g., a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented as embedded in the display device 160 (e.g., a display).

[0034] Processor 120 may run software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 connected to processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, processor 120 may load commands or data received from another component (e.g., sensor module 176 or communication module 190) into volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to an embodiment, processor 120 may include a main processor 121 (e.g., central processing unit (CPU) or application processor (AP)) and an auxiliary processor 123 (e.g., graphics processing unit (GPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)) that is operationally independent of or combined with the main processor 121. Additionally or alternatively, auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted specifically for a given function. The auxiliary processor 123 can be implemented separately from the main processor 121, or it can be implemented as part of the main processor 121.

[0035] When the main processor 121 is inactive (e.g., in sleep mode), the auxiliary processor 123 may take over from the main processor 121 to control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display device 160, sensor module 176, or communication module 190). Alternatively, when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 may work with the main processor 121 to control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display device 160, sensor module 176, or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally associated with the auxiliary processor 123.

[0036] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, software (e.g., program 140) and input or output data for commands associated with it. Memory 130 may include volatile memory 132 or non-volatile memory 134.

[0037] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0038] Input device 150 can receive commands or data from outside electronic device 101 (e.g., a user) that will be used by other components of electronic device 101 (e.g., processor 120). Input device 150 may include, for example, a microphone, mouse, keyboard, or digital pen (e.g., stylus).

[0039] The sound output device 155 can output sound signals to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. According to an embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

[0040] Display device 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display device 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display device 160 may include touch circuitry adapted to detect touch or sensor circuitry (e.g., a pressure sensor) adapted to measure the intensity of the force caused by touch.

[0041] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input device 150, or output sound via the sound output device 155 or headphones of an external electronic device (e.g., electronic device 102) that is directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0042] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0043] Interface 177 may support one or more specific protocols used to enable electronic device 101 to connect directly (e.g., wired) or wirelessly to external electronic devices (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0044] Connection end 178 may include a connector, through which electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0045] The tactile module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that can be recognized by a user through his touch or kinesthesia. According to embodiments, the tactile module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0046] Camera module 180 can capture still or moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0047] The power management module 188 manages the power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least a portion of, for example, a power management integrated circuit (PMI C).

[0048] Battery 189 can power at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable accumulator, or a fuel cell.

[0049] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors capable of operating independently of processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). One of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wi-Fi Direct, or Infrared Data Association (IRDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components (e.g., multiple chips) that are separate from each other. The wireless communication module 192 can identify and verify the electronic device 101 in the communication network (such as the first network 198 or the second network 199) using user information (e.g., the International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0050] Antenna module 197 can transmit or receive signals or power to or from the outside of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a PCB). According to an embodiment, antenna module 197 may include multiple antennas. In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and external electronic device via the selected at least one antenna. According to an embodiment, additional components besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally incorporated into antenna module 197.

[0051] At least some of the aforementioned components can be interconnected and communicate signals (e.g., commands or data) between them via an inter-peripheral communication scheme (e.g., bus, general purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).

[0052] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 connected to a second network 199. Each of electronic device 102 and electronic device 104 can be a device of the same type as electronic device 101, or a device of a different type. According to an embodiment, all or some operations that would be performed on electronic device 101 can be performed on one or more of external electronic devices 102, external electronic devices 104, or server 108. For example, if electronic device 101 is required to automatically perform a function or service, or is required to perform a function or service in response to a request from a user or another device, electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service, instead of running the function or service, or electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service in addition to running the function or service. Upon receiving the request, the one or more external electronic devices may perform at least a portion of the requested function or service, or perform additional functions or services related to the request, and transmit the result of the execution to electronic device 101. Electronic device 101 may provide the result as at least a partial response to the request, either with further processing or without further processing. For this purpose, technologies such as cloud computing, distributed computing, or client-server computing may be used.

[0053] The electronic device according to various embodiments can be one of a variety of types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. According to embodiments of this disclosure, the electronic device is not limited to those described above.

[0054] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the specific embodiments, but rather to include various changes, equivalents, or substitutions to the respective embodiments. In the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that nouns in the singular form corresponding to terms may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any one or all possible combinations of the items enumerated together with the corresponding phrase among the plurality of phrases. As used herein, terms such as “first” and “second” or “first” and “second” may be used to simply distinguish the respective component from another component and do not limit the component in other respects (e.g., importance or order). It will be understood that, whether the terms “operably” or “communically” are used or not, if an element (e.g., a first element) is referred to as “combined with another element (e.g., a second element),” “combined to another element (e.g., a second element),” “connected to another element (e.g., a second element),” or “attached to another element (e.g., a second element)”, it means that the first element can be directly (e.g., wiredly) connected to the second element, wirelessly connected to the second element, or connected to the second element via a third element.

[0055] As used herein, the term "module" can include a unit implemented in hardware, software, or firmware, and is used interchangeably with other terms (e.g., "logic," "logic block," "part," or "circuit"). A module can be a single integrated component adapted to perform one or more functions, or the smallest unit or part of such a single integrated component. For example, according to an embodiment, a module can be implemented in the form of an application-specific integrated circuit (ASIC).

[0056] The various embodiments set forth herein can be implemented as software (e.g., program 140) containing one or more instructions readable by a machine (e.g., electronic device 101) stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor, the processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. Machine-readable storage media may be provided in the form of non-transitory storage media. The term "non-transitory" means only that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being stored semi-permanently in the storage medium and data being temporarily stored in the storage medium.

[0057] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)) or may be distributed via an app store (e.g., a Play Store). TM The computer program product may be published online (e.g., downloaded or uploaded), or may be distributed directly between two user devices (e.g., smartphones) (e.g., downloaded or uploaded). If published online, at least a portion of the computer program product may be temporarily generated, or at least a portion of the computer program product may be temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a forwarding server).

[0058] According to various embodiments, each of the above-described components (e.g., a module or program) may include a single entity or multiple entities. According to various embodiments, one or more of the above-described components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding component of the multiple components performed one or more functions prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

[0059] Figure 2 This is a frame 200 illustrating a camera module 180 according to various embodiments. (Refer to...) Figure 2 The camera module 180 may include a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (e.g., a buffer memory), or an image signal processor 260. The lens assembly 210 may capture light emitted or reflected from an object whose image is to be captured. The lens assembly 210 may include one or more lenses. According to embodiments, the camera module 180 may include multiple lens assemblies 210. In this case, the camera module 180 may form, for example, a dual-camera system, a 360-degree camera, or a spherical camera. Some of the multiple lens assemblies 210 may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that differ from the lens properties of the other lens assemblies. The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

[0060] Flash 220 is capable of emitting light, wherein the emitted light is used to enhance light reflected from an object. According to embodiments, flash 220 may include one or more light-emitting diodes (LEDs) (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared (IR) LEDs, or ultraviolet (UV) LEDs) or xenon lamps. Image sensor 230 acquires an image corresponding to an object by converting light emitted or reflected from the object and transmitted through lens assembly 210 into an electrical signal. According to embodiments, image sensor 230 may include one image sensor selected from a plurality of image sensors with different properties (e.g., an RGB sensor, a black-and-white (BW) sensor, an IR sensor, or a UV sensor), a plurality of image sensors having the same properties, or a plurality of image sensors with different properties. Each image sensor included in image sensor 230 may be implemented using, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor.

[0061] Image stabilizer 240 can move image sensor 230 or at least one lens included in lens assembly 210 in a specific direction, or control the operability properties of image sensor 230 (e.g., adjust readout timing) in response to movement of camera module 180 or electronics 101 including camera module 180. This allows compensation for at least a portion of the negative effects (e.g., image blur) caused by movement of the image being captured. According to embodiments, image stabilizer 240 can use a gyroscope sensor (not shown) or an accelerometer sensor (not shown) disposed within or outside camera module 180 to sense such movement of camera module 180 or electronics 101. According to embodiments, image stabilizer 240 can be implemented as, for example, an optical image stabilizer.

[0062] Memory 250 may at least temporarily store at least a portion of the images acquired via image sensor 230 for subsequent image processing tasks. For example, if multiple images are captured rapidly or if image capture is delayed due to shutter lag, the acquired raw images (e.g., Bayer pattern images, high-resolution images) may be stored in memory 250, and their corresponding copy images (e.g., low-resolution images) may be previewed via display device 160. Then, if specified conditions are met (e.g., by user input or system commands), at least a portion of the raw images stored in memory 250 may be acquired and processed by, for example, image signal processor 260. According to embodiments, memory 250 may be configured as at least a portion of memory 130, or memory 250 may be configured as a separate memory operating independently of memory 130.

[0063] Image signal processor 260 can perform one or more image processing operations on images acquired via image sensor 230 or stored in memory 250. The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image compositing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Alternatively or additionally, image signal processor 260 can perform control (e.g., exposure time control or readout timing control) on at least one component included in camera module 180 (e.g., image sensor 230). Images processed by image signal processor 260 can be stored back in memory 250 for further processing, or the image can be provided to external components outside camera module 180 (e.g., memory 130, display device 160, electronic device 102, electronic device 104, or server 108). According to embodiments, image signal processor 260 can be configured as at least a part of processor 120, or image signal processor 260 can be configured as a separate processor operating independently of processor 120. If the image signal processor 260 is configured as a processor separate from the processor 120, the processor 120 can display at least one image processed by the image signal processor 260 as is via the display device 160, or the at least one image can be displayed after further processing.

[0064] According to an embodiment, the electronic device 101 may include a plurality of camera modules 180 with different attributes or functions. In this case, at least one of the plurality of camera modules 180 may form, for example, a wide-angle camera, and at least another of the plurality of camera modules 180 may form a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may form, for example, a front-facing camera, and at least another of the plurality of camera modules 180 may form a rear-facing camera.

[0065] Figure 3 This is a frame 300 illustrating a display device 160 according to various embodiments. (Refer to...) Figure 3The display device 160 may include a display 310 and a display driver integrated circuit (DDI) 330 for controlling the display 310. The DDI 330 may include an interface module 331, a memory 333 (e.g., a buffer memory), an image processing module 335, or a mapping module 337. The DDI 330 may receive image information containing image data or image control signals corresponding to commands for controlling the image data from another component of the electronic device 101 via the interface module 331. For example, according to an embodiment, the image information may be received from a processor 130 (e.g., a main processor 131 (e.g., an application processor)) or an auxiliary processor 133 (e.g., a graphics processing unit), wherein the auxiliary processor 133 operates independently of the functions of the main processor 131. The DDI 330 may communicate with, for example, a touch circuit 350 or a sensor module 176 via the interface module 331. The DDI 330 may also store at least a portion of the received image information in the memory 333, for example, frame-by-frame.

[0066] Image processing module 335 can perform preprocessing or postprocessing (e.g., adjustments to resolution, brightness, or size) on at least a portion of image data. According to embodiments, for example, preprocessing or postprocessing can be performed at least in part based on one or more features of the image data or one or more features of the display 310.

[0067] The mapping module 337 can generate voltage or current values ​​corresponding to image data preprocessed or post-processed by the image processing module 335. According to embodiments, for example, the generation of voltage or current values ​​can be performed at least in part based on one or more attributes of pixels (e.g., the array of pixels (such as RGB stripes or pentile structures) or the size of each sub-pixel). For example, at least some pixels of the display 310 can be driven at least in part based on voltage or current values, thereby enabling the display 310 to display visual information (e.g., text, images, or icons) corresponding to the image data.

[0068] According to an embodiment, the display device 160 may further include touch circuitry 350. Touch circuitry 350 may include touch sensor 351 and touch sensor IC 353 for controlling touch sensor 351. Touch sensor IC 353 may control touch sensor 351 to sense touch input or hover input for a specific location on display 310. For this purpose, for example, touch sensor 351 may detect (e.g., measure) a signal (e.g., voltage, light intensity, resistance, or one or more charge quantities) corresponding to a specific location on display 310. Touch circuitry 350 may provide input information (e.g., position, area, pressure, or time) indicating the touch input or hover input detected via touch sensor 351 to processor 130. According to an embodiment, at least a portion of touch circuitry 350 (e.g., touch sensor IC 353) may be formed as part of display 310 or DDI 330, or as part of another component (e.g., auxiliary processor 123) located outside display device 160.

[0069] According to an embodiment, the display device 160 may further include at least one sensor (e.g., a fingerprint sensor, an iris sensor, a pressure sensor, or an illuminance sensor) or control circuitry for said at least one sensor in the sensor module 176. In such a case, said at least one sensor or control circuitry for said at least one sensor may be embedded in a portion of a component of the display device 160 (e.g., display 310, DDI 330, or touch circuitry 350). For example, when the sensor module 176 embedded in the display device 160 includes a biometric sensor (e.g., a fingerprint sensor), the biometric sensor may acquire biometric information (e.g., a fingerprint image) corresponding to touch input received via a portion of the display 310. As another example, when the sensor module 176 embedded in the display device 160 includes a pressure sensor, the pressure sensor may acquire pressure information corresponding to touch input received via a portion or the entire area of ​​the display 310. According to an embodiment, the touch sensor 351 or the sensor module 176 may be arranged between pixels in the pixel layer of the display 310, or arranged above or below the pixel layer.

[0070] Figure 4 A block diagram 400 illustrates an electronic device according to various embodiments.

[0071] refer to Figure 4 Electronic devices (e.g.) Figure 1 The electronic device 101 may include a processor 120, a memory 130, a display device 160, a sensor module 176, and / or a camera module 180. Unless otherwise stated, it has the characteristics of… Figure 1 For a description of the configuration for the same identifier number, please refer to [reference needed]. Figure 1The given description.

[0072] Processor 120 may be operatively connected to other components within electronic device 101 (e.g., memory 130, display device 160, sensor module 410, and / or camera module 180). Processor 120 may control components of electronic device 101. For example, processor 120 may control components of electronic device 101 according to one or more instructions stored in memory 130.

[0073] The memory 130 can store the operation history of the electronic device 101 output to the display device 160. When the electronic device 101 has multiple display devices 160, the memory 130 can store the operation history of the electronic device 101 output to each display device 160. For example, the operation history may include a region of the user interface that includes different applications executed on the display device 160 and / or information about said applications.

[0074] Display device 160 can output information processed by electronic device 101. For example, display device 160 can use camera module 180 to output a user interface or graphical user interface (GUI) related to biometric authentication functionality. Furthermore, display device 160 can output notification objects including details related to events received by electronic device 101. Display device 160 can include various types of displays (e.g., ...). Figure 3The display 310 can be at least one of a flexible display, a three-dimensional (3D) display, and a foldable display. Furthermore, the electronic device 101 can include multiple displays 310. For example, the processor 120 can divide the output area into multiple areas and can execute a split-screen mode that outputs different content to each area. In this case, the electronic device 101 can operate in split-screen mode only when it is folded at a specified angle. In split-screen mode, the display 310 can be divided into multiple display areas. For example, the display 310 can be divided into a first display area and a second display area relative to a folding portion (e.g., a hinge portion). For example, the division between the first and second display areas can be a logical division of the display areas. For example, the display 310 can output a third display area obtained by further dividing the display areas. The third display area can be an area formed relative to the folding portion. The third display area can be an area formed to logically distinguish the first and second display areas. Meanwhile, the above division of display areas is not limited to logical division. The first and second display areas can refer to physically divided areas. For example, the first and second display areas can refer to an area of ​​a separate display device included in an electronic device. The number of segmented regions, their size, resolution, and / or the content that can be output are not limited to the examples above. For example, the number of segmented regions, their size, resolution, and / or the content that can be output may be limited depending on the user's intent. For example, processor 120 may identify information related to the content output from each of the first display area, the second display area, and / or the third display area, and may store said information in memory 130.

[0075] Camera module 180 may include multiple cameras. According to one embodiment, camera module 180 may be located on one side (e.g., the front and / or rear) of electronic device 101, and a surface of the electronic device may include a surface that is the same as or different from the surface on which display device 160 is disposed. For example, the multiple cameras may be configured to acquire images at different resolutions. Each of the multiple cameras may include at least one sensor. For example, some sensors may be image sensors that determine whether a user's face is present in an image captured by the lens when electronic device 101 performs biometric authentication.

[0076] Sensor module 176 may include at least one sensor (e.g., a Hall IC sensor, a digital Hall IC sensor, an accelerometer, a gyroscope sensor, and / or a 6-axis sensor). Electronic device 101 may identify its type based on information obtained using sensor module 176 (e.g., the folding angle of electronic device 101). For example, sensor module 176 may include a sensor for identifying the folding angle of the electronic device. For example, sensor module 176 may identify information including the folding angle by detecting shape changes (e.g., folding / unfolding) of electronic device 101, may identify the current shape of electronic device 101 based on the folding angle, and may deliver a signal including information about the shape to processor 120. Processor 120 may identify the folding / unfolding shape and / or folding angle of electronic device 101 based on the signals received from sensor module 176.

[0077] According to one embodiment, processor 120 can determine whether the folding angle of electronic device 101 is within a specified range (e.g., approximately 90 to 130 degrees) based on information including the folding angle obtained using sensor module 176. The state in which electronic device 101 is folded within the specified range can be defined as a half-folded state. Processor 120 can deliver information indicating that electronic device 101 is in a half-folded state to an application, and the application can provide a user interface optimized with reference to said information. For example, in the half-folded state, processor 120 can allow display device 160 to split the screen and output different content in multiple areas. When electronic device 101 is in a half-folded state, a user can use electronic device 101 when it is placed on a table, and this can be defined as table mode. Table 1 shows the operating states based on the folding angle of electronic device 101. Meanwhile, even if the folding angle meets specified conditions based on the installation state of electronic device 101, electronic device 101 can operate outside the half-folded state under the control of processor 120. This can be achieved through the methods described below. Figure 6a and Figure 6b The description will provide a more detailed description of the installation status.

[0078] [Table 1]

[0079] Half-folded state X O X

[0080] According to an embodiment, when electronic device 101 receives an event, processor 120 may allow display device 160 to display a notification object including information corresponding to the event. In this case, under the control of processor 120, the notification object may be output to at least a portion of only the segmented area. For example, when electronic device 101 is used in desktop mode, processor 120 may allow display device 160 to continuously output previously output content in a first display area (e.g., a display area located in the upper portion of electronic device 101) and output the notification object only in a second display area (e.g., a display area located in the lower portion of electronic device 101). In addition to outputting the notification object to display device 160, processor 120 may also provide various types of notification functions when electronic device 101 receives an event. For example, notification functions may be provided intuitively using edge lighting and haptic functions without outputting specific details about the received event to the display.

[0081] According to an embodiment, processor 120 can acquire at least one image through camera module 180 and can process the acquired at least one image. For example, processor 120 can extract a specified object (e.g., a face) from the acquired image. When it is determined that a facial object exists in the image, processor 120 can use the object to perform user authentication (e.g., biometric authentication). Biometric authentication can be iris authentication or facial authentication. As another example, when it is determined that multiple facial objects exist in the image, processor 120 may not perform user authentication.

[0082] Figure 4 The configuration of the electronic device 101 shown is merely an example, and the embodiments of this disclosure are not limited thereto. For example, the electronic device 101 may not include... Figure 4 At least one of the components shown. As another example, electronic device 101 may also include Figure 4 Components not shown in the diagram.

[0083] Figure 5 Electronic devices whose shape has been altered according to various embodiments are shown.

[0084] Electronic device 500 (such as Figure 1 The shape of the electronic device 501 can physically change when folded / unfolded. For example, the electronic device may include a housing and a display, each of which is flexible in at least a portion of the housing and the display. The electronic device can be folded (e.g., closed) or unfolded (e.g., opened) around the flexible portion of the electronic device 500. For example, the flexible portion of the electronic device 500 may be referred to as a folded portion. A folded portion refers to a part (e.g., a hinge) or area in which the shape of the electronic device 500 can be changed, and is not limited to a particular structure.

[0085] According to an embodiment, electronic device 500 (e.g., Figure 1 The electronic device 101 can be folded vertically. For example, the electronic device 500 may include a display 530 (e.g., Figure 3 The display 310 and the housing 510, each of which is flexible in a portion corresponding to at least one folding portion 520. The electronic device 500 can be folded up and down around the folding portion 520. Figure 5 The illustrated electronic device 500 is an inwardly folding electronic device in which the display 530 folds inward. However, embodiments of this disclosure are not limited thereto. For example, at the folding portion 520, the electronic device 500 may fold outward, or may fold both inward and outward. As another example, the illustrated display 530 has a single display, but embodiments of this disclosure are not limited thereto. The electronic device 500 may include multiple displays separated by the folding portion 520. The housing 510 may also include multiple housings separated by the folding portion 520. As another example, the electronic device 500 may be a combination of multiple electronic devices coupled to fold around the folding portion 520. In this case, the multiple electronic devices may be coupled to each other via separate structures (e.g., housings and / or hinges).

[0086] Figure 5 The physical shape variations of the electronic device 500 shown are examples, and the embodiments of this disclosure are not limited thereto. For example, the electronic device can be folded or unfolded about any axis.

[0087] Figure 6a Electronic devices whose shape has been altered according to various embodiments are shown.

[0088] refer to Figure 6a According to various embodiments, electronic device 101 (e.g., Figure 5 The electronic device 500 can be used in various operating modes (e.g., mobile mode 601, desktop mode 602, and tablet mode 603) in response to the folding angle.

[0089] In mobile mode 601, electronic device 101 can be used in the same form as a typical candybar smartphone. For example, in mobile mode 601, electronic device 101 can be used with a flexible display (e.g., Figure 3 Content is provided in a region of the display (310). In mobile mode 601, the folding angle can be 0 degrees.

[0090] In desktop mode 602, the electronic device 101 can function as a laptop computer when placed on a table or in the user's hand. In desktop mode 602, the folding angle (a2) can be between 90 and 130 degrees. In desktop mode 602, under the control of the electronic device 101, the folding angle can be adjusted based on the folding portion (e.g., ...). Figure 5 The folded portion 520) divides the display area, and different content can then be output to the divided area respectively. For example, when the electronic device 101 receives an event, under the control of the electronic device 101, the event notification can be output to the lower area, and previously output content can be continuously output to the upper area. According to an embodiment, when the electronic device 101 provides biometric authentication functionality in desktop mode 602, the electronic device 101 can apply authentication algorithms different from those in mobile mode 601 and tablet mode 603. For example, in desktop mode 602, the distance between the electronic device 101 and the user may be relatively long compared to other modes, thereby applying an authentication algorithm to increase the permissible range of facial recognition (e.g., the distance between the electronic device and the user or the angle of the face). According to various embodiments of this specification, in desktop mode, due to the camera angle between the user's face and the electronic device, the probability of detecting the lower part of the face may be high, thereby applying an algorithm that performs authentication by weighting the features of the lower part of the face.

[0091] Additionally, please refer to the following description. Figure 7a and Figure 7b The permissible range of facial recognition described above is described in more detail.

[0092] In tablet mode 603, electronic device 101 can be used in the same shape as a typical tablet PC. For example, electronic device 101 can be used with a flexible display (e.g., Figure 3 Content is provided in a region of the display (310). In tablet mode 603, the folding angle (a3) ​​of the electronic device 101 can be 180 degrees.

[0093] Figure 6b Electronic devices whose shape has been altered according to various embodiments are shown.

[0094] According to various embodiments of this specification, when folded at a specified angle (e.g., 90 to 130 degrees), the electronic device 101 can be installed and used in various shapes (e.g., first mounting state 604 and second mounting state 605).

[0095] According to an embodiment, a user can use the electronic device 101 in a first mounting state 604. The first mounting state 604 may refer to the state in which the side portion of the electronic device 101 is placed on the floor when the electronic device 101 is folded at a specified angle. The electronic device 101 can output content to a flexible display (e.g., Figure 3 A region of the display 310 (e.g., a first display area 610 or a second display area 620).

[0096] According to an embodiment, the second installation state 605 may refer to the state where the user places the casing including the first display area 610 on the floor when the electronic device 101 is folded. The electronic device 101 can output content to a flexible display (e.g., Figure 3 A region of the display 310 (e.g., a first display area 610 or a second display area 620).

[0097] Processor (e.g., Figure 1 The processor 120 can be based on the use of a sensor module (e.g., Figure 1 The sensor module 176 obtains the folding angle of the electronic device 101 to determine the operating mode of the electronic device 101 (e.g., mobile mode, desktop mode, or tablet mode). However, even if the folding angle meets the specified conditions in the mounted state (first mounted state 604 or second mounted state 605), the electronic device 101 may not operate in desktop mode.

[0098] Figure 7a This illustrates the distance between a user and an electronic device, which varies depending on the shape of the electronic device used by the user, according to an embodiment.

[0099] According to various embodiments of this specification, the foldable electronic device 101 can be used in various arrangements, such as arrangement 701 and arrangement 702, depending on the folding angle, the shape in which the user grips it, etc. For example, the user can use it in a mobile mode state (e.g., Figure 6a In mobile mode 601, the user can use the electronic device 101 by gripping the lower part with his / her hand. As another example, the user can use desktop mode (e.g., Figure 6a The electronic device 101 is used in the desktop mode 602.

[0100] The distances D1 and D2 between the user's face and the electronic device 101 can vary depending on the shape of the user's adoption of the electronic device 101. For example, distance D1 (e.g., 20cm to 50cm) can be smaller than distance D2 (e.g., 20cm to 90cm).

[0101] According to an embodiment, the electronic device 101 may use a camera module (e.g., Figure 4The camera module 180 acquires at least one image including the user's face. The size of the face included in the at least one image acquired by the camera module 180 can be determined by the distance between the user and the electronic device 101. In this case, the electronic device 101 can be configured to successfully perform face recognition, provided that the size of the face included in the at least one image is not less than a specified size.

[0102] refer to Figure 7a According to reference numeral 701, in an embodiment, the user can use a mobile mode (e.g., when the folding angle is 0 degrees) Figure 6a The mobile mode 601) employs electronic device 101. When electronic device 101 receives an event, the display device (… Figure 4 The display device 160 can provide a notification function to output a notification object including information about the received event in an area (e.g., a first display area 610 or a second display area 620). The user can identify additional details related to the event through facial recognition. The user may attempt facial recognition when the distance to the electronic device is D1.

[0103] refer to Figure 7a According to reference numeral 702, in one embodiment, the user can use a tabletop mode (e.g., when the folding angle is within 90 degrees to 130 degrees) Figure 6a The desktop mode 602) employs electronic device 101. When electronic device 101 receives an event, the display device (… Figure 4 The display device 160 can provide a notification function to output a notification object including information about the received event in an area (e.g., a first display area 610 or a second display area 620). The user can identify additional details related to the event through facial recognition. The user may attempt to recognize a face when the distance to the electronic device is D2. In reference numeral 702, the size of the face included in at least one image obtained by the electronic device 101 may be smaller than the face size in reference numeral 701. The electronic device 101 can apply an authentication algorithm that increases the permissible range of facial recognition at reference numeral 702. For example, as in reference numeral 702, when the user uses the electronic device 101 in a semi-folded state in tabletop mode, the processor (e.g., ...) Figure 4 The processor 120 can extend and apply the permissible range of facial recognition based on the distance between the electronic device 101 and the user. Table 2 shows the permissible range of facial recognition according to the operating mode of the electronic device 101.

[0104] [Table 2]

[0105]

[0106] Figure 7bThe illustration shows a user's facial image obtained by a camera during a face recognition-based biometric authentication process, according to an embodiment.

[0107] According to various embodiments of this specification, for the operation of the facial recognition function, the electronic device 101 may use a camera module (e.g., Figure 4 The camera module 180) acquires at least one image including the user's face. (Reference) Figure 7b Reference numeral 703, according to one embodiment, the facial image of the user obtained by the electronic device 101 can refer to various radii (710, 720, 730, 740).

[0108] According to an embodiment, when a user uses the electronic device 101 in a semi-folded state in tabletop mode, the processor (such as...) Figure 4 The processor 120 can extend the permissible range of facial recognition based on the distance between the electronic device 101 and the user. Table 3 shows the permissible range of facial recognition according to the operating mode of the electronic device 101.

[0109] [Table 3]

[0110]

[0111] Figure 8 This is a frame 800 for describing the structure of an image sensor according to various embodiments.

[0112] According to embodiments of this specification, the image sensor 810 (e.g., Figure 2 The image sensor 230 can be located in an electronic device (e.g., Figure 1 The camera module in the electronic device 101 (e.g., Figure 1 (The camera module) is a component.

[0113] refer to Figure 8 According to various embodiments, the image sensor 810 may include at least one of a pixel array 820, a row driver 830, a column readout circuit 840, a controller 850, a memory 860, and an interface 870.

[0114] Pixel array 820 may include a plurality of pixels 811 to 819. For example, pixel array 820 may have a structure in which the plurality of pixels 811 to 819 are arranged in an M×N matrix ("M" and "N" are natural numbers). Pixel array 820 with the plurality of pixels 811 to 819 arranged in an M×N two-dimensional shape may have "M" rows and "N" columns. Pixel array 820 may include a plurality of light-sensing elements, such as photodiodes or pinned photodiodes. Pixel array 820 may use the plurality of light-sensing elements to detect light and may convert the light into analog electrical signals to generate image signals.

[0115] Row driver 830 can drive pixel array 820 row by row. For example, row driver 830 can output to pixel array 820 a transfer control signal for controlling transfer transistors, a reset control signal for controlling reset transistors, or a selection control signal for controlling selection transistors. Each of the plurality of pixels 811 to 819 included in pixel array 820 includes a transfer transistor, a reset transistor, or a selection transistor. Row driver 830 can determine the row to be read. Column readout circuit 840 can receive analog electrical signals generated by pixel array 820. For example, column readout circuit 840 can receive analog electrical signals from column lines selected from the plurality of columns constituting pixel array 820. Column readout circuit 840 may include an analog-to-digital converter (hereinafter referred to as "ADC") 845 capable of converting the analog electrical signals received from the selected column lines into pixel data (or digital signals) and outputting pixel data. Meanwhile, the operation of column readout circuit 840, which receives analog electrical signals from pixel array 820, converts the received analog electrical signals into pixel data using ADC 845, and outputs pixel data, can be referred to as "reading". The column readout circuit 840 and ADC 845 can determine the column to be read.

[0116] According to an embodiment, the column readout circuit 840 of the image sensor 810 may include a plurality of ADCs 845. The ADCs 845 may be connected in parallel with a plurality of photodiodes included in the pixel array 820. The ADCs 845 can rapidly convert analog electrical signals simultaneously received from the multiple photodiodes into pixel data based on a parallel structure.

[0117] According to an embodiment, for operation of the electronic device in a low-power state, the column readout circuit 840 of the image sensor 810 can perform readout at a low frame rate (e.g., 10 frames per second (fps)). For example, performing readout at 10 fps could mean performing the operation of receiving analog electrical signals from the pixel array 820, converting the received analog electrical signals into pixel data using the ADC 845, and outputting the pixel data every 1 / 10 of a second. That is, performing readout at 10 fps could mean outputting ten image frames per second.

[0118] The controller 850 can obtain an image frame based on pixel data received from the column readout circuit 840. The controller 850 can output the image frame to an external circuit 880 (e.g., an image signal processor (ISP), a processor, a communication circuit, or an external server) via interface 870. According to an embodiment, the controller 850 can generate a transmission control signal for controlling a transmission transistor, a reset control signal for controlling a reset transistor, or a selection control signal for controlling a selection transistor in each of the plurality of pixels 811 to 819, and can provide the generated signals to the row driver 830. Furthermore, the controller 850 can generate a selection control signal for selecting at least one column line among the plurality of column lines constituting the pixel array 820, and can provide the generated signal to the column readout circuit 840. For example, based on the selection control signal provided from the controller 850, the column readout circuit 840 can enable some of the column lines and disable the others. Furthermore, the controller 850 can be implemented as a processor including a central processing unit (CPU) or an application processor (AP) (e.g., ...). Figure 1 The processor 120, or the type implemented as a block or module.

[0119] According to an embodiment, memory 860 can store at least one image frame read from column readout circuit 840 at an Nth frame rate (e.g., 60 fps) or an Mth frame rate (e.g., 30 fps), and can deliver the stored at least one image frame to external circuit 880 (e.g., an IP address, processor, communication circuit, or external server) via interface 870. That is, memory 860 can store at least one image frame read from column readout circuit 840 once every 1 / 60 second or once every 1 / 30 second, and can deliver the stored at least one image frame to external circuit 880 via interface 870. There is no limitation on the speed at which the image frame is delivered to external circuit 880. For example, electronic devices (e.g., Figure 1 The electronic device 101 can directly deliver the read image frame to the external circuit 880 through the interface 870, without storing the read image frame in the memory 860.

[0120] Simultaneously, the controller 850 can store only some of the image frames from the Nth image frames read out by the column readout circuit 840 at the Nth frame rate (e.g., 10 fps) in the memory 860, thereby achieving essentially the same effect as obtaining the Mth image frame read out at the Mth frame rate (e.g., 5 fps). For example, the controller 850 can store only one image frame from the eight image frames read out at 10 fps within 8 / 10 seconds in the memory 860. When only image frames selected at a 1:8 ratio from the image frames read out at 10 fps are stored in the memory 860, the image frames stored in the memory 860 can be substantially the same as the image frames read out by the column readout circuit 840 at 5 fps. For example, when a video consisting only of image frames obtained in 1 / 5-second intervals is defined as a "5 fps video," a video consisting only of image frames selected at a 1:8 ratio from the image frames read out at 10 fps can also be defined as a 5 fps video. Alternatively, a video consisting solely of image frames read out at 120fps via the column readout circuit 840 can also be defined as a 5fps video.

[0121] According to an embodiment, the column readout circuit 840 can perform readout at a first FPS (e.g., 10 to 30 fps) less than a specified FPS (e.g., 60 fps). For example, for operation of the display in a low-power state (e.g., low-power preview mode), the controller 850 can store multiple image frames in the memory 860, which are read out by the column readout circuit 840 at the first FPS.

[0122] Controller 850 may allow camera module 180 to process or edit pixel information received from processor 120. Based on the pixel information received from processor 120, processor 120 may allow controller 850 to operate only some of the plurality of pixels 811 to 819 included in pixel array 820. For example, controller 850 may allow row driver 830 to generate a selection signal for selecting one row line from a plurality of row lines. For example, a row line may refer to an even-numbered row line or an odd-numbered row line. For example, controller 850 may allow column readout circuitry 840 to generate a selection signal for selecting one column line from a plurality of column lines. For example, a column line may refer to an even-numbered column line or an odd-numbered column line. As another example, image signal processor may deliver a scaled-down image to a display based on at least one image data acquired by a camera. For example, when electronic device 101 performs a face recognition function, processor 120 may allow electronic device 101 to be driven at low power by reading only the region corresponding to the face in at least one acquired image. A scaled-down image may be output to the display in a low-power state. In this way, the electronic device can selectively use pixels, thereby reducing power consumption and efficiently performing data processing or calculations by the processor 120. See the description below. Figure 9 and Figure 10 The operation of the image sensor 810 described above will be described in more detail.

[0123] Figure 8 The configuration of the image sensor 810 shown is merely an example, and the embodiments disclosed herein are not limited thereto. For example, the image sensor 810 may not include... Figure 8 At least one of the components shown. As another example, the image sensor 810 may also include... Figure 8 Components not shown in the diagram.

[0124] Figure 9 Table 900 shows, according to an embodiment, various states of an image sensor based on the operating state of an electronic device.

[0125] According to various embodiments of this specification, electronic device 101 may allow image sensors to perform facial recognition and / or biometric authentication functions at low power.

[0126] According to an embodiment, electronic devices (such as...) Figure 1 Electronic device 101) can be used in a processor (such as Figure 1 The processor 120) is based on the use of sensor modules (such as Figure 4 The image sensor is turned off when the sensor module 176 in the image sensor module obtains the folding angle of the electronic device 101 and determines whether it is in a semi-folded state. When it is determined that the electronic device 101 is in a semi-folded state in tabletop mode (e.g., ...), Figure 6a When operating in desktop mode 602, the processor 120 can activate the image sensor (e.g., Figure 2 The image sensor 230) can perform facial recognition and / or biometric authentication functions (e.g., biometric authentication based on facial recognition or iris recognition).

[0127] According to an embodiment, electronic device 101 can determine whether a user's face is present in at least one image acquired using camera module 180. In determining whether a face is present, camera module 180 can operate in a first mode provided for user identification. The first mode can operate with less current consumption compared to a second mode, which will be described later. For example, in the first mode, camera module 180 can allow the image sensor to perform readout at a first output speed (e.g., 15 fps). As another example, in the first mode, camera module 180 can allow the image sensor to acquire an image with a first resolution (e.g., 640 × 480). When the presence of a face is determined, electronic device 101 can perform a biometric authentication function based on face recognition. In determining whether a face is present, the image sensor can output an image with the first resolution at the first output speed. Processor 120 can determine whether a user's face is present based on the image with the first resolution.

[0128] According to an embodiment, when the presence of a user's face is determined during facial recognition, the electronic device 101 can perform a biometric authentication function. During the execution of the biometric authentication function, the camera module 180 can operate in a second mode provided for user authentication. The second mode can operate with higher current consumption than the first mode. For example, in the second mode, the camera module 180 can allow the image sensor to perform readout at a second output speed (e.g., 30 fps). As another example, in the second mode, the camera module 180 can allow the image sensor to acquire an image with a second resolution (e.g., 1920 × 1080). The second output speed (e.g., 30 fps) and the second resolution (e.g., 1920 × 1080) can refer to values ​​higher than the first output speed (e.g., 15 fps) and the first resolution (e.g., 640 × 480). In this case, the image sensor can output an image at the second resolution at the second output speed. The processor 120 can perform the biometric authentication function based on the image at the second resolution.

[0129] Simultaneously, when performing biometric authentication, electronic device 101 can allow the image sensor to read only a portion of the image obtained by camera module 180 in the second mode. For example, in the second mode, electronic device 101 can use camera module 180 to perform biometric authentication based on facial recognition. In this case, in the second mode, electronic device 101 can perform biometric authentication by reading only the region corresponding to the user's face in at least one image obtained by camera module 180. For example, the region corresponding to the user's face could be the region used by electronic device 101 in determining the presence of a face in the first mode. In this way, electronic device 101 can efficiently perform biometric authentication. For example, since only the region corresponding to the face needs to be extracted and read, biometric authentication processing can be performed in a shorter time.

[0130] The frame rates (15fps and 30fps) and resolutions (640×480 and 1920×1080) used in this specification may vary depending on the electronic device (e.g., Figure 1 The settings or interfaces of the electronic device 101) Figure 1 The performance of the interface 177 varies.

[0131] Figure 10 An electronic device for selectively reading out images according to an embodiment is shown.

[0132] According to an embodiment, the electronic device 101 can be in a semi-folded state in a tabletop mode (e.g., Figure 6a The electronic device 101 operates in desktop mode 602. In desktop mode 602, the electronic device 101 can provide content to the user through a first display area 1010 and / or a second display area 1020. When the electronic device 101 receives an event, a notification object for providing a notification of the received event can be displayed in the second display area 1020. In this case, the first display area 1010 can continuously output the initially provided content, regardless of whether an event has been received. After displaying the notification object, the electronic device 101 can perform facial recognition and / or biometric authentication functions based on at least one image obtained using the camera module 180.

[0133] According to an embodiment, during the execution of facial recognition and / or biometric authentication functions, the electronic device 101 can extract and read only the region 1030 corresponding to the user's face from at least one image. Simultaneously, when the facial recognition and / or biometric authentication function fails to execute successfully on at least one image (e.g., when the face is closer or farther than a specified distance range), the electronic device 101 can provide guidance for successful biometric authentication. For example, the guidance can be provided in the form of images and / or text, including information indicating the user's location such that an external object (e.g., a face) is within a specified distance range from the electronic device 101. For example, the guidance can be provided in a pop-up format on a first display area 1010 and / or a second display area 1020.

[0134] After the authentication process is completed, the electronic device can output content related to the received event on the first display area 1010 and / or the second display area 1020.

[0135] Figure 11 The notification operation 1100 of the electronic device is shown according to an embodiment when multiple faces are detected.

[0136] According to various embodiments of this specification, when a user, together with an external user, is in desktop mode (e.g., ...), Figure 6a When the electronic device 101 is used in the desktop mode 602, notification functionality for events received by the electronic device 101 can be provided in a limited manner. For example, the electronic device 101 can provide content to the first display area 1010. When an event is received, the electronic device 101 can provide a notification object 1150 to the second display area 1020.

[0137] According to an embodiment, when two or more facial objects 1110 and 1120 are identified during the facial recognition process, the electronic device 101 can output a notification object 1150 to restrictively display content corresponding to the received event without performing biometric authentication. The notification object 1150 may include the event reception time, sender information, the number of received events, message content, etc. The restriction scope of the notification function may vary depending on the user's settings. For example, when two or more facial objects 1110 and 1120 are identified, the electronic device 101 may be configured to provide a notification function to display the objects after excluding at least one of the contents included in the notification object 1150.

[0138] According to an embodiment, when two or more facial objects 1110 and 1120 are identified during the facial recognition function, the electronic device 101 can change the position for displaying a notification object 1150 corresponding to the received event. For example, the electronic device 101 can extract the position of the user's facial object 1110 corresponding to the electronic device 101 among the two or more identified facial objects 1110 and 1120, and can display the notification object 1150 in an area corresponding to said position. For example, the area may be an area close to the user's facial object 1110.

[0139] Figure 12 The electronic device according to an embodiment is shown operating according to various notifications from an application.

[0140] According to various embodiments of this specification, electronic device 101 can provide notification functionality to display the content of notification objects differently for each application. For example, electronic device 101 can store security levels suitable for each application and can provide notification functionality in a restricted manner based on said security levels.

[0141] refer to Figure 12 Reference number 1201, according to an embodiment, electronic device 101 can receive instant messages and can provide notification functionality to display a first notification object 1210 corresponding to the message.

[0142] According to an embodiment, the first notification object 1210 may be displayed on the second display area 1020. The first notification object 1210 may include a first user interface 1220 (e.g., a user interface including "OK", "Reply", and "More" buttons). After successful facial recognition and / or biometric authentication, the electronic device 101 can provide the corresponding function by touching the buttons included in the first user interface 1220.

[0143] refer to Figure 12 According to reference numeral 1202, in an embodiment, electronic device 101 can provide a notification function, which includes a second notification object 1230 corresponding to an event received from a social networking service (SNS) application.

[0144] According to an embodiment, the second notification object 1230 may be displayed on the second display area 1020. The second notification object 1230 may include a second user interface 1240 (e.g., a user interface including "OK" and "Go to Application" buttons). After successful facial recognition and / or biometric authentication, the electronic device 101 can provide the corresponding function by touching the buttons included in the second user interface 1240.

[0145] Figure 13A flowchart illustrating a biometric authentication process 1300 for an electronic device according to various embodiments is shown.

[0146] refer to Figure 13 According to an embodiment, in operation 1305, the electronic device (e.g., Figure 1 The electronic device 101 can receive events and display notification information on a display. For example, the notification information may include content corresponding to the event (e.g., event reception time, sender information, number of events received, or message content).

[0147] In operation 1310, electronic device 101 can determine via sensor module (e.g., Figure 1 The sensor module 176 detects whether the folding state meets specified conditions. For example, specified conditions may mean that the folding angle of the electronic device 101 needs to be within 90 degrees to 130 degrees. This can be defined as a half-folding state that meets the specified conditions.

[0148] When the folded state of electronic device 101 in operation 1310 meets the specified conditions (e.g., operation 1310-Yes), the electronic device can perform operation 1315.

[0149] When the folded state of electronic device 101 does not meet the specified conditions in operation 1310 (e.g., operation 1310-No), electronic device 101 may continue to perform operation 1310.

[0150] According to an embodiment, in operation 1315, the electronic device 101 can use the camera module (e.g., ...) in a first mode. Figure 1 The camera module 180 acquires at least one first image. For example, the first mode may be an operating mode in which the camera module operates at a lower power than a second mode. The electronic device 101 may limit the operating time of the camera module. The operating time may be set by the user. For example, when the operating time of the camera module exceeds a specified time in the first mode, the electronic device 101 may allow the camera module to terminate its operation and may allow the display to be deactivated.

[0151] According to an embodiment, in operation 1320, electronic device 101 can determine the presence of a face based on at least one first image obtained. Electronic device 101 can search for facial objects from at least one first image. For example, when two or more facial objects are identified, electronic device 101 can determine that face recognition has failed. As another example, when two or more facial objects are identified, electronic device 101 can determine the face of a user corresponding to electronic device 101. In this case, electronic device 101 can extract the location of the user's facial object and can display a notification object in the area corresponding to said location. For example, the area corresponding to said location could be an area close to the user's facial object.

[0152] When it is determined in operation 1320 that a face exists in at least one first image (e.g., operation 1320-yes), the electronic device may perform operation 1325.

[0153] When it is determined in operation 1320 that a face is not present in at least one first image (e.g., operation 1320-No), the electronic device 101 may continue to perform operation 1315.

[0154] According to an embodiment, in operation 1325, electronic device 101 can use the camera module to acquire at least one second image in a second mode. For example, the second mode could be an operation mode in which the camera module operates at a higher power than in the first mode. Electronic device 101 can limit the operation time of the camera module. The operation time can be set by the user. When a face is determined to be present, electronic device 101 can display at least some content related to the event on a display (e.g., event reception time, sender information, number of received events, or message content).

[0155] According to an embodiment, in operation 1330, electronic device 101 can use at least one second image to perform a biometric authentication function. For example, the biometric authentication can be face-based biometric authentication. Electronic device 101 can read only one region from at least one second image. For example, electronic device 101 can perform face recognition by reading only one region (e.g., a region including a facial object). When biometric authentication fails with at least one second image, electronic device 101 can display a user interface on a display to provide guidance for successful biometric authentication. For example, the user interface can be provided in the form of images and / or text, which includes information for providing the user's location such that the face is within a specified distance range from electronic device 101. For example, this information can be displayed in a first display area (e.g., Figure 10 The first display area 1010) and / or the second display area (e.g., Figure 10 The user interface is provided in a pop-up format on the second display area (1020).

[0156] When biometric authentication is successful in operation 1330 (e.g., operation 1330 - Yes), the electronic device can perform operation 1335.

[0157] If biometric authentication fails in operation 1330 (e.g., operation 1330-No), electronic device 101 may continue to perform operation 1325.

[0158] According to an embodiment, in operation 1335, electronic device 101 can display content related to the received event on a display. For example, the event-related content may include the event reception time, sender information, the number of received events, and / or message content. In this case, electronic device 101 can output the event-related content to at least a portion of the second display area.

[0159] Figure 14 A flowchart illustrating the operation of an authentication process 1400 performed by an electronic device is shown conceptually according to various embodiments.

[0160] According to an embodiment, in operation 1405, the electronic device (e.g., Figure 1 The electronic device 101 can be in tabletop mode when folded at a specified angle (e.g., Figure 6a In desktop mode 602, primary authentication is performed. For example, primary authentication could be facial recognition (e.g., determining the presence of a face). When user authentication is provided in desktop mode 602, electronic device 101 may apply authentication algorithms different from those in other operating modes (e.g., mobile mode and / or tablet mode). For example, in desktop mode 602, the distance between electronic device 101 and the user may be relatively longer compared to other modes, and electronic device 101 may therefore apply authentication algorithms to increase the permissible range of facial recognition (e.g., the distance between the electronic device and the user or the angle of the face). When primary authentication is successful, electronic device 101 may perform operation 1410.

[0161] In operation 1410, electronic device 101 may provide limited functionality after initial unlocking. For example, when the electronic device receives an event, it may restrict at least a portion of the content corresponding to the event and display the restricted content on a display. As another example, the content of events received from applications with a specified security level or higher can be restricted by setting different security levels for each application. Applications requiring personal information may be set to a higher security level when running. The security level can be specified by the user.

[0162] In operation 1415, electronic device 101 can perform secondary authentication. For example, secondary authentication could be biometric authentication based on facial recognition. In this case, if biometric authentication fails, electronic device 101 can provide a user interface on a display including instructions for successful biometric authentication. After providing the user interface, electronic device 101 can re-perform secondary authentication by detecting changes in the user's facial composition using an image sensor or by detecting device movement using an accelerometer. When secondary authentication is successful, electronic device 101 can perform operation 1420.

[0163] In operation 1420, the electronic device 101 can provide additional functionality in a fully unlocked state. For example, it can display on a screen content that was restricted from display after the first unlock, corresponding to the received event. As another example, it can display information including content received from an application with a specified security level or higher.

[0164] According to various embodiments, a foldable electronic device may include: a first housing; a second housing; a foldable structure located between the first housing and the second housing; at least one sensor; a display; a camera capable of operating in a first mode for user identification or a second mode for user authentication with higher operating power than the first mode; a processor; and a memory operatively connected to the processor.

[0165] According to one embodiment, the memory may store one or more instructions, which, when executed, cause the processor to: display notification information of an event of the foldable electronic device on the display; acquire at least one first image using the camera in a first mode when the folding state of the foldable electronic device detected by the at least one sensor meets a specified condition; determine the presence of a face based on the at least one first image acquired by the camera; acquire at least one second image using the camera in a second mode when the presence of the face is determined in the at least one first image; perform facial recognition-based biometric authentication using at least a portion of the at least one second image; and display the content of the event of the foldable electronic device on the display when the biometric authentication is successful.

[0166] According to one embodiment, the one or more instructions, when executed, may cause the processor to: terminate the operation of the camera and disable the display when the camera's operation time exceeds a specified time in the first mode.

[0167] According to one embodiment, the one or more instructions may, when executed, cause the processor to continuously display the notification information of the event of the foldable electronic device on the display without performing the biometric authentication when multiple faces are present in the at least one first image.

[0168] According to one embodiment, the one or more instructions, when executed, may cause the processor to: extract the position of a user's face corresponding to the foldable electronic device from among the plurality of faces included in the at least one first image; and display the notification information of the event in an area on the display corresponding to the position.

[0169] According to one embodiment, the one or more instructions may, when executed, cause the processor to display a user interface on the display that provides guidance for the biometric authentication when the biometric authentication via the at least one second image fails.

[0170] According to one embodiment, the one or more instructions, when executed, may cause the processor to: display the user interface; and re-perform the biometric authentication based on detecting changes in the user's facial composition or detecting device movement using the at least one sensor.

[0171] According to one embodiment, the one or more instructions may, when executed, cause the processor to perform the biometric authentication by reading only a portion of the at least one second image.

[0172] According to one embodiment, the one or more instructions may, when executed, cause the processor to apply different facial recognition algorithms during the execution of biometric authentication, based on whether the folded state of the foldable electronic device, detected by using the at least one sensor, meets specified conditions.

[0173] According to one embodiment, the one or more instructions may, when executed, cause the processor to: when it is determined that the face exists in the at least one first image, display at least a portion of the content corresponding to the event on a display.

[0174] According to one embodiment, the one or more instructions may, when executed, cause the processor to: when the biometric authentication is successful, further include additional content from the content corresponding to the event, in addition to the previously displayed content, and display the additional content and the previously displayed content on a display.

[0175] According to various embodiments, a method for providing notification functionality to a foldable electronic device may include: displaying notification information of an event of the foldable electronic device on a display; determining whether a folded state of the foldable electronic device, detected by using at least one sensor, meets the specified conditions; when the specified conditions are met, acquiring at least one first image using a camera in a first mode provided for user identification; determining the presence of a face based on the at least one first image acquired using the camera; when the presence of the face in the at least one first image is determined, acquiring at least one second image using the camera in a second mode for user authentication; performing biometric authentication based on face recognition using at least a portion of the at least one second image; and when the biometric authentication is successful, displaying the content of the event of the foldable electronic device on the display.

[0176] According to one embodiment, obtaining the at least one first image may include: terminating the operation of the camera and disabling the display when the camera's operation time exceeds a specified time in the first mode.

[0177] According to one embodiment, determining the presence of the face based on the at least one first image obtained by using the camera may further include: when it is determined that multiple faces exist in the at least one first image, continuously displaying the notification information of the event of the foldable electronic device on the display without performing the biometric authentication.

[0178] According to one embodiment, determining the presence of the face based on the at least one first image obtained by using the camera may further include: extracting the position of the face corresponding to the user of the foldable electronic device from among the plurality of faces included in the at least one first image; and displaying the notification information of the event in an area on the display corresponding to the position.

[0179] According to one embodiment, performing face-based biometric authentication using at least a portion of the at least one second image may further include: displaying a user interface on the display to provide guidance for the biometric authentication when biometric authentication via the at least one second image fails.

[0180] According to one embodiment, performing facial recognition-based biometric authentication using at least a portion of the at least one second image may further include: displaying the user interface; and re-performing the biometric authentication based on detecting changes in the user's facial composition or detecting device movement using the at least one sensor.

[0181] According to one embodiment, performing face-based biometric authentication by using at least a portion of the at least one second image may further include performing biometric authentication by reading only a portion of the at least one second image.

[0182] According to one embodiment, performing face-based biometric authentication using at least a portion of the at least one second image may further include: applying different face recognition algorithms during the biometric authentication process based on whether the folding state of the foldable electronic device detected by using the at least one sensor meets specified conditions.

[0183] According to one embodiment, obtaining at least one second image using the camera in a second mode for user authentication when it is determined that a face exists in the at least one first image may further include: displaying at least a portion of content corresponding to the event on the display when obtaining the at least one second image.

[0184] According to one embodiment, the content of the event of displaying the foldable electronic device on the display when the biometric authentication is successful may include: displaying other content on the display besides the at least one portion.

Claims

1. A foldable electronic device, comprising: First outer shell; Second outer shell; A foldable structure is located between the first outer shell and the second outer shell; At least one sensor; monitor; The camera is capable of operating in a first mode provided for user identification or a second mode provided for user authentication; processor; as well as A memory operably connected to the processor, wherein the memory stores one or more instructions that, when executed, cause the processor to: The display shows notification information about events related to the foldable electronic device; Based on the fact that the folding state of the foldable electronic device detected by the at least one sensor meets a specified condition, at least one first image is obtained in the first mode using the camera at a first output rate. The presence of a face is determined based on the at least one first image obtained using the camera; Based on determining that the face exists in the at least one first image, at least one second image is obtained in the second mode using the camera at a second output rate higher than the first output rate; Perform facial recognition-based biometric authentication using the region corresponding to the face in the at least one second image; as well as Based on the successful biometric authentication, the content of the event of the foldable electronic device is displayed on the display.

2. The foldable electronic device according to claim 1, wherein, When the one or more instructions are executed, they cause the processor to: In the first mode, if the camera's operation time exceeds a specified time, the operation of the camera is terminated and the display is disabled.

3. The foldable electronic device according to claim 1, wherein, When the one or more instructions are executed, they cause the processor to: Based on the presence of multiple faces in the at least one first image, notification information about the event concerning the foldable electronic device is continuously displayed on the display without performing the biometric authentication.

4. The foldable electronic device according to claim 3, wherein, When the one or more instructions are executed, they cause the processor to: Extract the position of the face corresponding to the user of the foldable electronic device from among the plurality of faces included in the at least one first image; and The notification information about the event is displayed in the area corresponding to the location on the display.

5. The foldable electronic device according to claim 1, wherein, When the one or more instructions are executed, they cause the processor to: If the biometric authentication fails via the at least one second image, a user interface providing guidance for the biometric authentication is displayed on the screen.

6. The foldable electronic device according to claim 5, wherein, When the one or more instructions are executed, they cause the processor to: Display the user interface; and The biometric authentication is performed again based on the detection of changes in the user's facial composition or device movement using the at least one sensor.

7. The foldable electronic device according to claim 1, wherein, When the one or more instructions are executed, they cause the processor to: The biometric authentication is performed by reading only a portion of the at least one second image.

8. The foldable electronic device according to claim 1, wherein, When the one or more instructions are executed, they cause the processor to: During the biometric authentication process, different facial recognition algorithms are applied based on whether the folded state of the foldable electronic device, detected using the at least one sensor, meets specified conditions.

9. The foldable electronic device according to claim 1, wherein, When the one or more instructions are executed, they cause the processor to: Based on the determination that the face exists in the at least one first image, at least a portion of the content corresponding to the event is displayed on the display.

10. The foldable electronic device according to claim 9, wherein, When the one or more instructions are executed, they cause the processor to: Based on the successful biometric authentication, in addition to the previously displayed content, further additional content from the content corresponding to the event is included, and the additional content and the previously displayed content are displayed on the display.

11. The foldable electronic device according to claim 1, wherein, The second mode operates at a higher power than the first mode.

12. A method for providing notification functionality in a foldable electronic device, the method comprising: Display notification information about events related to the foldable electronic device on the display screen; Determine whether the folding state of the foldable electronic device, detected using at least one sensor, meets specified conditions; Based on satisfying the specified conditions, at least one first image is obtained using a camera at a first output rate in a first mode provided for user identification; The presence of a face is determined based on the at least one first image obtained using the camera; Based on determining that the face exists in the at least one first image, at least one second image is obtained using the camera in a second mode for user authentication at a second output rate higher than the first output rate; Perform facial recognition-based biometric authentication using the region corresponding to the face in the at least one second image; as well as Based on the successful biometric authentication, the content of the event of the foldable electronic device is displayed on the display.

13. The method of claim 12, wherein obtaining the at least one first image comprises: In the first mode, if the camera's operation time exceeds a specified time, the operation of the camera is terminated and the display is disabled.

14. The method according to claim 12, wherein, Determining the presence of the face based on at least one first image obtained using the camera includes: Based on the determination that multiple faces exist in the at least one first image, notification information about the event concerning the foldable electronic device is continuously displayed on the display without performing the biometric authentication.

15. The method according to claim 14, wherein, Determining the presence of the face based on the at least one first image obtained using the camera also includes: Extracting the position of the face corresponding to the user of the foldable electronic device from among the plurality of faces included in the at least one first image; and The notification information about the event is displayed in the area corresponding to the location on the display.

Images