Electronic device comprising fingerprint sensor and operation method thereof

Abstract

The present invention relates to an electronic device comprising a fingerprint sensor and an operation method thereof, and may comprise: acquiring a first fingerprint image through a camera; acquiring a reference fingerprint image; generating fingerprint information regarding a plurality of areas of the reference fingerprint image; displaying, on a display, a first visual object for guiding a fingerprint input; acquiring a second fingerprint image through the fingerprint sensor; on the basis of a cropped image, identifying, from among the plurality of areas, a first area corresponding to a location of the second fingerprint image; on the basis of whether or not an image corresponding to the second fingerprint image is included in fingerprint information regarding the first area, determining whether or not the second fingerprint image has been registered; updating a registered fingerprint image so that the registered fingerprint image includes the second fingerprint image; and controlling the display to display a second visual object obtained by updating at least a partial area of the first visual object on the basis of the first area.

Description

Electronic device including a fingerprint sensor and method of operating the same

[0001] Embodiments disclosed in this document relate to an electronic device including a fingerprint sensor and a method of operating the same.

[0002] As the use of electronic devices increases, they can be equipped with a variety of functions. As the functionality available on electronic devices diversifies, the security of information stored on them becomes increasingly important. To address this need, authentication technologies based on user biometric information are being developed. These biometric authentication technologies may involve, for example, acquiring biometric information from a user, such as fingerprints, irises, voice, face, or blood vessels, and comparing this information with previously registered user biometric information to determine whether the user is authenticated.

[0003] Among the biometric technologies exemplified above, fingerprint recognition technology is currently the most widely commercialized for various reasons, including convenience, security, and cost-effectiveness. Fingerprint recognition technology enhances the security of user devices and facilitates various application services, such as mobile payments.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0005] An electronic device according to an embodiment may include a camera. The electronic device may include a fingerprint sensor. The electronic device may include a display. The electronic device may include at least one processor. The electronic device may include a memory storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a first fingerprint image of the user through the camera. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a reference fingerprint image based on the first fingerprint image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate fingerprint information for a plurality of regions of the reference fingerprint image based on the reference fingerprint image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a first visual object for guiding a fingerprint input of the user through a display based on the reference fingerprint image and the fingerprint information. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a second fingerprint image of the user through the fingerprint sensor. The instructions, when executed by at least one processor individually or collectively, may cause the electronic device to identify a first region corresponding to a location of a second fingerprint image among a plurality of regions based on at least one crop image corresponding to the second fingerprint image among crop images acquired based on a reference fingerprint image. The instructions, when executed by at least one processor individually or collectively, may cause the electronic device to determine whether to register a second fingerprint image based on whether fingerprint information of the first region includes an image corresponding to the second fingerprint image.The instructions, when executed individually or collectively by at least one processor, may cause the electronic device to update the registered fingerprint image to include the second fingerprint image based on the determination result. The instructions, when executed individually or collectively by at least one processor, may cause the electronic device to control the display to display a second visual object that updates at least a portion of the first visual object based on the first region.

[0006] According to an embodiment, an operating method of an electronic device may include an operation of acquiring a first fingerprint image of a user through a camera. The operating method may include an operation of acquiring a reference fingerprint image based on the first fingerprint image. The operating method may include an operation of generating fingerprint information for a plurality of regions of the reference fingerprint image based on the reference fingerprint image. The operating method may include an operation of displaying a first visual object for guiding a fingerprint input of a user through a display based on the reference fingerprint image and the fingerprint information. The operating method may include an operation of acquiring a second fingerprint image of the user through a fingerprint sensor. The operating method may include an operation of identifying a first region corresponding to a location of the second fingerprint image among a plurality of regions based on at least one crop image corresponding to the second fingerprint image among crop images acquired based on the reference fingerprint image. The operating method may include an operation of determining whether the second fingerprint image is registered based on whether fingerprint information of the first region includes an image corresponding to the second fingerprint image. The operating method may include an operation of updating the registered fingerprint image so that the registered fingerprint image includes the second fingerprint image based on a result of the determination. The method of operation may include controlling the display to display a second visual object that updates at least a portion of a first visual object based on the first region.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0009] FIG. 3 is a diagram for explaining an operation of generating fingerprint information based on a fingerprint image acquired through a camera module according to one embodiment.

[0010] FIG. 4 is a drawing for explaining a UI for an operation of acquiring a fingerprint image through a camera module according to one embodiment.

[0011] FIG. 5 is a diagram for explaining an operation of changing a distorted portion of a fingerprint image using an artificial intelligence model according to one embodiment.

[0012] FIG. 6 is a drawing for explaining an operation of removing a deformed portion of a fingerprint image using an artificial intelligence model according to one embodiment.

[0013] FIG. 7 is a drawing illustrating a visual object for guiding a user's fingerprint input according to one embodiment.

[0014] FIG. 8 is a drawing for explaining a UI representing a visual object for guiding a user's fingerprint input according to one embodiment.

[0015] FIG. 9 is a diagram for explaining an operation for identifying an area corresponding to a fingerprint image acquired through a fingerprint sensor according to one embodiment.

[0016] FIG. 10 is a drawing for explaining a UI that represents a visual object indicating an area where fingerprint registration has been completed according to one embodiment.

[0017] FIG. 11 is a drawing for explaining a UI that guides a user's fingerprint input according to one embodiment.

[0018] FIG. 12 is a drawing for explaining a UI showing a visual object indicating a portion where fingerprint registration is completed according to one embodiment.

[0019] FIG. 13 is a drawing for explaining a UI showing a 3D guide for guiding a user's fingerprint input according to one embodiment.

[0020] FIG. 14 is a flowchart illustrating an operating method of an electronic device according to one embodiment.

[0021] FIG. 15 is a flowchart illustrating a process for completing fingerprint registration according to one embodiment.

[0022] Figure 16 is a flowchart illustrating a process for fingerprint registration according to one embodiment.

[0023] FIG. 17 is a diagram for explaining an operation of comparing a fingerprint image acquired through a fingerprint sensor and a fingerprint image acquired through a camera in a fingerprint authentication mode according to one embodiment.

[0024] FIG. 18 is a flowchart illustrating a process for fingerprint authentication according to one embodiment.

[0025] FIG. 19 is a conceptual diagram illustrating a concept of controlling functions related to fingerprint registration and fingerprint authentication in an electronic device according to one embodiment.

[0026] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0027] Hereinafter, various embodiments of the present invention will be described with reference to the attached drawings. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure are included.

[0028] An electronic device according to various embodiments of the present document may include, for example, at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop personal computer (PC), a laptop personal computer (PC), a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device. According to various embodiments, the wearable device may include at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD)), a fabric or clothing-integrated type (e.g., an electronic garment), a body-attached type (e.g., a skin pad or a tattoo), or a bio-implantable type (e.g., an implantable circuit).

[0029] In some embodiments, the electronic device may be a home appliance. The home appliance may include, for example, at least one of a television, a digital video disk (DVD) player, an audio device, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box, a game console, an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

[0030] In one embodiment, the electronic device may be any of various medical devices (e.g., various portable medical measuring devices (e.g., blood glucose meter, heart rate meter, blood pressure meter, or body temperature meter), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computer tomography (CT), camera, or ultrasound), navigation device, global navigation satellite system (GNSS), event data recorder (EDR), flight data recorder (FDR), automobile infotainment device, electronic equipment for ships (e.g., marine navigation device or gyrocompass), avionics, security device, head unit for vehicles, industrial or home robot, automatic teller's machine (ATM) of financial institution, point of sales (POS) of store, or internet of things device (e.g., light bulb, various sensors, electric or gas meter, sprinkler device, fire alarm, thermostat, It may include at least one of the following: a streetlight, a toaster, exercise equipment, a hot water tank, a heater, or a boiler.

[0031] According to one embodiment, the electronic device may include at least one of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., a water, electricity, gas, or radio wave measuring device). In various embodiments, the electronic device may be a combination of one or more of the various devices described above. The electronic device according to one embodiment may be a flexible electronic device. In addition, the electronic device according to the embodiment of the present document is not limited to the devices described above, and may include new electronic devices developed according to technological advancements.

[0032] Using fingerprints on recent mobile devices may require a fingerprint registration process and a fingerprint authentication process. In existing mobile devices equipped with fingerprint sensors, a template for the partial fingerprint input by the fingerprint sensor is stored during the fingerprint registration process. During the fingerprint authentication process, the template for the partial fingerprint input by the fingerprint sensor is compared with the template stored during the registration process to perform authentication.

[0033] One of the challenges with existing mobile devices that incorporate fingerprint sensors is that, during the registration process, only the percentage of registered fingerprints is provided to the user, making it difficult for the user to determine which portions of their fingerprint were input through the fingerprint sensor and which were not. Furthermore, if an unregistered template (e.g., the edge of the fingerprint) is input during subsequent authentication, fingerprint authentication may fail.

[0034] When new templates are included in the authentication process, an improved method for storing additional templates in the authentication template area to accommodate the new area may be provided. However, this may still result in unregistered areas, as users may have difficulty accurately identifying where their fingerprints were registered. Furthermore, storage limitations may result in previously registered templates being deleted.

[0035] To address the aforementioned problems, the present disclosure may provide a user interface guide that indicates which portion of a fingerprint is currently being input via a fingerprint sensor when a user registers a fingerprint on a mobile device. Accordingly, the user can recognize which portion of the fingerprint is currently being input during fingerprint registration and can change, update, and register the entire fingerprint template without missing any portions, significantly improving the success rate of subsequent fingerprint authentication.

[0036] In addition, the purpose of the present disclosure may be to store a new fingerprint template utilizing the entire fingerprint during the fingerprint authentication process to support additional registration of unregistered portions during the registration process, and to enable efficient storage and use of the template without overlapping areas.

[0037] An electronic device according to an embodiment may include a camera. The electronic device may include a fingerprint sensor. The electronic device may include a display. The electronic device may include at least one processor. The electronic device may include a memory storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a first fingerprint image of the user through the camera. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a reference fingerprint image based on the first fingerprint image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate fingerprint information for a plurality of regions of the reference fingerprint image based on the reference fingerprint image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a first visual object for guiding a fingerprint input of the user through a display based on the reference fingerprint image and the fingerprint information. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a second fingerprint image of the user through the fingerprint sensor. The instructions, when executed by at least one processor individually or collectively, may cause the electronic device to identify a first region corresponding to a location of a second fingerprint image among a plurality of regions based on at least one crop image corresponding to the second fingerprint image among crop images acquired based on a reference fingerprint image. The instructions, when executed by at least one processor individually or collectively, may cause the electronic device to determine whether to register a second fingerprint image based on whether fingerprint information of the first region includes an image corresponding to the second fingerprint image.The instructions, when executed individually or collectively by at least one processor, may cause the electronic device to update the registered fingerprint image to include the second fingerprint image based on the determination result. The instructions, when executed individually or collectively by at least one processor, may cause the electronic device to display a second visual object that updates at least a portion of the first visual object based on the first region.

[0038] Therefore, according to the present disclosure, by providing the user with a partial image of a fingerprint input via a fingerprint sensor from a full fingerprint image, the user can naturally recognize whether the previously input fingerprint has been properly registered and whether there are still portions to be registered. This allows the entire fingerprint to be registered without any missing portions, improving authentication results during fingerprint authentication.

[0039] Furthermore, according to the present disclosure, a user can confirm the location of a partially input fingerprint through the fingerprint sensor, allowing the user to input and register the entire fingerprint data even if the fingerprint sensor is reduced in size. This can have a positive effect on improving the authentication rate during future fingerprint authentication.

[0040] In addition, according to the present disclosure, by registering a partial fingerprint template based on a full fingerprint image during fingerprint registration and fingerprint authentication, duplicate templates can be minimized, thereby reducing storage usage.

[0041] In addition, according to the present disclosure, since the location of each partial template is known, the most suitable fingerprint template for each location can be selected, and the recognition rate can be improved by performing authentication using the template.

[0042] Below, with reference to FIG. 1, we specify and expand upon devices to which various embodiments disclosed in this document can be applied or expanded.

[0043] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0044] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0045] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0046] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0047] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0048] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0049] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0050] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0051] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0052] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0053] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0054] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0055] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0056] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0057] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0058] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0059] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi Direct (wireless fidelity direct), or IrDA (infrared data association)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0060] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0061] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197). According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., a bottom surface) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent to a second surface (e.g., a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0062] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0063] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0064] FIG. 2 is a block diagram illustrating the configuration of an electronic device (200) according to one embodiment. The electronic device (200) of FIG. 2 may be referenced by the electronic device (101) of FIG. 1.

[0065] Referring to FIG. 2, an electronic device (200) according to an embodiment may include a camera module (210) (e.g., the camera module (180) of FIG. 1), a sensor module (220) (e.g., the sensor module (176) of FIG. 1), a display (230) (e.g., the display module (160) of FIG. 1), and a processor (240) (e.g., the processor (1200) of FIG. 1). However, the components of the electronic device (200) are not limited thereto. For example, the electronic device (200) may omit at least one of the components described above, or may further include at least one other component. For example, the electronic device (200) may further include a memory (250) (e.g., memory (130) of FIG. 1), a touch screen (not shown), a battery (e.g., battery (189) of FIG. 1), a circuit board (not shown), and / or a communication module (e.g., communication module (190) of FIG. 1).

[0066] According to one embodiment, the electronic device (200) can capture a subject through the camera module (210). For example, the electronic device (200) can obtain an image through the camera module (210). For example, the electronic device (200) can obtain an image of a user's finger through the camera module (210). However, the present invention is not limited thereto. For example, the camera module (210) can also capture images and videos of the user's finger. In the present disclosure, the camera module (210) may be referred to as a camera.

[0067] According to one embodiment, the camera module (210) may include at least one lens assembly (not shown), at least one image sensor (not shown), at least one image stabilizer (not shown), at least one image signal processor (not shown), a memory, and / or at least one flash (not shown). In one embodiment, the at least one lens assembly may collect light emitted from a subject that is a target of image capturing. For example, the at least one lens assembly may collect light emitted from a user's finger (or, the user's fingerprint). The at least one lens assembly may include one or more lenses. According to one embodiment, the camera module (210) may include a plurality of lens assemblies. In this case, the camera module (210) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that differ from the lens properties of the other lens assemblies. A lens assembly may include, for example, a wide-angle lens or a telephoto lens.

[0068] In one embodiment, the flash may emit light that is used to enhance light emitted or reflected from a subject. In one embodiment, the flash may include one or more light-emitting diodes (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs), or a xenon lamp. The image sensor may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly into an electrical signal. In one embodiment, the image sensor may include one image sensor selected from among image sensors having different properties, such as, for example, an RGB sensor, a black and white (BW) sensor, an infrared (IR) sensor, or an ultraviolet (UV) sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0069] According to one embodiment, the image stabilizer may move at least one lens or image sensor included in the lens assembly in a specific direction or control the operating characteristics of the image sensor (e.g., adjust the read-out timing) in response to movement of the camera module (210) or the electronic device (200) including the same. This may compensate for at least some of the negative effects of the movement on the captured image. According to one embodiment, the image stabilizer may detect such movement of the camera module (210) or the electronic device (200) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (210). According to one embodiment, the image stabilizer may be implemented as, for example, an optical image stabilizer. The memory may at least temporarily store at least a portion of the image acquired through the image sensor for subsequent image processing operations. For example, when image acquisition is delayed due to a shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) may be stored in memory, and a corresponding copy image (e.g., a low-resolution image) may be previewed through the display (230). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory may be acquired and processed, for example, by an image signal processor. According to one embodiment, the memory may be configured as at least a portion of the memory (250), or as a separate memory that operates independently therefrom.

[0070] According to one embodiment, the image signal processor can perform one or more image processing operations on an image acquired through an image sensor or an image stored in a memory. The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor may perform control (e.g., exposure time control, or read-out timing control) on at least one of the components included in the camera module (210) (e.g., an image sensor). An image processed by the image signal processor may be stored back in memory for further processing or provided to an external component of the camera module (210) (e.g., a memory (250), a display (230), an electronic device (200), or a server). According to one embodiment, the image signal processor may be configured as at least a part of the processor (240), or may be configured as a separate processor that operates independently of the processor (240). When the image signal processor is configured as a separate processor from the processor (240), the image At least one image processed by the signal processor can be displayed through the display (230) as is or after additional image processing by the processor.

[0071] According to one embodiment, the electronic device (200) may include a plurality of camera modules (210), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (210) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (210) may be a front camera, and at least another may be a rear camera.

[0072] According to one embodiment, the camera module (210) may be a separate camera module (210) that operates independently from the camera included in the sensor module (220).

[0073] According to one embodiment, the sensor module (220) may include a fingerprint sensor. The electronic device (200) may obtain a user's fingerprint image through the fingerprint sensor. The fingerprint sensor may obtain the user's fingerprint information using an electrostatic method. For example, the fingerprint sensor may obtain the fingerprint by utilizing the principle that the part (ridge) where the fingerprint touches the electrode is detected, and the part (valley) where the fingerprint is not touched is not detected. However, the present invention is not limited thereto. For example, the fingerprint sensor may obtain the fingerprint information using at least one of an optical method or an ultrasonic method. For example, the optical method may capture the fingerprint image using a photosensitive diode to obtain the user's fingerprint. For example, the ultrasonic method may generate ultrasonic waves through a piezoelectric element and obtain the user's fingerprint by utilizing the path difference of the ultrasonic waves that hit and reflect off the ridges and valleys of the fingerprint.

[0074] According to one embodiment, the sensor array of the fingerprint sensor may be configured with a plurality of lines that receive fingerprint images. At least one processor (240) may check for overlap in images received by each line of the sensor array of the fingerprint sensor to calculate a fingerprint input speed. At least one processor (240) may extract an original fingerprint image by removing duplicate fingerprint images from each line.

[0075] According to one embodiment, the sensor module (220) may include a touch sensor (not shown) that detects contact by a part of the user's body and receives a touch input from the user, and / or a pressure sensor (not shown) that receives an input of the user pressing (or clicking) a button of the electronic device.

[0076] According to one embodiment, the electronic device (200) may include a touch screen (not shown). The touch screen may perform input and display functions. To this end, the touch screen may include a touch panel and a display (230).

[0077] According to one embodiment, the touch panel may be configured with a touch detection sensor of a capacitive overlay type, a resistive overlay type, an infrared beam detection type, or a pressure detection sensor. In addition to the above sensors, any type of sensor device capable of detecting contact or pressure of an object may be configured as the touch panel of the present embodiment. The touch panel may detect a user's touch input, generate a detection signal, and transmit the detection signal to the processor (240). The detection signal may include coordinate data of a touch input by the user. When the user inputs a touch position movement motion, the touch panel may generate a detection signal including coordinate data of a touch position movement path, and transmit the detection signal to the processor (240).

[0078] According to one embodiment, the display (230) may be formed of a liquid crystal display (LCD), organic light emitting diodes (OLED), or active matrix organic light emitting diodes (AMOLED), and may visually provide a user with various information, such as menus, input data, function setting information, and other information of the electronic device (200). For example, the display (430) may display a visual object to guide a user's fingerprint input.

[0079] According to one embodiment, the touch screen may include a fingerprint sensing area of ​​the fingerprint sensor in at least a portion or the entire area of ​​the touch panel or the display (230). For example, the fingerprint sensing area of ​​the fingerprint sensor may be arranged on the surface of a cover glass provided on the upper side to protect the display (230) through printing or etching. The fingerprint sensing area of ​​the fingerprint sensor may be arranged on the upper or lower side of the touch panel. The fingerprint sensing area of ​​the fingerprint sensor may be arranged within pixels of the touch panel or in a black masking area between pixels.

[0080] According to one embodiment, the electronic device (200) may include at least one processor (240). The at least one processor (240) may be electrically and / or operatively connected to various components of the electronic device (200). For example, the at least one processor (240) may be connected to a camera module (210), a sensor module (220), a display (230), and / or a memory (250).

[0081] According to one embodiment, at least one processor (240) may control the overall operation of the electronic device (200) and the signal flow between internal components, and may perform a function of processing data. The processor (240) may be composed of, for example, a central processing unit (CPU), an application processor, and a communication processor. The processor (240) may be formed as a single-core processor or a multi-core processor, and may be composed of a plurality of processors.

[0082] According to one embodiment, at least one processor (240) may acquire a fingerprint image of a user (e.g., a first fingerprint image) through the camera module (210). For example, in response to the fingerprint registration mode being executed, the at least one processor (240) may acquire a fingerprint image of the user through the camera module (210). In one example, the fingerprint image acquired through the camera module (210) may include a full fingerprint image. The at least one processor (240) may acquire a reference fingerprint image based on the first fingerprint image. For example, the at least one processor (240) may acquire the reference fingerprint image using a first artificial intelligence model trained to convert the fingerprint image acquired through the camera module (210) into a reference fingerprint image. In one example, the at least one processor (240) may acquire crop images based on the reference fingerprint image. The crop images may include a fingerprint partial image including a region of the fingerprint image.

[0083] According to one embodiment, at least one processor (240) may generate fingerprint information for a plurality of regions of a reference fingerprint image based on a reference fingerprint image. The fingerprint information for the plurality of regions of the reference fingerprint image may include fingerprint information for each of the plurality of regions. For example, at least one processor (240) may generate fingerprint information for each of a plurality of grid regions that divide the reference fingerprint image based on the reference fingerprint image. In one example, at least one processor (240) may generate a visual object to guide a user's fingerprint input using the fingerprint information. At least one processor (240) may display the generated visual object through the display (230).

[0084] According to one embodiment, at least one processor (240) may acquire a user's fingerprint image (e.g., a second fingerprint image) via a fingerprint sensor. In one example, the fingerprint image acquired via the fingerprint sensor may include a partial fingerprint image, but is not limited thereto.

[0085] According to one embodiment, at least one processor (240) may identify, based on crop images acquired based on at least one of the reference fingerprint images, an area corresponding to a location of a fingerprint image acquired through a fingerprint sensor among a plurality of areas of the reference fingerprint image. For example, at least one processor (240) may identify, based on at least one crop image corresponding to a fingerprint image acquired through the fingerprint sensor among the crop images, an area corresponding to a location of a fingerprint image acquired through the fingerprint sensor among a plurality of areas of the reference fingerprint image.

[0086] According to one embodiment, at least one processor (240) may determine whether a fingerprint image acquired through a fingerprint sensor is registered based on whether the fingerprint information of the identified area includes an image corresponding to the fingerprint image acquired through the fingerprint sensor. For example, at least one processor (240) may determine whether a fingerprint image acquired through the fingerprint sensor is included in a registered fingerprint image.

[0087] According to one embodiment, at least one processor (240) may update the registered fingerprint image so that the registered fingerprint image includes a fingerprint image acquired through the fingerprint sensor based on the determination result. For example, at least one processor (240) may update the registered fingerprint image so that the registered fingerprint image includes a fingerprint image acquired through the fingerprint sensor based on the determination result that the fingerprint information of the identified area does not include an image corresponding to a fingerprint image acquired through the fingerprint sensor (e.g., based on the fingerprint image acquired through the fingerprint sensor is not included in the registered fingerprint image).

[0088] According to one embodiment, at least one processor (240) may control the display (230) to display a visual object in which at least a portion of the visual object is updated based on the identified region. For example, at least one processor (240) may control the display (230) to display a visual object in which at least a portion of the visual object corresponding to the identified region is updated. For example, at least one processor (240) may control the display (230) to display a visual object in which at least a portion of the visual object corresponding to a region other than the identified region among a plurality of regions of the reference fingerprint image is updated. Accordingly, the user can easily know which region of the fingerprint is the region in which the fingerprint is registered.

[0089] According to one embodiment, the memory (250) performs the function of storing a program for processing and controlling the processor (240), an operating system (OS), various applications, and input / output data, and a program for controlling the overall operation of the electronic device (200) may be stored. The memory (250) may store a user interface (UI) provided by the electronic device (200) and various setting information required for processing functions in the electronic device (200).

[0090] According to one embodiment, the memory (250) may store various programs and data related to the fingerprint recognition function based on the fingerprint sensor of the electronic device (200). For example, the memory (250) may store a program for processing a function according to a line scan method or an area scan method of the fingerprint sensor, and data processed according to the program. The memory (250) may store a user's fingerprint (e.g., a user's fingerprint image) recognized through a touch method or a swipe method. The fingerprint image stored in the memory (250) may be used to later confirm whether a fingerprint recognized through the fingerprint sensor matches.

[0091] According to one embodiment, the electronic device (200) may include a battery (not shown). The battery may be electrically connected to components of the electronic device (200). The battery may supply power to components of the electronic device (200). For example, the battery may supply power to a camera module (210), a sensor module (220), and / or a display (230) of the electronic device (200).

[0092] According to one embodiment, the electronic device (200) may include a circuit board (e.g., a printed circuit board (PCB), a printed board assembly (PBA), a flexible PCB (FPCB), or a rigid-flexible PCB (RFPCB)) that accommodates components for driving the electronic device (200). For example, the circuit board (not shown) may include at least one integrated circuit chip. At least one of a processor (240), a memory (250), a power management module (e.g., a power management module (188) of FIG. 1), or a communication module (e.g., a communication module (190) of FIG. 1) may be provided in the integrated circuit chip.

[0093] In one embodiment, a circuit board (not shown) may be disposed within the electronic device (200). For example, the circuit board may include a first circuit board and a second circuit board. In one embodiment, a communication module may be mounted on the first circuit board, and a processor (240) may be mounted on the second circuit board. In one embodiment, the circuit board may be electrically connected to a battery via a power transmission structure. In one embodiment, the circuit board may be an interposer board.

[0094] According to one embodiment, the electronic device (200) may include a communication module (e.g., the communication module (190) of FIG. 1). The electronic device (200) may communicate with an external device using the communication module. For example, the electronic device (200) may transmit the user's fingerprint information and / or information related to the fingerprint information to an external device (e.g., the external electronic device (102, 104) of FIG. 1 or the server (108) of FIG. 1) or receive the user's fingerprint information from the external device using the communication module. For example, the electronic device (200) may transmit the acquired user's fingerprint information to the external device (102, 104, 108) or receive the user's fingerprint information from the external device. However, the present invention is not limited thereto.

[0095] FIG. 3 is a diagram for explaining an operation of generating fingerprint information based on a fingerprint image acquired through a camera module according to one embodiment.

[0096] The contents of Fig. 3 may be referenced by the contents of Figs. 1 and 2. The same terms and / or the same reference numerals are used for the same or substantially the same configurations as those described above.

[0097] Referring to FIG. 3, at least one processor (e.g., processor (240) of FIG. 2) may acquire a finger image (310) via a camera module (e.g., camera module (210) of FIG. 2). For example, the at least one processor may acquire a finger image (310) including a fingerprint image of the user. In one example, in response to the fingerprint registration mode being executed, the at least one processor may acquire the finger image (310) via the camera module in a secure execution environment (TEE). For example, the finger image (310) may be acquired via a secure camera. However, the present invention is not limited thereto. For example, the at least one processor may acquire the finger image (310) via the camera module in a physically separate secure environment (e.g., eSE or secure processor).

[0098] In one embodiment, at least one processor may acquire an image of a single finger of the user. However, this is not limited to this. For example, at least one processor may acquire images of multiple fingers of the user simultaneously via a camera module. For example, at least one processor may acquire fingerprint images of multiple fingers of the user simultaneously.

[0099] In one example, at least one processor may acquire multiple images or videos of a user's finger movements through a camera module. For example, at least one processor may acquire multiple images or videos of a user's finger taken from various angles. At least one processor may select at least one image or a captured image of a video from among the multiple images. For example, at least one processor may select an image that satisfies a specified condition (e.g., when the image has a sharpness equal to or greater than a reference value, or when the image includes a fingerprint) from among the multiple images, or a captured image that satisfies the specified condition from among the captured images of the video. In one example, at least one processor may generate an image to which the user's fingerprint is applied continuously through an image panorama function.

[0100] According to one embodiment, at least one processor may obtain a first fingerprint image (320) through a camera module. For example, the at least one processor may obtain a first fingerprint image (320) including a fingerprint portion based on a finger image (310) obtained through the camera module. For example, the at least one processor may obtain a first fingerprint image (320) that is a fingerprint region of the finger image (310). For example, the at least one processor may obtain the first fingerprint image (320) through a process of removing an area other than the fingerprint region from the finger image (310). In one example, the at least one processor may obtain the first fingerprint image (320) through a camera module in a secure execution environment (TEE) in response to the fingerprint registration mode being executed. For example, the first fingerprint image (320) may be obtained through a secure camera. However, the present invention is not limited thereto. For example, at least one processor may acquire a first image (320) through a camera module in a physically separated secure environment (e.g., an eSE or a secure processor). In one example, the operation of acquiring a fingerprint image (320) based on a finger image (310) may be omitted. For example, at least one processor may acquire a first fingerprint image (320) that includes only a fingerprint region of a finger through the camera module.

[0101] According to one embodiment, at least one processor may obtain a reference fingerprint image (330) based on a first fingerprint image (320). For example, the at least one processor may obtain the reference fingerprint image (330) using a first artificial intelligence model trained to convert the first fingerprint image (320) into a reference fingerprint image (330). For example, the at least one processor may obtain the reference fingerprint image (330) using a first artificial intelligence model trained to convert the first fingerprint image (320) into a reference fingerprint image (330) similar to a fingerprint image obtained through a fingerprint sensor. In one example, the at least one processor may obtain the reference fingerprint image (330) based on the first fingerprint image (320) in a secure execution environment (TEE). However, the present invention is not limited thereto. For example, at least one processor may obtain a reference fingerprint image (330) based on a first fingerprint image (320) in a physically separate secure environment (e.g., an eSE or secure processor).

[0102] According to one embodiment, at least one processor may generate fingerprint information (340) for a plurality of regions (341) of the reference fingerprint image (330) based on the reference fingerprint image (330). The fingerprint information (340) may include fingerprint information for each of the plurality of regions (341). In one example, the at least one processor may generate a plurality of grid regions for dividing the reference fingerprint image (330). The plurality of regions (341) may correspond to each of the plurality of grid regions for dividing the reference fingerprint image (330). For example, the fingerprint information (340) for each of the plurality of regions (341) may include fingerprint information for each of the grid regions. In one example, the fingerprint information (340) for the plurality of regions (341) of the reference fingerprint image (330) may include fingerprint information in the form of a map. For example, the fingerprint information (340) may include an image map or a grid map. In one example, at least one processor may generate fingerprint information (340) for multiple regions (341) of a reference fingerprint image (330) based on a reference fingerprint image (330) in a secure execution environment (TEE). However, the present invention is not limited thereto. For example, at least one processor may generate fingerprint information (340) for multiple regions (341) of a reference fingerprint image (330) based on a reference fingerprint image (330) in a physically separate secure environment (e.g., an eSE or a secure processor).

[0103] In one example, fingerprint information (340) for multiple regions (341) may include information regarding whether the region is a fingerprint region, location information of the region, size information of the region, shape information of the region, information regarding whether a fingerprint image (a fingerprint image acquired from a fingerprint sensor) corresponding to the region is registered, and / or quality information of a fingerprint image (a fingerprint image acquired from a fingerprint sensor) corresponding to the region. For example, referring to FIG. 3, fingerprint information (340) may include information indicating that a first region (351) is included in a fingerprint region, and information indicating that a second region (352) is included in a non-fingerprint region.

[0104] FIG. 4 is a drawing for explaining a UI for an operation of acquiring a fingerprint image through a camera module according to one embodiment.

[0105] The contents of FIG. 4 may be referenced by the contents of FIG. 1, FIG. 2, and / or FIG. 3. The same terms are used for components that are identical or substantially identical to those described above.

[0106] According to one embodiment, at least one processor (e.g., at least one processor (240) of FIG. 2) can guide a user's action to acquire a fingerprint image (e.g., a first fingerprint image (320) of FIG. 3) (or a finger image (e.g., a finger image (310) of FIG. 3)) via a camera module (e.g., a camera module (210) of FIG. 2). For example, referring to FIG. 4, the at least one processor can display a visual object through a display (e.g., a display (230) of FIG. 2) that guides the user's action to acquire a fingerprint image. For example, the at least one processor can display a visual object through the display that guides the user's fingerprint to be captured from various angles.

[0107] Referring to FIG. 4, for example, at least one processor may control the display to display a first visual object (311) that guides the front of the user's fingerprint to be captured, a second visual object (312) and a third visual object (313) that guide the side of the user's finger to be captured, and a fourth visual object (314) and a fifth visual object (315) that guide the top of the user's finger to be captured. However, the present invention is not limited thereto. For example, at least one processor may control the display to display a visual object that guides multiple fingers of the user to be captured at once. By capturing multiple fingers of the user at once, at least one processor may acquire images (e.g., fingerprint images) of multiple fingers of the user at once.

[0108] FIG. 5 is a diagram for explaining an operation of changing a distorted portion of a fingerprint image using an artificial intelligence model according to one embodiment.

[0109] The contents of FIG. 5 may be referenced by the contents of FIG. 1, FIG. 2, FIG. 3, and / or FIG. 4 described above. The same terms are used for configurations that are identical or substantially identical to those described above.

[0110] Referring to FIG. 5, at least one processor (e.g., at least one processor (240) of FIG. 2) may acquire a second reference fingerprint image (430b) using a second artificial intelligence model trained to change a distorted portion (421) of a first fingerprint image (420). For example, referring to FIG. 5 (a), the distorted portion (421) may include a curved portion of the finger, such as a side of the finger, an edge of the finger, and / or a top of the finger. In one example, the distorted portion (421) may include a portion in which the fingerprint gap is narrower than an actual fingerprint (or a fingerprint image acquired through a fingerprint sensor), or a fingerprint is captured as being more deformed. Below, the first artificial intelligence model and the second artificial intelligence model are described respectively, but the first artificial intelligence model and the second artificial intelligence model may be implemented as a single artificial intelligence model.

[0111] According to one embodiment, at least one processor may obtain a first reference fingerprint image (430a) using a first artificial intelligence model trained to transform a first fingerprint image (420) into a first reference fingerprint image (430a). Referring to (b) of FIG. 5, the first reference fingerprint image (430a) may include a first portion (431a) corresponding to a distorted portion (421) of the first fingerprint image (420).

[0112] According to one embodiment, at least one processor may obtain a second reference fingerprint image (430b) using a second artificial intelligence model trained to change a first portion (431a) of a first reference fingerprint image (430a). For example, the at least one processor may input a fingerprint image obtained through a camera module and a fingerprint image obtained through a fingerprint sensor into the second artificial intelligence model, generate a first parameter based on the fingerprint image obtained through the camera module and the fingerprint image obtained through the fingerprint sensor using the second artificial intelligence model, and obtain a second reference fingerprint image (430b) in which a first portion (431a) of the first reference fingerprint image (430a) is changed based on the first parameter.

[0113] Referring to (c) of FIG. 5, the second reference fingerprint image (430b) may include a second portion (431b) that is a modified first portion (431a) of the first reference fingerprint image (430a). For example, the first portion (431a) may include a distorted portion, and the second portion (431b) may not include a distorted portion by changing the distorted portion.

[0114] According to one embodiment, at least one processor acquires a second reference fingerprint image (430b) using a second artificial intelligence model trained to change a distorted portion (421) of a first fingerprint image (420), so that the second reference fingerprint image (430b) can correspond to a fingerprint image acquired through a fingerprint sensor. For example, a second portion (431b) of the second reference fingerprint image (430b) can have a fingerprint shape (e.g., an undistorted fingerprint shape) similar to a fingerprint shape of a portion of a fingerprint image acquired through a fingerprint sensor that corresponds to the second portion (431b).

[0115] FIG. 6 is a drawing for explaining an operation of removing a deformed portion of a fingerprint image using an artificial intelligence model according to one embodiment.

[0116] The contents of FIG. 6 may be referenced by the contents of FIG. 1, FIG. 2, FIG. 3, FIG. 4, and / or FIG. 5 described above. The same terms are used for configurations that are identical or substantially identical to those described above.

[0117] Referring to FIG. 6, at least one processor (for example, at least one processor (240) of FIG. 2) may acquire a fourth reference fingerprint image (530b) using a third artificial intelligence model trained to remove (or change) a deformed portion (521) of a first fingerprint image (520). For example, referring to FIG. 6 (a), the deformed portion (521) may include a wrinkled portion of a finger. In one example, the deformed portion (521) may include a portion in which the fingerprint is captured as being more deformed than an actual fingerprint (or a fingerprint image acquired through a fingerprint sensor) due to the wrinkling of the finger. Below, the first artificial intelligence model and the third artificial intelligence model are described respectively, but the first artificial intelligence model and the third artificial intelligence model may be implemented as a single artificial intelligence model. In one example, at least two of the first artificial intelligence model, the second artificial intelligence model (for example, the second artificial intelligence model of FIG. 5), or the third artificial intelligence model may be implemented as a single artificial intelligence model.

[0118] According to one embodiment, at least one processor may obtain a third reference fingerprint image (530a) using a first artificial intelligence model trained to convert a first fingerprint image (520) into a third reference fingerprint image (530a). For example, the at least one processor may input a fingerprint image obtained through a camera module and a fingerprint image obtained through a fingerprint sensor into the third artificial intelligence model, generate a second parameter based on the fingerprint image obtained through the camera module and the fingerprint image obtained through the fingerprint sensor using the third artificial intelligence model, and obtain a fourth reference fingerprint image (530b) in which a third portion (531a) of the third reference fingerprint image (530a) is removed (or changed) based on the second parameter. Referring to FIG. 6 (b), the third reference fingerprint image (530a) may include a third portion (531a) corresponding to a deformed portion (521) of the first fingerprint image (520).

[0119] According to one embodiment, at least one processor may obtain a fourth reference fingerprint image (530b) using a third artificial intelligence model trained to remove (or change) a third portion (531a) of a third reference fingerprint image (530a). Referring to (c) of FIG. 6, the fourth reference fingerprint image (530b) may include a fourth portion (531b) from which the third portion (531a) of the third reference fingerprint image (530a) is removed (or changed). For example, the third portion (531a) may include a deformed portion, and the fourth portion (531b) may not include a deformed portion by removing (or changing) the deformed portion.

[0120] According to one embodiment, at least one processor acquires a fourth reference fingerprint image (530b) using a third artificial intelligence model trained to remove (or change) a deformed portion (521) of a first fingerprint image (520), so that the fourth reference fingerprint image (530b) can correspond to a fingerprint image acquired through a fingerprint sensor. For example, a fourth portion (531b) of the fourth reference fingerprint image (530b) can have a fingerprint shape (e.g., an undistorted fingerprint shape) similar to a fingerprint shape of a portion of a fingerprint image acquired through a fingerprint sensor that corresponds to the fourth portion (531b).

[0121] According to one embodiment, at least one processor may acquire a fifth reference fingerprint image (e.g., a reference fingerprint image obtained through a camera module, in which a distorted portion of the fingerprint image is changed and a deformed portion of the fingerprint image is removed (or changed)) using a fourth artificial intelligence model trained to merge a second reference fingerprint image obtained using a second artificial intelligence model (e.g., the second reference fingerprint image (431b) of FIG. 5) and a fourth reference fingerprint image (531b) obtained using a third artificial intelligence model. For example, the at least one processor may input the second reference fingerprint image (431b) and the fourth reference fingerprint image (531b) into the fourth artificial intelligence model, generate a third parameter based on the second reference fingerprint image (431b) and the fourth reference fingerprint image (531b) using the fourth artificial intelligence model, and acquire a fifth reference fingerprint image obtained through the camera module, in which a distorted portion of the fingerprint image is changed and a deformed portion is removed (or changed) based on the third parameter.

[0122] According to one embodiment, at least one processor may generate fingerprint information for a plurality of regions of the fifth reference fingerprint image based on the fifth reference fingerprint image. According to one embodiment, at least one processor may obtain crop images based on the fifth reference fingerprint image.

[0123] FIG. 7 is a diagram illustrating a visual object for guiding a user's fingerprint input according to one embodiment. FIG. 8 is a diagram illustrating a UI representing a visual object for guiding a user's fingerprint input according to one embodiment.

[0124] The contents of FIGS. 7 and 8 may be referenced by the contents of FIGS. 1, 2, 3, 4, 5, and / or 6. The same terms are used for the same or substantially the same configurations as those described above.

[0125] According to one embodiment, at least one processor (e.g., processor (240) of FIG. 2) may display a first visual object (620) for guiding a user's fingerprint input through a display based on a generated reference fingerprint image (e.g., reference fingerprint image (330) of FIG. 3, a second reference fingerprint image (430b) of FIG. 5, a fourth reference fingerprint image (530b) or a fifth reference fingerprint image of FIG. 6) and fingerprint information for a plurality of areas of the reference fingerprint image (e.g., fingerprint information (340) of FIG. 3).

[0126] Hereinafter, with reference to FIG. 7, an operation of generating a guide screen displaying a first visual object (620) based on the reference fingerprint image and fingerprint information for a plurality of areas of the reference fingerprint image will be described.

[0127] According to one embodiment, at least one processor may generate an image (610) including a grid area having a size substantially the same as the size of the reference fingerprint image based on size information of the reference fingerprint image. At least one processor may generate a visual object (620) corresponding to the size of the image (610).

[0128] According to one embodiment, at least one processor may generate an image (610) including a grid area having a shape substantially identical to the shape of the reference fingerprint image based on shape information of the reference fingerprint image. At least one processor may generate a visual object (620) corresponding to the shape of the image (610).

[0129] Accordingly, the visual object (620) may vary depending on the user. For example, the size and / or shape of the visual object (620) may vary depending on the user.

[0130] According to one embodiment, the visual object (620) may include an actual fingerprint image of the user and / or an image in graphic form.

[0131] Referring to FIG. 8, at least one processor may display at least one visual object on the screen (600) to guide a user's fingerprint input. For example, at least one processor may display a first visual object (620) on the screen (600). However, the present invention is not limited thereto. For example, at least one processor may display a second visual object (630) indicating a rate at which the user's fingerprint is registered and / or a third visual object (640) including text to guide the user's fingerprint input on the screen (600).

[0132] According to one embodiment, the electronic device provides the user with a first visual object (620), thereby allowing the user to determine which part of the user's fingerprint has been registered.

[0133] FIG. 9 is a diagram for explaining an operation for identifying an area corresponding to a fingerprint image acquired through a fingerprint sensor according to one embodiment.

[0134] The contents of Fig. 9 may be referenced by the contents of Figs. 1, 2, 3, 4, 5, 6, 7 and / or 8 described above. The same terms are used for the same or substantially the same configuration as described above.

[0135] Referring to (a) of FIG. 9, according to one embodiment, at least one processor (e.g., processor (240) of FIG. 2) may obtain crop images (730) based on a reference fingerprint image (e.g., reference fingerprint image (330) of FIG. 3, second reference fingerprint image (430b) of FIG. 5, fourth reference fingerprint image (530b) of FIG. 6, or fifth reference fingerprint image). In one example, each of the crop images (730) may have different positions and / or sizes in the reference fingerprint image.

[0136] According to one embodiment, at least one process may list at least one crop image included in the crop images (730). For example, referring to (b) of FIG. 9, the crop images (730) may include a first crop image (731), a second crop image (732), a third crop image (733), and a fourth crop image (734). However, the present invention is not limited thereto. For example, the crop images (730) may omit at least one of the crop images described above, or may further include at least one crop image.

[0137] According to one embodiment, at least one processor may compare the cropped images with a fingerprint image (e.g., a second fingerprint image) acquired through the fingerprint sensor. For example, the at least one processor may determine whether the fingerprint image (710) acquired through the fingerprint sensor corresponds to at least one cropped image among the cropped images (730) (e.g., a first cropped image (731), a second cropped image (732), a third cropped image (733), and a fourth cropped image (734)). For example, referring to (c) of FIG. 9, the at least one processor may compare a fingerprint pattern of at least one cropped image (e.g., a second cropped image (732)) included in the cropped images (730) with a fingerprint pattern included in the fingerprint image (710) acquired through the fingerprint sensor.

[0138] According to one embodiment, in response to determining that a fingerprint image (710) acquired through the fingerprint sensor corresponds to at least one crop image among the crop images, at least one processor may obtain information about the fingerprint image (710) acquired through the fingerprint sensor based on a crop image corresponding to the fingerprint image (710) acquired through the fingerprint sensor. For example, the at least one processor may obtain position information, size information, and / or rotation angle information of the fingerprint image (710) acquired through the fingerprint sensor based on a second crop image (732) corresponding to the fingerprint image (710) acquired through the fingerprint sensor. For example, the at least one processor may compare the fingerprint image (710) with the second crop image (732). Based on the comparison result, the at least one process may obtain size information of a rotation angle of the fingerprint image (710) based on the second crop image (732).

[0139] According to one embodiment, at least one processor may identify an area corresponding to a location of a fingerprint image (710) acquired through the fingerprint sensor, among a plurality of areas (741) of a reference fingerprint image, based on location information, size information, and / or rotation angle information of the fingerprint image (710) acquired through the fingerprint sensor. For example, referring to (d) of FIG. 9, at least one processor may identify a first area (810) corresponding to a location of the fingerprint image (710) among a plurality of areas (741) based on information about the fingerprint image (710). In one example, at least one processor may perform an operation to identify an area corresponding to a fingerprint image (710) acquired through the fingerprint sensor in a secure execution environment (TEE). However, the present invention is not limited thereto. For example, at least one processor may perform an operation to identify an area corresponding to a fingerprint image (710) acquired through a fingerprint sensor in a physically separated secure environment (e.g., an eSE or a secure processor).

[0140] According to one embodiment, at least one processor may determine whether a fingerprint image (710) acquired through a fingerprint sensor is registered based on whether the fingerprint information of the identified first region (810) among the fingerprint information (740) for the plurality of regions (741) includes an image corresponding to the fingerprint image (710) acquired through the fingerprint sensor. For example, the at least one processor may determine that the fingerprint image (710) is not registered based on whether the fingerprint information of the first region (810) does not include an image corresponding to the fingerprint image (710). For example, the at least one processor may determine that the fingerprint image (710) is registered based on whether it is included in the fingerprint information of the first region (810).

[0141] According to one embodiment, at least one processor may register a fingerprint image (710) based on the determination result. For example, at least one processor may update the registered fingerprint image so that the registered fingerprint image includes the fingerprint image (710) based on the determination result that the fingerprint image (710) is not registered.

[0142] According to one embodiment, at least one processor may store fingerprint data regarding a fingerprint image (710) in a memory (e.g., memory (250) of FIG. 2) based on the determination result.

[0143] According to one embodiment, at least one processor may control the display to display a visual object that updates at least a portion of a visual object based on the first region (810). For example, the at least one processor may control the display to display a visual object that updates at least a portion of a visual object (e.g., the first visual object (620) of FIG. 8) based on the first region (810) corresponding to a location of a fingerprint image (710) included in a registered fingerprint image.

[0144] The contents of displaying updated visual objects according to one embodiment will be described in detail later in FIGS. 10 to 12.

[0145] FIG. 10 is a drawing for explaining a UI showing a visual object (920) that displays an area (1010) in which fingerprint registration is completed according to one embodiment.

[0146] The contents of Fig. 10 may be referenced by the contents of Figs. 1, 2, 3, 4, 5, 6, 7, 8 and / or 9 described above. At least the same terminology is used for the same or substantially the same configuration as described above.

[0147] Referring to FIGS. 9 and 10 , according to one embodiment, at least one processor (e.g., processor (240) of FIG. 2 ) may control a display to display a visual object that updates at least a portion of a visual object based on a first region (810). For example, at least one processor may display a visual object (920) that updates a first region (1010) of a visual object based on a first region (810) corresponding to a location of a fingerprint image (710) included in a registered fingerprint image on a screen (911). Accordingly, a user may confirm the first region (1010) of the visual object as a registered fingerprint region among the user's fingerprints. However, the present invention is not limited thereto. For example, although not illustrated in FIG. 10, at least one processor may display on the screen an updated visual object in which at least a portion of a visual object corresponding to a second region other than the first region (1010) among a plurality of regions (e.g., a plurality of grid regions) is updated based on a first region (810) corresponding to a location of a fingerprint image (710) included in a registered fingerprint image. Accordingly, the user may identify the second region other than the first region (1010) of the visual object as an unregistered fingerprint region among the user's fingerprints.

[0148] FIG. 11 is a drawing for explaining a UI that guides a user's fingerprint input according to one embodiment.

[0149] The contents of Fig. 11 may be referenced by the contents of Figs. 1, 2, 3, 4, 5, 6, 7, 8, 9 and / or 10 described above. At least the same terminology is used for the same or substantially the same configuration as described above.

[0150] Referring to FIGS. 9 and 10, at least one processor (e.g., processor (240) of FIG. 2) may update the registered fingerprint image to include the fingerprint image (710) based on a determination that the fingerprint image (710) is unregistered.

[0151] According to one embodiment, at least one processor can display a visual object (920) on a screen (911) that updates a first area (1010) of a visual object based on a first area (810) corresponding to a location of a fingerprint image (710) included in a registered fingerprint image.

[0152] Referring to FIGS. 9 and 11 , according to one embodiment, at least one processor may display a visual object (1122) including text guiding the user to re-enter a fingerprint on the screen (1112) based on a determination that a fingerprint image (710) is registered. However, the present invention is not limited thereto. For example, at least one processor may display a visual object (1122) including text guiding the user to re-enter a fingerprint on the screen (1112) based on a determination that the user's fingerprint input does not satisfy a specified condition. For example, the at least one processor may display a visual object (1122) including text guiding the user to re-enter a fingerprint on the screen (1112) when the quality of the user's fingerprint input is lower than a reference value, when the degree of touch input for the user's fingerprint input is lower than a reference value, and / or when an error occurs in the fingerprint input or fingerprint acquisition.

[0153] According to one embodiment, at least one processor may, based on a determination that a user's fingerprint input does not satisfy a specified condition, display a visual object (1120) on a screen (1111) with a first region (1121) of the visual object updated based on the user's fingerprint input region that does not satisfy the specified condition. Accordingly, the user may confirm the first region (1121) of the visual object (1120) as an region where the user's fingerprint input does not satisfy the specified condition.

[0154] FIG. 12 is a drawing for explaining a UI showing a visual object indicating a portion where fingerprint registration is completed according to one embodiment.

[0155] The contents of Fig. 12 may be referenced by the contents of Figs. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 and / or 11 described above. At least the same terminology is used for the same or substantially the same configuration as described above.

[0156] Referring to FIG. 12, at least one processor (e.g., processor (240) of FIG. 2) can display a second visual object (1220) that updates a first area (1221) of a first visual object on a first screen (1211) based on an area corresponding to a location of a fingerprint image (fingerprint image acquired from a fingerprint sensor) included in a registered fingerprint image.

[0157] According to one embodiment, at least one processor may additionally acquire a fingerprint image of a user through a fingerprint sensor at least once. According to one embodiment, at least one processor may display a third visual object (1230) that further updates at least one area (e.g., the second area (1222) and / or the third area (1223)) of a second visual object on the second screen (1212) based on at least one area corresponding to a location of an additionally acquired fingerprint image (a fingerprint image additionally acquired from the fingerprint sensor) included in a registered fingerprint image.

[0158] According to one embodiment, at least one processor may display, on the screen (1212), a fourth visual object (1240) indicating a ratio of a user's fingerprint to be registered and / or a fifth visual object (1250) including text for guiding a user's fingerprint input. In one example, the fourth visual object (1240) may include a ratio of a fingerprint image acquired (input) through a fingerprint sensor to a fingerprint image acquired through a camera that matches. In one example, the fifth visual object (1250) may include text for guiding a user's fingerprint input (e.g., fingerprint re-entry) based on a ratio of a fingerprint to be registered (e.g., a ratio of fingerprint images included in registered fingerprint images). For example, the at least one processor may display the fifth visual object (1250) including text for guiding a user's fingerprint input (e.g., fingerprint re-entry) until the ratio of a fingerprint to be registered is greater than or equal to a threshold value (e.g., until the ratio of fingerprint images included in registered fingerprint images is greater than or equal to a threshold value).

[0159] FIG. 13 is a drawing for explaining a UI showing a 3D guide for guiding a user's fingerprint input according to one embodiment.

[0160] The contents of Fig. 13 may be referenced by the contents of Figs. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 and / or 12 described above. At least the same terms were used for the same or substantially the same configurations as described above.

[0161] According to one embodiment, at least one processor (e.g., processor (240) of FIG. 2) may display a visual object including a 3D model of a finger via a display. For example, the 3D model of the finger may display an area where a fingerprint is registered and / or an area where a fingerprint is not registered.

[0162] Referring to FIG. 13, according to one embodiment, at least one processor may display a visual object (1300) that guides a fingerprint input method through a display. For example, at least one processor may display a visual object (1300) that guides a fingerprint input method for an area where a fingerprint is not registered and / or an area where a fingerprint is not input through a display. For example, at least one processor may display a visual object that includes information about a user's fingerprint input posture for an area where a fingerprint is not registered and / or an area where a fingerprint is not input, and / or information about a fingerprint input location for an area where a fingerprint is not registered and / or an area where a fingerprint is not input. In one example, the visual object (1300) may include at least one of text, 3D, or video formats, but is not limited thereto.

[0163] FIG. 14 is a flowchart illustrating an operating method of an electronic device according to one embodiment.

[0164] The contents of FIG. 14 may be referenced by the contents of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7, FIG. 8, FIG. 9, FIG. 10, FIG. 11, FIG. 12 and / or FIG. 13 described above.

[0165] Referring to FIG. 14, an operating method (1400) of an electronic device (for example, the electronic device (200) of FIG. 2) according to an embodiment may include an operation (1410) of obtaining a first fingerprint image through a camera, an operation (1420) of obtaining a reference fingerprint image based on the first fingerprint image, an operation (1430) of generating fingerprint information based on the reference fingerprint image, an operation (1440) of displaying a first visual object, an operation (1450) of obtaining a second fingerprint image through a fingerprint sensor, an operation (1460) of identifying an area corresponding to a location of the second fingerprint image, an operation (1470) of determining whether the second fingerprint image is registered, an operation (1480) of updating the registered fingerprint image based on the determination result, and an operation (1490) of displaying a second visual object. However, the operating method (1400) of the electronic device is not limited thereto. For example, the operating method (1400) of the electronic device may omit at least one operation or may further include at least one operation. The method of operating the electronic device (1400) is not limited to the sequence of operations described above.

[0166] According to one embodiment, in operation 1410, at least one processor (e.g., at least one processor (240) of FIG. 2) may acquire a first fingerprint image through a camera module (e.g., the camera module (210) of FIG. 2). In one example, the at least one processor may acquire the first fingerprint image based on the finger image acquired through the camera module. Operation 1410 may be referenced by the contents regarding the finger image (310) and the first fingerprint image (320) of FIG. 3. According to one embodiment, in operation 1410, the at least one processor may acquire the first fingerprint image through the camera module in a secure execution environment (TEE). For example, the first fingerprint image may be acquired through a secure camera. However, the present invention is not limited thereto. For example, in operation 1410, at least one processor may acquire a first image via a camera module in a physically separated secure environment (e.g., an eSE or secure processor).

[0167] According to one embodiment, in operation 1420, at least one processor may obtain a reference fingerprint image based on the first fingerprint image. For example, the at least one processor may obtain the reference fingerprint image using a first artificial intelligence model trained to convert the first fingerprint image into a reference fingerprint image. According to one embodiment, the at least one processor may obtain the reference fingerprint image using a second artificial intelligence model trained to change a distorted portion of the first fingerprint image and / or a third artificial intelligence model trained to remove a deformed portion of the first fingerprint image. Operation 1420 may be referenced by the contents of FIG. 3 regarding the first fingerprint image (320) and the reference fingerprint image (330), and the contents of FIGS. 5 and 6.

[0168] According to one embodiment, in operation 1420, at least one processor may obtain a reference fingerprint image based on a first fingerprint image in a secure execution environment (TEE). For example, the reference fingerprint image may be obtained through a secure camera, but the present invention is not limited thereto. For example, in operation 1420, at least one processor may obtain a reference fingerprint image based on the first fingerprint image in a physically separate secure environment (e.g., an eSE or a secure processor).

[0169] According to one embodiment, in operation 1430, at least one processor may generate fingerprint information based on a reference fingerprint image. For example, at least one processor may generate fingerprint information for a plurality of regions of the reference fingerprint image based on the reference fingerprint image. Operation 1430 may be referenced by reference fingerprint image (330) of FIG. 3 and fingerprint information (340) for a plurality of regions (341) of the reference fingerprint image (330).

[0170] According to one embodiment, in operation 1430, at least one processor may generate fingerprint information based on a reference fingerprint image in a secure execution environment (TEE). However, the present invention is not limited thereto. For example, in operation 1420, at least one processor may generate fingerprint information based on a reference fingerprint image in a physically separate secure environment (e.g., an eSE or a secure processor). According to one embodiment, the at least one processor may transmit the generated fingerprint information to a rich execution environment (REE). For example, the at least one processor may transmit information regarding the size of a map (e.g., an image map) including a plurality of grid regions that divide the reference fingerprint image (330) to the REE.

[0171] According to one embodiment, in operation 1440, at least one processor may display a first visual object via a display. For example, at least one processor may generate a first visual object to guide a user's fingerprint input based on a reference fingerprint image and fingerprint information. At least one processor may display the generated first visual object via a display. Operation 1440 may be referenced by the contents of FIGS. 7 and 8 .

[0172] According to one embodiment, in operation 1440, at least one processor may generate a first visual object for guiding a user's fingerprint input based on a reference fingerprint image and fingerprint information in a rich execution environment (REE). At least one processor may display the generated first visual object through a display in the rich execution environment.

[0173] In one embodiment, at step 1450, at least one processor may acquire a second fingerprint image through a fingerprint sensor. For example, the at least one processor may acquire a second fingerprint image through the fingerprint sensor that is distinct from a first fingerprint image acquired through a camera. In one example, the first fingerprint image acquired through the camera may include a full fingerprint image, and the second fingerprint image acquired through the fingerprint sensor may include a partial fingerprint image. However, the present invention is not limited thereto.

[0174] According to one embodiment, in operation 1450, at least one processor may obtain a second fingerprint image through a fingerprint sensor in a secure execution environment (TEE). However, the present invention is not limited thereto. For example, in operation 1450, at least one processor may obtain the second fingerprint image through a fingerprint sensor in a physically separated secure environment (e.g., an eSE or a secure processor). According to one embodiment, in operation 1460, at least one processor may identify an area corresponding to a location of the second fingerprint image. For example, the at least one processor may identify a first area corresponding to a location of the second fingerprint image among a plurality of areas of a reference fingerprint image based on at least one crop image corresponding to the second fingerprint image. Operation 1460 may be referred to by the contents of FIG. 9.

[0175] In one embodiment, at operation 1460, at least one processor may identify an area corresponding to a location of a second fingerprint image in a secure execution environment (TEE), but is not limited thereto. For example, at operation 1460, at least one processor may identify an area corresponding to a location of a second fingerprint image in a physically separate secure environment (e.g., an eSE or a secure processor).

[0176] According to one embodiment, in operation 1470, at least one processor may determine whether a second fingerprint image is registered. For example, at least one processor may determine whether a second fingerprint image is registered based on whether the fingerprint information in the first region includes an image corresponding to the second fingerprint image.

[0177] According to one embodiment, in operation 1470, at least one processor may determine whether to register a second fingerprint image in a secure execution environment (TEE), but is not limited thereto. For example, in operation 1470, at least one processor may determine whether to register a second fingerprint image in a physically separate secure environment (e.g., an eSE or a secure processor).

[0178] According to one embodiment, in operation 1480, at least one processor may update the registered fingerprint image to include the second fingerprint image based on the determination result of operation 1470. For example, the at least one processor may update the registered fingerprint image to include the second fingerprint image based on the fingerprint information of the first region not including an image corresponding to the second fingerprint image.

[0179] In one embodiment, at operation 1480, at least one processor may update the enrolled fingerprint image to include a second fingerprint image in a secure execution environment (TEE), but is not limited thereto. For example, at operation 1480, at least one processor may update the enrolled fingerprint image to include a second fingerprint image in a physically separate secure environment (e.g., an eSE or a secure processor).

[0180] According to one embodiment, in operation 1480, at least one processor may store fingerprint data regarding a second fingerprint image in a memory (e.g., memory (250) of FIG. 2) based on the determination result of operation 1470. For example, at least one processor may store fingerprint data regarding a second fingerprint image in the memory based on the fingerprint information of the first region not including an image corresponding to the second fingerprint image. Operations 1470 and 1480 may be referenced by the contents of FIG. 9.

[0181] According to one embodiment, in operation 1490, at least one processor may control the display to display a second visual object. For example, the at least one processor may control the display to display a second visual object that updates at least a portion of the first visual object based on the first region. In one example, the at least one processor may control the display to display a second visual object in which a portion of the first visual object corresponding to the first region is updated. In one example, the at least one processor may also control the display to display a second visual object in which a portion of the first visual object corresponding to a second region excluding the first region among a plurality of regions of the reference fingerprint image is updated. Operation 1490 may be referred to by the contents of FIGS. 9 and 10 .

[0182] In one embodiment, at operation 1490, at least one processor may control a display to display a second visual object in a rich execution environment (REE).

[0183] FIG. 15 is a flowchart illustrating a process for completing fingerprint registration according to one embodiment.

[0184] The contents of FIG. 15 may be referenced by the contents of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7, FIG. 8, FIG. 9, FIG. 10, FIG. 11, FIG. 12, FIG. 13 and / or FIG. 14 described above.

[0185] Referring to FIG. 15, a process (1500) for completing fingerprint registration of an electronic device according to an embodiment may include an operation (1510) of acquiring a user's fingerprint image through a fingerprint sensor, an operation (1520) of identifying an area corresponding to a location of the user's fingerprint image, an operation (1530) of determining whether the user's fingerprint image is registered, an operation (1540) of updating the registered fingerprint image based on the determination result, an operation (1550) of determining whether an area greater than a reference value among a plurality of areas of a reference fingerprint image is registered, and an operation (1550) of completing fingerprint registration. However, the process (1500) for completing fingerprint registration of an electronic device is not limited thereto. For example, the process (1500) for completing fingerprint registration of an electronic device may omit at least one operation or may further include at least one operation.

[0186] In one embodiment, operation 1510 may be referenced by operation 1450 of FIG. 14. Operation 1520 may be referenced by operation 1460 of FIG. 14. Operation 1530 may be referenced by operation 1470 of FIG. 14. Operation 1540 may be referenced by operation 1480 of FIG. 14.

[0187] In one embodiment, operation 1510 may be a subsequent operation to operation 1490 of FIG. 14, but is not limited thereto. For example, operation 1550 may be a subsequent operation to operation 1480 of FIG. 14.

[0188] According to one embodiment, in operation 1550, at least one processor may determine whether an area greater than or equal to a reference value among a plurality of areas of a reference fingerprint image has been registered. For example, in response to acquiring a user fingerprint image through a fingerprint sensor, the at least one processor may determine whether an area greater than or equal to a reference value among fingerprint areas included in a plurality of areas of the reference fingerprint image has been registered.

[0189] According to one embodiment, at least one processor may perform an operation of completing fingerprint registration at operation 1560 based on a determination that an area greater than or equal to a reference value among a plurality of areas of a reference fingerprint image has been registered at operation 1550. For example, at least one processor may perform an operation of completing fingerprint registration based on a determination that a registered fingerprint image corresponding to the reference fingerprint image has been acquired.

[0190] According to one embodiment, at least one processor may re-perform operation 1510 based on a determination that an area greater than or equal to a reference value among the plurality of areas of the reference fingerprint image has not been registered in operation 1550. For example, at least one processor may re-perform operation 1510 based on a determination that an enrolled fingerprint image corresponding to the reference fingerprint image has not been acquired. For example, at least one processor may additionally acquire a user's fingerprint image through a fingerprint sensor based on a determination that an area greater than or equal to a reference value among the plurality of areas of the reference fingerprint image has not been registered.

[0191] Figure 16 is a flowchart illustrating a process for fingerprint registration according to one embodiment.

[0192] The contents of FIG. 16 may be referenced by the contents of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7, FIG. 8, FIG. 9, FIG. 10, FIG. 11, FIG. 12, FIG. 13, FIG. 14 and / or FIG. 15 described above.

[0193] According to one embodiment, a process (1600) for a fingerprint registration operation may include an operation (1610) of executing a fingerprint registration mode, and an operation (1620) of acquiring a first fingerprint image via a camera.

[0194] According to one embodiment, a method of operating an electronic device (e.g., a method of operating an electronic device (1400) of FIG. 14) may further include, but is not limited to, an operation (1610) of executing a fingerprint registration mode.

[0195] According to one embodiment, in operation 1610, at least one processor (e.g., processor (240) of FIG. 2) may execute a fingerprint registration mode. For example, the at least one processor may check contextual information related to a fingerprint input to the fingerprint sensor. For example, the at least one processor may check whether an application related to the fingerprint input is running. For example, the at least one processor may check whether a fingerprint input window is running. The at least one processor may determine whether contextual information related to the fingerprint input to the fingerprint sensor includes first contextual information related to fingerprint registration. For example, the at least one processor may check whether an application related to fingerprint registration is running. For example, the at least one processor may check whether a fingerprint registration window is running.

[0196] According to one embodiment, at least one processor may execute a fingerprint registration mode when it determines that context information related to a fingerprint input to the fingerprint sensor includes first context information related to fingerprint registration.

[0197] According to one embodiment, at operation 1620, in response to the fingerprint registration mode being executed, at least one processor may acquire a first fingerprint image via the camera. Operation 1620 may be referenced by operation 1410 of FIG. 14.

[0198] According to one embodiment, operation 1430 of FIG. 14 may be performed after operation 1620.

[0199] FIG. 17 is a diagram for explaining an operation of comparing a fingerprint image acquired through a fingerprint sensor and a fingerprint image acquired through a camera in a fingerprint authentication mode according to one embodiment.

[0200] The contents of FIG. 17 may be referenced by the contents of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7, FIG. 8, FIG. 9, FIG. 10, FIG. 11, FIG. 12, FIG. 13, FIG. 14, FIG. 15 and / or FIG. 16 described above.

[0201] FIG. 17 may be a drawing for explaining the operation of the fingerprint authentication mode after fingerprint registration is completed.

[0202] According to one embodiment, at least one processor (e.g., processor (240) of FIG. 2) may obtain a fingerprint image (1710) of a user for fingerprint authentication via a fingerprint sensor after a fingerprint registration process (e.g., process (1500) of FIG. 15). For example, the at least one processor may obtain a partial fingerprint image of the user for fingerprint authentication. However, the present invention is not limited thereto.

[0203] According to one embodiment, at least one processor may obtain crop images based on a reference fingerprint image (e.g., the reference fingerprint image (330) of FIG. 3, the second reference fingerprint image (430b) of FIG. 5, the fourth reference fingerprint image (530b) of FIG. 6, or the fifth reference fingerprint image). Referring to FIG. 17 (a), the at least one processor may identify, based on at least one crop image corresponding to a user's fingerprint image (1710) for fingerprint authentication obtained through a fingerprint sensor among the obtained crop images, an area corresponding to a location of the user's fingerprint image (1710) for fingerprint authentication obtained through a fingerprint sensor among a plurality of areas of the reference fingerprint image. The content of identifying the area corresponding to the location of the user's fingerprint image (1710) for fingerprint authentication obtained through the fingerprint sensor among the plurality of areas of the reference fingerprint image may be referred to by the content of FIG. 9. For example, an operation of identifying an area corresponding to a location of a user's fingerprint image (1710) for fingerprint authentication obtained through a fingerprint sensor among a plurality of areas of a reference fingerprint image may be substantially the same as an operation of identifying an area corresponding to a location of a user's fingerprint image for fingerprint registration obtained through a fingerprint sensor among a plurality of areas of a reference fingerprint image.

[0204] According to one embodiment, at least one processor may identify an area corresponding to a location of a user's fingerprint image (1710) for fingerprint authentication acquired through a fingerprint sensor among a plurality of areas of the reference fingerprint image in a secure execution environment (TEE). However, the present invention is not limited thereto. For example, at least one processor may identify an area corresponding to a location of a user's fingerprint image (1710) for fingerprint authentication acquired through a fingerprint sensor among a plurality of areas of the reference fingerprint image in a physically separate secure environment (e.g., an eSE or a secure processor).

[0205] According to one embodiment, at least one processor may determine whether fingerprint information in an area corresponding to a location of a fingerprint image (1710) among fingerprint information (1720) for a plurality of areas includes an image corresponding to at least a portion of the fingerprint image (1710). For example, at least one processor may determine whether fingerprint information in a third area (1721) corresponding to the location of the fingerprint image (1710) includes an image corresponding to at least a portion of the fingerprint image (1710).

[0206] According to one embodiment, at least one processor may determine whether fingerprint authentication is successful based on whether fingerprint information in a third area (1721) corresponding to a location of a fingerprint image (1710) includes an image corresponding to at least a portion of the fingerprint image (1710). For example, at least one processor may determine that fingerprint authentication is successful based on the determination result that the fingerprint information in the third area (1721) includes an image corresponding to at least a portion (e.g., the first portion (1711)) of the fingerprint image (1710). For example, at least one processor may determine that fingerprint authentication is successful based on a determination that, as a result of the determination, the fingerprint information of the third region (1721) includes an image corresponding to at least a portion (e.g., the first portion (1711)) of the fingerprint image (1710) and that the fingerprint information of the third region (1721) does not include an image corresponding to the remaining portion (e.g., the second portion (1722)) of the fingerprint image (1710). For example, at least one processor may determine that fingerprint authentication is unsuccessful based on a determination that, as a result of the determination, the fingerprint information of the third region (721) does not include an image corresponding to at least a portion of the fingerprint image.

[0207] According to one embodiment, at least one processor may compare the first portion (1711) of the fingerprint image (1710) with at least a portion of the registered fingerprint image based on the determination result that the fingerprint information of the third area (1721) includes an image corresponding to the first portion (1711) of the fingerprint image (1710). For example, the at least one processor may compare the quality of the first portion (1711) of the fingerprint image (1710) with the quality of at least a portion of the registered fingerprint image.

[0208] According to one embodiment, at least one processor may update the enrolled fingerprint image to include the first portion (1711) based on the comparison result. For example, the at least one processor may update the enrolled fingerprint image to include the first portion (1711) based on a determination that the first portion (1711) of the fingerprint image (1710) has a better quality than at least a portion of the enrolled fingerprint image. Accordingly, the quality of the enrolled fingerprint image may also be improved during fingerprint authentication.

[0209] According to one embodiment, at least one processor may update the registered fingerprint image so that the registered fingerprint image includes the second portion (1712) based on the determination result that the fingerprint information of the third region (1721) does not include an image corresponding to the second portion (1712) of the fingerprint image (1710). For example, the at least one processor may update the registered fingerprint image so that the second portion (1712) is included in an area of ​​the registered fingerprint image corresponding to the fourth region (1722). Accordingly, the electronic device may acquire a fingerprint image for a portion of the user's finger that was not acquired in the fingerprint registration operation as the registered fingerprint image in the fingerprint authentication operation. For example, the electronic device may additionally acquire a fingerprint image for a fingerprint of the side or top of the user's finger as the registered fingerprint image.

[0210] FIG. 18 is a flowchart illustrating a process (1800) for fingerprint authentication according to one embodiment.

[0211] The contents of FIG. 18 may be referenced by the contents of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7, FIG. 8, FIG. 9, FIG. 10, FIG. 11, FIG. 12, FIG. 13, FIG. 14, FIG. 15, FIG. 16 and / or FIG. 17 described above.

[0212] Referring to FIG. 18, a process (1800) for a fingerprint authentication operation may include an operation of executing a fingerprint authentication mode (1810), an operation of acquiring a third fingerprint image through a fingerprint sensor (1820), an operation of identifying an area corresponding to a location of the third fingerprint image (1830), an operation of determining whether fingerprint information for a reference fingerprint image includes an image corresponding to at least a portion of the third fingerprint image (1840), an operation of comparing at least a portion of the third image with at least a portion of an enrolled fingerprint image in response to a determination that the fingerprint information includes an image corresponding to at least a portion of the third image (1850), an operation of updating the enrolled fingerprint image to include a first portion in response to a comparison result (1860), and an operation of updating the enrolled fingerprint image to include a second portion in response to a determination that the fingerprint information does not include an image corresponding to the third image (1870). However, the process (1800) for a fingerprint authentication operation is not limited thereto. For example, a process (1800) for a fingerprint authentication operation may omit at least one operation or include at least one additional operation.

[0213] According to one embodiment, in operation 1810, at least one process (e.g., process (240) of FIG. 2) may execute a fingerprint authentication mode. For example, the at least one processor may check contextual information related to a fingerprint input to the fingerprint sensor. For example, the at least one processor may check whether an application related to the fingerprint input is running. For example, the at least one processor may check whether a fingerprint input window is running. The at least one processor may determine whether contextual information related to the fingerprint input to the fingerprint sensor includes second contextual information related to fingerprint authentication. For example, the at least one processor may check whether an application related to fingerprint authentication is running. For example, the at least one processor may check whether a fingerprint authentication window is running.

[0214] According to one embodiment, at least one processor may execute a fingerprint authentication mode when it determines that context information related to a fingerprint input to the fingerprint sensor includes second context information related to fingerprint authentication.

[0215] According to one embodiment, in operation 1820, at least one processor, in response to the fingerprint authentication mode being executed, may acquire a third fingerprint image for user fingerprint authentication via the fingerprint sensor. Operation 1820 may be referenced by operation 1450 of FIG. 14.

[0216] According to one embodiment, in operation 1830, at least one processor may identify an area corresponding to a location of a third fingerprint image. For example, the at least one processor may identify a third area corresponding to a location of the third fingerprint image among a plurality of areas of the reference fingerprint image based on at least one crop image corresponding to the third fingerprint image. Operation 1830 may be referenced by operation 1460 of FIG. 14 and / or the contents of FIG. 9.

[0217] In one embodiment, at operation 1840, at least one processor may determine whether the fingerprint information includes an image corresponding to at least a portion of a third image. For example, the at least one processor may determine whether the fingerprint information in the third region includes an image corresponding to at least a portion of the third image.

[0218] According to one embodiment, in operation 1840, at least one processor may determine whether fingerprint authentication is successful based on whether fingerprint information in a third region corresponding to a location of a third fingerprint image includes an image corresponding to at least a portion of the third fingerprint image. For example, the at least one processor may determine that fingerprint authentication is successful based on the determination result that the fingerprint information in the third region includes an image corresponding to at least a portion of the third fingerprint image. For example, the at least one processor may determine that fingerprint authentication is successful based on the determination result that the fingerprint information in the third region includes an image corresponding to at least a portion of the third fingerprint image and that the fingerprint information in the third region does not include an image corresponding to the remaining portion of the third fingerprint image. For example, the at least one processor may determine that fingerprint authentication is unsuccessful based on the determination result that the fingerprint information in the third region does not include an image corresponding to the third fingerprint image. In one embodiment, at least one processor may request the user to re-authenticate their fingerprint based on the determination that the fingerprint information in the third region does not contain an image corresponding to at least a portion of the third fingerprint image. For example, the at least one processor may request the user to re-enter a fingerprint image via the fingerprint sensor.

[0219] According to one embodiment, in operation 1850, at least one processor may compare at least a portion of the third image with at least a portion of the registered image based on the determination result that the fingerprint information of the third region includes an image corresponding to a first portion of the third fingerprint image.

[0220] In one embodiment, at operation 1860, the at least one processor may, in response to the comparison result, update the enrolled fingerprint image to include the first portion. For example, the at least one processor may update the enrolled fingerprint image to include at least a portion of the third fingerprint image based on a determination that at least a portion of the third fingerprint image has a better quality than at least a portion of the enrolled fingerprint image.

[0221] According to one embodiment, in operation 1870, at least one processor may update the registered fingerprint image to include the second portion based on the determination result that the fingerprint information of the third region does not include an image corresponding to the second portion of the third fingerprint image.

[0222] Actions 1840 to 1870 may be referenced by the contents of FIG. 17.

[0223] FIG. 19 is a conceptual diagram illustrating a concept of controlling functions related to fingerprint registration and fingerprint authentication in an electronic device according to one embodiment.

[0224] The contents of Fig. 19 may be referenced by the contents of Figs. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 and / or 18 described above. The same terms are used for the same or substantially the same configurations as described above.

[0225] Referring to FIG. 19, an electronic device according to an embodiment may include a camera module (1910) (e.g., the camera module (210) of FIG. 2), a fingerprint sensor (1920) (e.g., the fingerprint sensor of FIG. 2), a display (1930) (e.g., the display (230) of FIG. 2), a processor (1940) (e.g., the processor (240) of FIG. 2), and a memory (1950) (e.g., the memory (250) of FIG. 2). However, the components of the electronic device are not limited thereto. For example, the electronic device may omit at least one of the above-described components or may further include at least one other component.

[0226] According to one embodiment, the electronic device may utilize hardware and / or software modules to support various fingerprint registration-related functions and / or fingerprint authentication-related functions. For example, the processor (1940) may drive the first control module (1960) and the second control module (1970) by executing commands stored in the memory (1950). The first control module (1960) may include at least one control module for controlling the display (1930). The second control module (1970) may include at least one module for supporting functions related to the camera module (1910) and the fingerprint sensor (1920). For example, the second control module (1970) may include a fingerprint image conversion module (1971), a fingerprint image comparison module (1972), a fingerprint data storage module (1973), and a fingerprint data comparison module (1974). However, the present invention is not limited thereto. In various embodiments, software modules other than those illustrated in FIG. 19 may be implemented. For example, at least two modules may be integrated into a single module, or a single module may be split into two or more modules. Furthermore, hardware and software modules may share a single function, thereby improving operational performance. For example, an electronic device may include both a hardware-implemented encoder and a software-implemented encoder, and some of the data acquired through at least one camera module may be processed by a hardware encoder, while the remaining portion may be processed by a software encoder.

[0227] According to one embodiment, the first control module (1960) may control the display (1930) to display a visual object (e.g., the first visual object (311) of FIG. 4) that guides the user to capture a finger (or fingerprint) when the camera module (1910) captures a picture. The first control module (1960) may control the display (1930) to display a visual object (e.g., the first visual object (620) of FIG. 8) that guides the user to input a fingerprint. The first control module (1960) may control the display (1930) to display a visual object (e.g., the visual object (1300) of FIG. 13) that guides a fingerprint input method.

[0228] According to one embodiment, the fingerprint image conversion module (1971) can convert a user's finger image or a user's fingerprint image acquired through the camera module (1910) into a reference fingerprint image, or fingerprint information for a plurality of regions of the reference fingerprint image. For example, the fingerprint image conversion module (1971) can obtain a reference fingerprint image using a first artificial intelligence model trained to convert a fingerprint image acquired through the camera module (1910) into a reference fingerprint image. For example, the fingerprint image conversion module (1971) can obtain a reference fingerprint image using a second artificial intelligence model trained to change a distorted portion of a user's fingerprint image acquired through the camera module (1910), and / or a third artificial intelligence model trained to remove (or change) a distorted portion of a user's fingerprint image acquired through the camera module (1910).

[0229] According to one embodiment, the fingerprint image comparison module (1972) can compare a fingerprint image acquired through the camera module (1910) with a fingerprint image acquired through the fingerprint sensor (1920). The fingerprint image comparison module (1972) can compare a fingerprint image acquired through the camera module (1910) with a fingerprint image acquired through the fingerprint sensor (1920) to find a section where a fingerprint has been entered.

[0230] According to one embodiment, the fingerprint data storage module (1973) may store fingerprint data regarding a fingerprint image acquired through a fingerprint sensor (1920) and included in a registered fingerprint image in a memory (1950).

[0231] According to one embodiment, the fingerprint data comparison module (1974) can compare fingerprint data regarding a fingerprint image stored in a memory (1950) with fingerprint data regarding a fingerprint image acquired through a fingerprint sensor (1920).

[0232] In the embodiment of FIG. 19, the functions performed by the first control module (1960), the fingerprint image conversion module (1971), the fingerprint image comparison module (1972), the fingerprint data storage module (1973), and the fingerprint data comparison module (1974) can be understood as being performed by the processor (1940) executing commands stored in the memory (1950). In addition, in various embodiments, the electronic device may utilize one or more hardware processing circuits to perform various functions and operations disclosed in the present document. For example, an application processor (AP) included in a mobile device, an image signaling processor (ISP) mounted on a camera module, a DDIC, a touch IC, a communication processor (CP), a hardware encoder, and the like can be utilized to implement various embodiments disclosed in the present document. The processor (1940) in this document can be understood to include at least one processor described above.

[0233] Additionally, the hardware / software connection relationships illustrated in Figure 19 are for convenience of explanation and do not limit the flow / direction of data or commands. Components included in an electronic device may have various electrical / operational connection relationships.

[0234] As described above, an electronic device according to one embodiment (e.g., electronic device (200) of FIG. 2) may include a camera, a fingerprint sensor, a display, and at least one processor connected to the camera, the fingerprint sensor, and the display. The at least one processor may be configured to acquire a first fingerprint image of the user through the camera, acquire a reference fingerprint image based on the first fingerprint image, generate fingerprint information for a plurality of regions of the reference fingerprint image based on the reference fingerprint image, display a first visual object for guiding fingerprint input of the user through the display based on the reference fingerprint image and the fingerprint information, acquire a second fingerprint image of the user through the fingerprint sensor, identify a first region corresponding to a location of the second fingerprint image among the plurality of regions based on at least one crop image corresponding to the second fingerprint image among crop images acquired based on the reference fingerprint image, determine whether the second fingerprint image is registered based on whether the fingerprint information of the first region includes an image corresponding to the second fingerprint image, and update the registered fingerprint image based on a result of the determination so that the registered fingerprint image includes the second fingerprint image, and control the display to display a second visual object that updates at least a portion of the first visual object based on the first region.

[0235] According to one embodiment, the at least one processor may, in response to the fingerprint registration mode being executed, acquire the first fingerprint image of the user through the camera.

[0236] According to one embodiment, the electronic device further includes a memory, and the at least one processor can store fingerprint data regarding the second fingerprint image in the memory based on the determination result.

[0237] In one embodiment, the at least one processor may update the registered fingerprint image to include the second fingerprint image based on the fingerprint information of the first area not including an image corresponding to the second fingerprint image.

[0238] In one embodiment, the at least one processor can control the display to display the second visual object, wherein at least a portion of the first visual object corresponding to the first region is updated.

[0239] In one embodiment, the at least one processor can control the display to display the updated second visual object in at least a portion of the first visual object corresponding to a second region other than the first region among the plurality of regions.

[0240] According to one embodiment, the at least one processor, in response to the fingerprint authentication mode being executed, acquires a third fingerprint image of the user through the fingerprint sensor, and identifies a third region corresponding to a location of the third fingerprint image among the plurality of regions based on at least one crop image corresponding to the third fingerprint image among the crop images acquired based on the reference fingerprint image, and determines whether fingerprint information of the third region includes an image corresponding to at least a portion of the third fingerprint image.

[0241] According to one embodiment, the at least one processor may compare the first portion of the third fingerprint image with at least a portion of the registered fingerprint image based on the determination result that the fingerprint information of the third area includes an image corresponding to the first portion of the third fingerprint image, and update the registered fingerprint image so that the registered fingerprint image includes the first portion based on the comparison result.

[0242] According to one embodiment, the at least one processor may update the registered fingerprint image so that the registered fingerprint image includes the second portion, based on the determination result that the fingerprint information of the third region does not include an image corresponding to the second portion of the third fingerprint image.

[0243] According to one embodiment, the plurality of regions may correspond to each of the plurality of grid regions that segment the reference fingerprint image.

[0244] According to one embodiment, the at least one processor may obtain the reference fingerprint image using a first artificial intelligence model trained to convert the first fingerprint image into the reference fingerprint image.

[0245] According to one embodiment, the at least one processor may obtain the reference fingerprint image using a second artificial intelligence model learned to change a distorted portion of the first fingerprint image.

[0246] According to one embodiment, the at least one processor may obtain the reference fingerprint image using a third artificial intelligence model learned to remove a deformed portion of the first fingerprint image.

[0247] As described above, an operating method of an electronic device according to an embodiment (for example, an operating method (1400) of FIG. 14) includes: an operation of acquiring a first fingerprint image of a user through a camera; an operation of acquiring a reference fingerprint image based on the first fingerprint image; an operation of generating fingerprint information for a plurality of areas of the reference fingerprint image based on the reference fingerprint image; an operation of displaying a first visual object for guiding a fingerprint input of the user through a display based on the reference fingerprint image and the fingerprint information; an operation of acquiring a second fingerprint image of the user through a fingerprint sensor; an operation of identifying a first area corresponding to a location of the second fingerprint image among the plurality of areas based on at least one crop image corresponding to the second fingerprint image among crop images acquired based on the reference fingerprint image; an operation of determining whether the second fingerprint image is registered based on whether the fingerprint information of the first area includes an image corresponding to the second fingerprint image; an operation of updating the registered fingerprint image so that the registered fingerprint image includes the second fingerprint image based on a result of the determination; and an operation of displaying the display based on the first area. It may include an action to control display of a second visual object that has updated at least a portion of the first visual object.

[0248] According to one embodiment, the operation of updating the registered fingerprint image may include an operation of storing data regarding the second fingerprint image in memory based on the fingerprint information of the first area not including an image corresponding to the second fingerprint image.

[0249] In one embodiment, the act of controlling the display to display the second visual object may include the act of controlling the display to display the second visual object in which at least a portion of the first visual object corresponding to the first area is updated.

[0250] According to one embodiment, the act of controlling the display to display the second visual object may include an act of controlling the display to display the second visual object in which at least a portion of the first visual object corresponding to a second area other than the first area among the plurality of areas is updated.

[0251] According to one embodiment, the method of operating the electronic device may further include, in response to a fingerprint authentication mode being executed, acquiring a third fingerprint image of the user through the fingerprint sensor, identifying a third region corresponding to the third fingerprint image among the plurality of regions based on at least one crop image corresponding to the third fingerprint image among the crop images acquired based on the reference fingerprint image, and determining whether fingerprint information of the third region includes an image corresponding to at least a portion of the third fingerprint image.

[0252] According to one embodiment, the method of operating the electronic device may further include comparing at least a portion of the third fingerprint image with at least a portion of the registered fingerprint image based on the fingerprint information of the third area including an image corresponding to at least a portion of the third fingerprint image, and updating the registered fingerprint image based on a result of the comparison so that the registered fingerprint image includes at least a portion of the third fingerprint image.

[0253] According to one embodiment, the method of operating the electronic device may further include an operation of updating the registered fingerprint image so that the registered fingerprint image includes at least a portion of the third fingerprint image, based on the fingerprint information of the third area not including an image corresponding to at least a portion of the third fingerprint image.

[0254] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0255] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0256] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0257] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0258] According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0259] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Additionally or alternatively, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0260] In the present disclosure, a function or operation performed by an electronic device may be performed by one or more processors executing one or more instructions included in a program stored in a memory. The function or operation of the electronic device mentioned in the present disclosure may be performed by one processor executing one or more instructions, or may be performed by a combination of multiple processors executing one or more instructions. The processor mentioned in the present disclosure may be understood to include a circuit for performing an operation or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a microprocessor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on a chip (SoC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operation of the electronic device described above.

[0261] In the present disclosure, a program (software module, software) may be stored in a non-volatile memory including a random access memory (RAM), a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, it may be stored in a memory formed by a combination of some or all of these. The memory may be formed by a single storage medium or may be formed by a combination of a plurality of storage media. The one or more instructions may be stored in a single storage medium or may be distributed and stored in a plurality of storage media.

Claims

1. In electronic devices, camera; Fingerprint sensor; display; at least one processor; and Contains memory that stores instructions, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Obtain the user's first fingerprint image through the above camera, Obtain a reference fingerprint image based on the first fingerprint image above, Based on the above reference fingerprint image, fingerprint information for multiple areas of the above reference fingerprint image is generated, A first visual object for guiding the user's fingerprint input based on the above reference fingerprint image and the above fingerprint information is displayed through the display, Obtaining a second fingerprint image of the user through the fingerprint sensor, Based on at least one crop image corresponding to the second fingerprint image among the crop images acquired based on the above reference fingerprint image, a first region corresponding to the location of the second fingerprint image is identified among the plurality of regions, Whether the second fingerprint image is registered is determined based on whether the fingerprint information of the first area includes an image corresponding to the second fingerprint image, Based on the above judgment result, the registered fingerprint image is updated so that the registered fingerprint image includes the second fingerprint image, Causing the display to display a second visual object that updates at least a portion of the first visual object based on the first region; Electronic devices.

2. In claim 1, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: In response to the fingerprint registration mode being executed, acquiring the user's first fingerprint image through the camera; Electronic devices.

3. In claim 1 or claim 2, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Based on the above judgment result, fingerprint data regarding the second fingerprint image is stored in the memory. Electronic devices.

4. In any one of the claims set forth above, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: If the second fingerprint image does not contain an image corresponding to the second fingerprint image, the registered fingerprint image is updated so that the registered fingerprint image includes the second fingerprint image. Electronic devices.

5. In any one of the claims set forth above, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Controlling the display so that at least a portion of the first visual object corresponding to the first region displays the updated second visual object; Electronic devices.

6. In any one of the claims set forth above, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Controlling the display so that at least a portion of the first visual object corresponding to a second area other than the first area among the plurality of areas displays the updated second visual object. Electronic devices.

7. In any one of the claims set forth above, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: In response to the fingerprint authentication mode being executed, a third fingerprint image of the user is acquired through the fingerprint sensor, Based on at least one crop image corresponding to the third fingerprint image among the crop images acquired based on the above reference fingerprint image, a third region corresponding to the location of the third fingerprint image is identified among the plurality of regions, To determine whether the fingerprint information of the third area includes an image corresponding to at least a part of the third fingerprint image. Electronic devices.

8. In claim 7, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: As a result of the judgment, if the fingerprint information of the third area includes an image corresponding to the first part of the third fingerprint image, the first part of the third fingerprint image is compared with at least a part of the registered fingerprint image, Based on the above comparison result, updating the registered fingerprint image so that the registered fingerprint image includes the first part. Electronic devices.

9. In claim 7 or claim 8, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: As a result of the judgment, if the fingerprint information of the third area does not include an image corresponding to the second part of the third fingerprint image, the registered fingerprint image is updated so that the registered fingerprint image includes the second part. Electronic devices.

10. In any one of the claims set forth above, The above multiple regions correspond to each of the multiple grid regions that divide the reference fingerprint image. Electronic devices.

11. In any one of the claims set forth above, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Obtaining the reference fingerprint image by using the first artificial intelligence model learned to convert the first fingerprint image into the reference fingerprint image. Electronic devices.

12. In any one of the claims set forth above, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Obtaining the reference fingerprint image by using a second artificial intelligence model learned to change the distorted portion of the first fingerprint image. Electronic devices.

13. In any one of the claims set forth above, The above instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Obtaining the reference fingerprint image by using a third artificial intelligence model trained to remove the deformed portion of the first fingerprint image. Electronic devices.

14. In the method of operating an electronic device, An action of acquiring a user's first fingerprint image via a camera; An operation of obtaining a reference fingerprint image based on the first fingerprint image; An operation of generating fingerprint information for a plurality of areas of the reference fingerprint image based on the reference fingerprint image; An action of displaying a first visual object to guide fingerprint input of the user based on the reference fingerprint image and the fingerprint information; An action of acquiring a second fingerprint image of the user through a fingerprint sensor; An operation of identifying a first region corresponding to a location of the second fingerprint image among the plurality of regions based on at least one crop image corresponding to the second fingerprint image among crop images acquired based on the reference fingerprint image; An operation of determining whether or not to register the second fingerprint image based on whether the fingerprint information of the first area includes an image corresponding to the second fingerprint image; Based on the above judgment result, an operation of updating the registered fingerprint image so that the registered fingerprint image includes the second fingerprint image; and An operation for controlling the display to display a second visual object that updates at least a portion of the first visual object based on the first region; How it works.

15. In claim 14, The action of updating the above registered fingerprint image is: If the fingerprint information of the first area does not include an image corresponding to the second fingerprint image, the method comprises storing fingerprint data regarding the second fingerprint image in memory. How it works.

Images