Method of obtaining a reference image and method of performing optical object recognition

Abstract

A method of obtaining a reference image and a method of performing optical object recognition are provided. The method of obtaining a reference image includes: driving a subset of light sources among a plurality of light sources; receiving, using an object recognition sensor, light reflected from a first target object; obtaining a first reference image based on the reflected light; obtaining a first target image related to the first target object based on the reflected light; obtaining at least one first environmental information using at least one environmental sensor while the subset of light sources is driven, the first environmental information being related to a surrounding environment; storing the first reference image together with the first environmental information; and obtaining a first effective image for the first target object based on the first target image and the first reference image.

Description

[0001] This application claims priority and benefit to Korean Patent Application No. 10-2020-0050646, filed on April 27, 2020, with the Korean Intellectual Property Office (KIPO), the entire contents of which are incorporated herein by reference. Technical Field

[0002] Various example embodiments generally relate to semiconductor integrated circuits, systems including semiconductor integrated circuits, non-transitory computer-readable media storing computer instructions, methods for obtaining reference images for optical object recognition using semiconductor integrated circuits, and / or methods for performing optical object recognition using methods for obtaining reference images. Background Technology

[0003] Biometric information is widely used for personal authentication due to its immutability and uniqueness for each individual. One type of biometric information is fingerprints. Fingerprint recognition can be easily performed and serves as an excellent method for determining a person's identity. Optical fingerprint recognition obtains a fingerprint image based on the differences in light reflected from the ridges and valleys of the fingers. Recently, in addition to optical fingerprint recognition, optical object recognition, which obtains object images based on light reflected from various objects, has been studied. Summary of the Invention

[0004] At least one example embodiment of the inventive concept provides a method for obtaining a reference image for optical object recognition that supports efficient optical object recognition.

[0005] At least one example embodiment of the inventive concept provides a method for performing optical object recognition using a method of obtaining a reference image.

[0006] According to at least one example embodiment, a method for obtaining a reference image for optical object recognition may include: driving a subset of light sources from a plurality of light sources included in at least one display panel, the subset of light sources corresponding to an object recognition window that is a local area of ​​the display panel; receiving light reflected from a first target object through the object recognition window using an object recognition sensor, the light being emitted by the subset of light sources, the first target object being the target of optical object recognition; obtaining a first reference image based on the reflected light when the subset of light sources is driven; obtaining a first target image associated with the first target object based on the reflected light when the subset of light sources is driven; obtaining at least one first environmental information related to the surrounding environment using at least one environmental sensor when the subset of light sources is driven; storing the first reference image and the first environmental information together; and obtaining a first valid image for optical object recognition associated with the first target object based on the first target image and the first reference image.

[0007] According to at least one example embodiment, a method for performing optical object recognition may include: driving a subset of a plurality of light sources included in a display panel, the subset of light sources corresponding to an object recognition window that is a local area of ​​the display panel; when the subset of light sources is driven, obtaining a plurality of reference images using an object recognition sensor, each of the plurality of reference images being an image excluding a first target object; when the subset of light sources is driven, obtaining a plurality of environmental information corresponding to the plurality of reference images using at least one environmental sensor; when the subset of light sources is driven, obtaining a first target image including the first target object using an object recognition sensor; when the subset of light sources is driven, obtaining current environmental information corresponding to the first target image using an environmental sensor; selecting a first reference image among the plurality of reference images based on the current environmental information and the plurality of environmental information; and obtaining a first valid image for the first target object based on the first target image and the first reference image.

[0008] According to at least one example embodiment, a method for obtaining a reference image for optical object recognition may include: driving a subset of light sources from a plurality of light sources included in a display panel, the subset of light sources corresponding to an object recognition window that is a local area of ​​the display panel; receiving light reflected through the object recognition window using an object recognition sensor, the light being emitted by the subset of light sources; sequentially obtaining a plurality of images based on the reflected light while the subset of light sources is driven; obtaining at least one of the plurality of images as a first reference image, the first reference image being an image excluding a first target object, the step of obtaining at least one of the plurality of images as the first reference image including: obtaining an image corresponding to the first target object based on the reflected light while the subset of light sources is driven. The associated first target image is used to obtain multiple first values ​​by performing spatial signal processing on the multiple images, and multiple second values ​​by performing frequency signal processing on the multiple images. Based on the corresponding first values, the corresponding second values, a spatial domain threshold, and a frequency domain threshold, at least one image is selected as a first reference image among the multiple images. When a subset of the light sources is driven, at least one first environmental information is obtained using at least one environmental sensor, the first environmental information being related to the surrounding environment where the subset of light sources is driven. The first reference image and the first environmental information for the first reference image are stored together. A first effective image for optical object recognition associated with a first target object is obtained based on the first target image and the first reference image.

[0009] In the methods for obtaining a reference image for optical object recognition and performing optical object recognition according to some example embodiments, a target-object-free image or an image without a target object (e.g., a reference image) can be effectively obtained for performing optical object recognition. Furthermore, the target-object-free image and the image including the target object can be obtained in the same environment, or multiple images obtained sequentially in the same environment can be selected such that the noise, interference, and other characteristics of the target-object-free image and the target image including the target object are equal or matched. Therefore, robust object image recovery can be performed, or pure information related to the target object can be robustly recovered based on the target-object-free image, and effective optical object recognition can be supported. Attached Figure Description

[0010] The illustrative, non-limiting exemplary embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0011] Figure 1 This is a flowchart illustrating a method for obtaining a reference image for optical object recognition according to some example embodiments.

[0012] Figure 2 This is a plan view of an electronic device according to some example embodiments.

[0013] Figure 3 It is based on at least one example embodiment along Figure 2 A cross-sectional view of an example electronic device taken by line A-A'.

[0014] Figure 4 This illustrates at least one example embodiment. Figure 2 A block diagram of an example electronic device.

[0015] Figure 5A , Figure 5B , Figure 5C and Figure 5D This is an illustration for describing a method of obtaining a reference image for optical object recognition according to some example embodiments.

[0016] Figure 6 and Figure 7 This illustrates the acquisition according to at least one example embodiment. Figure 1 A flowchart illustrating an example of a method using a reference image.

[0017] Figure 8A , Figure 8B , Figure 8C , Figure 8D , Figure 8E , Figure 8F and Figure 8G It is used to describe the inspection according to at least one example embodiment. Figure 7The diagram shows whether the display panel or electronic device in the diagram is in an unused operating state.

[0018] Figure 9 This illustrates the acquisition according to at least one example embodiment. Figure 1 A flowchart of another example of the method using a reference image.

[0019] Figure 10 This illustrates the checking of whether the first reference image is, according to at least one example embodiment. Figure 9 The flowchart shows an example of a contaminated image.

[0020] Figure 11A , Figure 11B , Figure 11C , Figure 11D and Figure 12 This is used to describe checking whether a first reference image is, according to at least one example embodiment. Figure 10 A diagram illustrating the operation of a contaminated image.

[0021] Figure 13 This illustrates the checking of whether the first reference image is, according to at least one example embodiment. Figure 9 The flowchart is another example of a contaminated image.

[0022] Figure 14 This is used to describe checking whether a first reference image is, according to at least one example embodiment. Figure 13 A diagram illustrating the operation of a contaminated image.

[0023] Figure 15 This is a flowchart illustrating a method for obtaining a reference image for optical object recognition according to some example embodiments.

[0024] Figure 16 This is a flowchart illustrating a method for performing optical object recognition according to some example embodiments.

[0025] Figure 17 This illustrates a selection based on at least one example embodiment. Figure 16 A flowchart of an example of the first reference image in the diagram.

[0026] Figure 18 This illustrates the acquisition according to at least one example embodiment. Figure 16 A flowchart of an example of the first valid image in the diagram.

[0027] Figure 19A and Figure 19B This is a diagram illustrating a method for performing optical object recognition according to some example embodiments.

[0028] Figure 20 This is a block diagram illustrating an electronic device according to some example embodiments. Detailed Implementation

[0029] Various exemplary embodiments will be described more fully with reference to the accompanying drawings illustrating the embodiments. However, exemplary embodiments of the inventive concept can be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Throughout this application, the same reference numerals denote the same elements.

[0030] Figure 1 This is a flowchart illustrating a method for obtaining a reference image for optical object recognition according to some example embodiments.

[0031] Reference Figure 1 The method for obtaining a reference image for optical object recognition, according to some example embodiments, is performed by an electronic device, which includes, but is not limited to, a display panel, an object recognition sensor, and / or at least one environmental sensor, and may include more or fewer constituent components. The display panel includes multiple light sources (e.g., pixels, etc.), the object recognition sensor performs optical object recognition using light provided from (and / or emitted from, etc.) the multiple light sources, and the at least one environmental sensor obtains environmental information related to the surrounding environment. (Refer to...) Figures 2 to 4 Describe the detailed configuration of the electronic device.

[0032] In a method for obtaining a reference image for optical object recognition according to some example embodiments, some light sources (e.g., a first set of light sources, a subset of light sources, etc.) among a plurality of light sources included in a display panel are driven (operation S100). Some light sources are configured (and / or arranged) to correspond to an object recognition window, which is a local area of ​​the display panel (e.g., a subset, a sub-region, etc.). According to at least one example embodiment, all of some light sources may emit light with the same grayscale value, or some light sources may emit light with different grayscale values ​​and / or different color values, etc.

[0033] When some light source is driven, a first reference image for optical object recognition is obtained or captured based on the reflected light (e.g., light reflected from a first object) received by the object recognition sensor through the object recognition window (operation S200). Figure 1 In the example, only one image may be obtained or acquired as a reference image; however, the example embodiment is not limited to this. Additionally, for example, all of some light sources may be turned on substantially simultaneously or concurrently, but the example embodiment is not limited to this. Light generated from some light sources may be emitted to an object recognition window and may be reflected by any object (or arbitrary object) located on and / or placed on the object recognition window. The reflected light may be provided to the object recognition sensor, so that the object recognition sensor can obtain an image corresponding to the object recognition window and / or any object based on the reflected light.

[0034] In some example embodiments, the first reference image may be an image excluding the first object. Typically, the panel, including the display panel and the touch sensor panel, has a complex internal structure with patterns including multiple layers of wiring and electrodes. When an object is placed on the object recognition window and a light source in the object recognition window illuminates it, the reflected light received through the object recognition window may include information about the object placed on the object recognition window (e.g., an image) and information about the internal structure of the panel (e.g., an image), such as a bottom view of the object and / or the internal structure of the panel. Therefore, in order to obtain only information about the object placed on the object recognition window (e.g., the target object), information about the internal structure of the panel, as an interference component, should first be obtained. A compensation factor should then be applied to the obtained subsequent image signal of the target object (e.g., a second reference image), which removes the interference component from the obtained image signal of the target object, etc., but the example embodiments are not limited to this. Therefore, the first reference image may represent the interference component and may be referred to as a calibration image or calibration data, etc.

[0035] In some example embodiments, the first object placed on the object recognition window may be a user's finger, including the user's fingerprint. In this example, the object recognition window and the object recognition sensor may be a fingerprint recognition window and a fingerprint recognition sensor, respectively. However, the example embodiments are not limited to this, and the first object may be an object including biometric information for user authentication and security (such as the face and / or iris of a person and / or a user), or may be one of various objects including objects that do not correspond to the biometric information of a person (such as a uniquely identifiable object).

[0036] When some light sources are driven, at least one environmental sensor is used to obtain at least one first environmental information (operation S300). The first environmental information is related to the surrounding environment at which some of the light sources are driven. The first environmental information may represent environmental information about the surrounding environment at the time point when the first reference image is obtained.

[0037] In some example embodiments, the first environmental information may include at least one or any combination of temperature information, humidity information, pressure information, motion information, time information, spatial information, illuminance information, acceleration information, vibration information, external force information, and / or impact information. However, the example embodiments are not limited thereto, and the first environmental information may also include at least one of various other environmental information and / or display setting information.

[0038] The first reference image and first environmental information for the first reference image are stored together (operation S400). Therefore, if the reference image is stored together... Figure 16When a valid image is obtained, the first reference image can be selected based on or taking into account the first environmental information.

[0039] In methods for obtaining reference images for optical object recognition according to some example embodiments, object-free images or images without objects (e.g., a first reference image, a calibration image, etc.) for performing optical object recognition can be efficiently obtained. The optical object recognition obtains only object-related pure information by subtracting the object-free image (e.g., the first reference image, a calibration image, etc.) from an image including the object (e.g., a second reference image, etc.). Furthermore, the object-free image and the image including the object can be obtained in the same environment (e.g., the same physical location and / or the same physical / environmental conditions (e.g., lighting conditions, etc.)), such that the noise, interference, and other characteristics of the object-free image and the image including the object are equal or matched. As described above, by obtaining and storing environmental information simultaneously with obtaining the object-free image, robust object image recovery can be performed, or object-related pure information can be robustly recovered based on the object-free image, and efficient optical object recognition can be supported.

[0040] although Figure 1 The example shows that operation S200 is performed before operation S300, but the example embodiment is not limited thereto, and operations S200 and S300 may be performed substantially simultaneously, or operation S300 may be performed before operation S200.

[0041] Figure 2 This is a plan view of an electronic device according to some example embodiments.

[0042] Reference Figure 2 The electronic device 100 includes at least one panel 110 connected to a user interface. A user of the electronic device 100 can view information, graphics, etc., output from the electronic device 100 through the panel 110. The user of the electronic device 100 can also input at least one signal to the electronic device 100 through the panel 110. For example, the panel 110 may include a display panel for outputting visual information to the user and / or a touch sensor panel for sensing the user's touch input, or any combination thereof. Although only a single panel 110 is used... Figure 2 The example is shown, but the example embodiment is not limited thereto, and there may be two or more display panels included in the electronic device 100, etc.

[0043] The Object Recognition Window (ORW) can be configured (e.g., placement, location, inclusion, etc.) on panel 110. For example, referencing... Figure 3The object recognition sensor used for object detection can be configured (e.g., arranged, located, included, etc.) to spatially correspond to the position of the object recognition window ORW. Although the object recognition window ORW is located in... Figure 2 As shown as rectangles in the following figures, the shape, position, and / or number of one or more object recognition windows (ORWs) may be changed in other example embodiments.

[0044] In some example embodiments, electronic device 100 may be or include any mobile system, such as mobile phone, smartphone, tablet computer, laptop computer, personal digital assistant (PDA), portable multimedia player (PMP), digital camera, portable game console, music player, camcorder, video player, navigation device, wearable device, Internet of Things (IoT) device, Internet of Everything (IoE) device, e-book reader, virtual reality (VR) device, augmented reality (AR) device, robotic device, or drone, etc.

[0045] One or more example embodiments may provide at least one interface for detecting objects. For example, in the case of fingerprint detection, the fingerprint detection function may be performed when a user touches and / or approaches panel 110. According to some example embodiments, the interface for object detection and the object recognition sensor may share an area on the electronics 100 with panel 110, so the interface and object recognition sensor may not require additional area on the electronics 100, but the example embodiments are not limited thereto. Therefore, the size of the electronics 100 may be reduced, or the spare area may be used for one or more other purposes.

[0046] Figure 3 It is based on at least one example embodiment along Figure 2 A cross-sectional view of an example electronic device taken by line A-A'.

[0047] Reference Figure 3 The Object Recognition Window (ORW) can be displayed on a local area (or part, sub-area, etc.) of the panel 110 in object recognition mode. The panel 110 may include at least one display panel 111 and at least one touch sensor panel 115, but is not limited thereto.

[0048] The display panel 111 may include multiple light sources 112. For example, as shown in the reference... Figure 4 The plurality of light sources 112 may be included in a plurality of pixels included in the display panel 111. Among the plurality of light sources 112, only some light sources 113 (e.g., a subset of light sources, a first subset, etc.) that are set (and / or located, included, arranged, etc.) to correspond to the object recognition window ORW can be driven substantially simultaneously in object recognition mode. Figure 3 In the image, some light sources 113 that are driven and emit light are indicated by shaded lines.

[0049] The object recognition sensor 130 may be disposed below the panel 110 such that the object recognition sensor 130 may overlap with the object recognition window (ORW) in the vertical direction. In other words, the panel 110 may include a first surface on which an image is displayed and a second surface opposite to the first surface, and the object recognition sensor 130 may be disposed below the second surface of the panel 110. However, the example embodiment is not limited to this, and the object recognition sensor 130 and / or ORW may be arranged in alternative directions and / or orientations, etc.

[0050] The object recognition sensor 130 may include at least one lens 132 and / or at least one image sensor 134, etc. The lens 132 may be disposed (and / or located, included, arranged, etc.) below the panel 110 (e.g., disposed between the panel 110 and the image sensor 134), and may converge and / or focus reflected light received through the object recognition window (ORW) onto the image sensor 134. The image sensor 134 may be disposed (and / / or located, included, arranged, etc.) below the lens 132, and may generate an image signal for an object in a local area based on the reflected light converged by the lens 132. In some example embodiments, the lens 132 may be omitted from the object recognition sensor 130.

[0051] For example, when a user places their finger 10 on a surface... Figure 3 In the case of fingerprint detection on the object recognition window (ORW) shown, light generated from some light sources 113 within the object recognition window (ORW) can be reflected from the finger 10 and / or the fingerprint of the finger 10, and the reflected light from the fingerprint (e.g., related to and / or corresponding to the fingerprint) can be provided to the object recognition sensor 130. The object recognition sensor 130 can capture an image signal of the fingerprint or information related to the shape of the fingerprint (e.g., a fingerprint image) based on the reflected light from the fingerprint received through the object recognition window (ORW).

[0052] For example, when obtaining a reference image for object recognition (e.g., a first reference image, a calibration image, etc.), the object may not be placed on the object recognition window ORW, or any flat, non-bent white or black object may be set (and / or located, included, arranged, etc.) on the object recognition window ORW to facilitate obtaining the reference image, and reflected light based on light generated from some light source 113 within the object recognition window ORW may be provided to the object recognition sensor 130, etc. The object recognition sensor 130 may capture an image signal for the reference image (e.g., an image representing the internal structure of panel 110) based on the reflected light received through the object recognition window ORW, but is not limited thereto.

[0053] Although not in Figure 3As shown, the object recognition sensor 130 may also include at least one filter for, for example, adjusting the frequency characteristics and / or polarization characteristics of the reflected light to be provided to the image sensor 134, but the example embodiment is not limited thereto.

[0054] Figure 4 This illustrates at least one example embodiment. Figure 2 A block diagram of an example electronic device.

[0055] Reference Figure 4 The electronic device 100 includes at least one panel 110 and / or at least one object recognition sensor 130, but the example embodiments are not limited thereto. Panel 110 may include a display panel 111 and / or a touch sensor panel 115, but is not limited thereto. The electronic device 100 may also include a processing circuitry system including one or more of a display driver 120, a touch controller 125, at least one processor 140, a memory 150, and / or at least one environmental sensor 160, but the example embodiments are not limited thereto. According to some example embodiments, the processing circuitry system is capable of performing the functions of one or more of the display driver 120, touch controller 125, at least one processor 140, memory 150, and / or at least one environmental sensor 160. The processing circuitry system may include hardware (such as a processor, processor core, logic circuitry, storage device, etc.), hardware / software combinations (such as at least one processor core executing software and / or executing any instruction set, etc.), or combinations thereof. For example, the processing circuitry system may more specifically include, but is not limited to, field-programmable gate arrays (FPGAs), programmable logic units, application-specific integrated circuits (ASICs), system-on-a-chip (SoCs), etc. In other example embodiments, the display driver 120, touch controller 125, at least one processor 140, memory 150 and / or at least one environmental sensor 160 may be combined into a single circuit, or one or more separate circuits / components / elements, etc.

[0056] The touch sensor panel 115 can sense the touch of an object (e.g., a user's finger) and / or the proximity of an object (e.g., a user's finger). The touch sensor panel 115 can generate a sensing signal in response to the touch or proximity of an object. The touch sensor panel 115 may include, but is not limited to, a plurality of sensing capacitors formed along rows and columns. Figure 4An example sensing capacitor CS is shown. The capacitance value of the sensing capacitor may change in response to the contact or proximity of an object, so the touch sensor panel 115 may sense contact and / or proximity based on the capacitance value of the sensing capacitor. However, the example embodiment is not limited to this, and the touch sensor panel 115 may use other techniques to sense the contact and / or proximity of an object (such as sensing the amount of pressure applied to the touch sensor panel 115, sensing the light level near the touch sensor panel 115, etc.).

[0057] Touch controller 125 can control the operation of touch sensor panel 115. Touch controller 125 can process at least one operation related to contact and / or proximity of an object based on sensing signals output from touch sensor panel 115. For example, touch controller 125 can identify contact and / or proximity of an object based on changes in the capacitance value of a sensing capacitor, but the example embodiment is not limited thereto. For example, when the sensing signals are related to the execution or operation of a particular application, touch controller 125 can output at least one command to at least one processor 140, etc., so that the particular application will be executed or operated.

[0058] Display panel 111 outputs visual information (e.g., text, graphics, images, videos, etc.) to the user. Display panel 111 may include multiple pixels arranged along rows and columns to display visual information (e.g., images, videos, text, etc.). Figure 4 An example pixel PX is shown, but the example embodiment is not limited thereto. Each pixel can be configured to emit light of a specific color that forms an image (e.g., light with a desired color value and / or no light emission, etc.). When multiple pixels emit light together, the display panel 111 can display desired and / or anticipated visual information (e.g., images, videos, text, etc.).

[0059] In some example embodiments, the display panel 111 may be an electroluminescent display panel, but is not limited thereto. At least one light-emitting diode (LED) or at least one organic light-emitting diode (OLED) that generates light through the recombination of electrons and holes can be used to drive the electroluminescent display panel with fast response speed and low power consumption. Compared to a liquid crystal display panel using a backlight unit, the pixels of an electroluminescent display panel can emit light themselves, and reflected light received through the object recognition window ORW (or object recognition window ORW') can be provided to the object recognition sensor 130 below the display panel 111 through the space (or gap) between the pixels. Therefore, according to at least one example embodiment, the LED or organic light-emitting diode included in the pixels may correspond to a light source included in the display panel. However, the example embodiments are not limited thereto, and the display panel 111 may be any display panel having a structure in which reflected light received through the object recognition window ORW or ORW' can be provided to the object recognition sensor 130, etc.

[0060] Display driver 120 can control the operation of display panel 111 and can drive display panel 111. For example, display driver 120 can appropriately drive (e.g., supply drive voltage, etc.) each pixel of display panel 111 in response to at least one command from processor 140, such that a desired or anticipated image is displayed on display panel 111, but is not limited thereto. For example, display driver 120 can partially drive display panel 111 such that pixels corresponding to object recognition windows ORW' emit light, but is not limited thereto. Although not explicitly stated in Figure 4 As shown, however, the display driver 120 may include a data driver, a scan driver, a timing controller, a gamma circuit, etc.

[0061] Each coordinate on the touch sensor panel 115 can be matched with a corresponding coordinate on the display panel 111. For example, the display panel 111 can display interface information on a specific area P. A user can touch or approach a specific area Q on the touch sensor panel 115 to input commands through the displayed interface information. Here, the coordinates of the specific area Q can be matched and / or correspond to the coordinates of the specific area P. Therefore, touch or proximity to the specific area Q can be processed in relation to the interface information displayed on the specific area P.

[0062] In some example embodiments, the touch sensor panel 115 may be implemented separately from the display panel 111. For example, as Figure 4 As shown, the touch sensor panel 115 can be placed on or above the display panel 111. However, the example embodiment is not limited to this. For example, with... Figure 4 As shown, the display panel 111 may be placed on or above the touch sensor panel 115. Alternatively, the touch sensor panel 115 and the display panel 111 may be implemented in a single panel, etc.

[0063] Object recognition sensor 130 can be used to detect objects. Object recognition sensor 130 can generate / output image signals associated with objects on and / or near the object recognition window (ORW). For example, in the case of fingerprint detection, object recognition sensor 130 can operate to obtain image signals associated with the fingerprint of a finger that has touched or approached the object recognition window (ORW), but the example embodiment is not limited thereto. See also... Figure 3 The object recognition sensor 130 may include a lens 132 and an image sensor 134, but the example embodiment is not limited thereto. For example, the lens 132 may be omitted, and there may be more than one lens 132 and / or more than one image sensor 134, etc.

[0064] The object recognition sensor 130 can provide optical object recognition and / or optical-based object detection capabilities. For example, the image sensor 134 included in the object recognition sensor 130 may include one or more photodiodes capable of generating current in response to light, but the example embodiments are not limited thereto.

[0065] For reference Figure 2 The object recognition window ORW can be disposed on panel 110 (e.g., on touch sensor panel 115). Alternatively, object recognition window ORW' can be disposed on display panel 111 to correspond to the object recognition window ORW. Object recognition sensor 130 can be located below display panel 111 (e.g., disposed below display panel 111, included below display panel 111, located below display panel 111, etc.) to spatially correspond to the positions of object recognition window ORW and object recognition window ORW', but the example embodiment is not limited thereto.

[0066] In some example embodiments, the position of the object recognition window ORW may be related to coordinates on the touch sensor panel 115, and the position of the object recognition window ORW' may be related to coordinates on the display panel 111, etc. Furthermore, the position and size of each of the object recognition windows ORW and ORW' may be modified or changed according to the arrangement of the object recognition sensor 130.

[0067] At least one processor 140 can control the overall operation of the electronic device 100. The processor 140 can process / execute various arithmetic / logic operations to provide the functions of the electronic device 100, etc.

[0068] Processor 140 can communicate with display driver 120, touch controller 125, object recognition sensor 130, memory 150, and / or environmental sensor 160, etc. Processor 140 can control, but is not limited to, the operation of display driver 120, touch controller 125, object recognition sensor 130, memory 150, and / or environmental sensor 160, etc. Processor 140 can process commands, requests, and / or responses related to the operation of display driver 120, touch controller 125, object recognition sensor 130, memory 150, and / or environmental sensor 160, etc.

[0069] For example, processor 140 can process commands received from touch controller 125 to understand (e.g., implement, execute, etc.) user commands input through touch sensor panel 115. For example, processor 140 can provide various information to display driver 120 to display a desired or anticipated image on display panel 111. For example, processor 140 can control the timing / sequence of operation of display panel 111 and / or object recognition sensor 130, causing object recognition sensor 130 to generate signals related to object images and / or reference images. For example, processor 140 can generate and / or analyze information related to object images and / or reference images based on signals output from object recognition sensor 130. For example, processor 140 can receive and / or analyze information about the surrounding environment from environmental sensor 160, etc. For example, processor 140 can store relevant data in memory 150 and / or load relevant data from memory 150.

[0070] In some example embodiments, processor 140 may include one or more special-purpose circuits (e.g., field-programmable gate arrays (FPGAs) and / or application-specific integrated circuits (ASICs) to perform various operations. For example, processor 140 may include one or more processor cores capable of performing various operations. For example, processor 140 may be implemented using a special-purpose (e.g., custom-designed) processor, a general-purpose processor loaded with special-purpose computer-readable instructions for implementing one or more methods of the example embodiments, thereby turning a general-purpose processor into a special-purpose processor, and / or an application processor, etc.

[0071] At least one environmental sensor 160 can collect, sense, and / or determine environmental information such as the surrounding environment in which the electronic device 100 is driven. For example, the environmental sensor 160 may collect environmental information when an image of a reference image (e.g., a calibration image) and / or a target object image is acquired (e.g., at the same time as the reference image, calibration image, and / or target object image are captured, and / or before or after the reference image is captured, etc.), may collect environmental information to check for unused (or inactive) states, or may collect environmental information periodically or intermittently.

[0072] In some example embodiments, the environmental sensor 160 may include at least one or any combination of a temperature sensor, humidity sensor, pressure sensor, motion sensor, time sensor, space sensor, illuminance sensor, acceleration sensor, vibration sensor, external force sensor, impact sensor, etc. However, the example embodiments are not limited thereto, and the environmental sensor 160 may also include at least one sensor (such as a radiation sensor, dust sensor, or electrical stress sensor, etc.) for collecting environmental information.

[0073] Although not in Figure 4As shown, however, the sensor control circuitry for controlling the environmental sensor 160 may be included in the processing circuitry system (e.g., processor 140, etc.) or may be implemented separately from the processing circuitry system and / or processor 140. For example, the sensor control circuitry may include, but is not limited to, a parameter adjustment unit (e.g., parameter adjustment circuitry system, parameter adjustment function, parameter adjustment module, etc.) for determining the type and settings of the sensor, a control unit (e.g., control circuitry system, control module, control function, etc.) for controlling the operation of the sensor, and a triggering unit (e.g., triggering, activating, etc.) for turning the sensor on / off (e.g., triggering, activating, etc.) based on values ​​received from the parameter adjustment unit and / or control unit, etc., to turn the sensor on / off (e.g., trigger, ignite, etc.).

[0074] The memory 150 may store data related to or relating to the operation of the electronic device 100. For example, the memory 150 may store reference images and environmental information for use in performing a method for obtaining a reference image according to at least one example embodiment.

[0075] In some example embodiments, memory 150 may include at least one of various volatile memories (such as dynamic random access memory (DRAM) or static random access memory (SRAM)) and / or at least one of various non-volatile memories (such as flash memory, phase-change random access memory (PRAM), resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), nanofloating gate memory (NFGM) or polymer random access memory (PoRAM).

[0076] In some example embodiments, the display driver 120, touch controller 125, object recognition sensor 130, processor 140, memory 150, and ambient sensor 160 may each be implemented using separate circuits / modules / chips. In other example embodiments, based on functionality, some of the display driver 120, touch controller 125, object recognition sensor 130, processor 140, memory 150, and ambient sensor 160 may be combined into a single circuit / module / chip, or may be further divided into multiple circuits / modules / chips.

[0077] Electronic device 100 can perform according to reference Figure 1 The described example embodiment describes a method for obtaining a reference image. For example, display panel 111 and display driver 120 can perform this. Figure 1 In operation S100, the object recognition sensor 130 can perform... Figure 1 Operation S200 in the process can be performed by environmental sensor 160. Figure 1 Operation S300 in the memory 150 can be executed. Figure 1The operation S400 in the process. Additionally, the electronic device 100 can perform operations as described in reference to... Figure 15 The method described for obtaining the reference image and / or referring to Figure 16 The methods described are for performing optical object recognition, but are not limited thereto.

[0078] In some example embodiments, at least some of the components included in the electronic device 100 may be omitted. For example, touch sensor panel 115 and touch controller 125 may be omitted when touch detection is not required.

[0079] Figure 5A , Figure 5B , Figure 5C and Figure 5D This is an illustration for describing a method of obtaining a reference image for optical object recognition according to some example embodiments.

[0080] In the following description, some example embodiments will be based on fingerprint recognition. However, the example embodiments are not limited thereto, and the example embodiments may be used or employed to identify one of a variety of objects, including objects related to non-biological characteristics (such as uniquely identifiable inanimate objects, etc.).

[0081] Figure 5A The image shown is based on reflected light that does not have a fingerprint (e.g., does not have the user's finger) or light reflected by an object used to obtain a reference image. Figure 5A The image can correspond to Figure 1 The reference image obtained by the object recognition sensor (e.g., a first reference image, a calibration image, etc.) may include only information about the internal structure of the display panel, but is not limited thereto.

[0082] Figure 5B The image is shown based on light reflected from an object (e.g., a user's finger, fingerprint, etc.). Figure 5B The image can also be obtained by an object recognition sensor and may include both fingerprint information and information about the internal structure of the display panel, but is not limited to this.

[0083] Figure 5C and Figure 5D Showing based on Figure 5A and Figure 5B Examples of pure object images (e.g., pure fingerprint images, pure target images, etc.) obtained from images. For example, images containing only pure interference components without fingerprints. Figure 5A The image can be obtained first, including both fingerprints and interfering components (e.g., unwanted image components, sensor images, etc.). Figure 5BThe image can then be obtained, and the pure fingerprint image (e.g., a pure object image and / or a pure target image, etc.) can subsequently be obtained by applying and / or performing compensation to remove interfering components. For example, a pure fingerprint image can be obtained by performing simple surface (or face) subtraction. Figure 5B Images and Figure 5A The difference between the images is obtained, but the example embodiments are not limited thereto.

[0084] like Figure 5C As shown, a pure fingerprint image in which interference components are completely removed can be obtained. However, as... Figure 5D As shown, even if the interfering component is not completely removed, at least one residual component RES may be retained or may be left behind. For example, when a user's finger is placed on an object recognition window, the interfering component may vary according to temperature and / or pressure. Specifically, deformations (e.g., rotation, scaling, translation, etc.) may occur. Figure 5B On the interference components in the image, therefore because Figure 5A Interference components in the image and Figure 5B There are spatial mismatches among the interfering components in the image, so residual components RES may appear, but are not limited to this.

[0085] To reduce and / or minimize such residual components RES, as referred to Figure 1 The reference image may be obtained in the same environment as an image including a fingerprint, and / or, as the reference... Figure 15 At least one of a plurality of images obtained sequentially or continuously can be obtained as a reference image.

[0086] Figure 6 and Figure 7 This illustrates the acquisition according to some example embodiments. Figure 1 A flowchart illustrating an example of a method using a reference image. (The text will omit the reference image.) Figure 1 Description of repeated elements.

[0087] Reference Figure 6 In the method for obtaining a reference image for optical object recognition according to at least one example embodiment, user touch input can be sensed via a touch sensor panel (operation S500). For example, Figure 6 The operation of S500 can be performed through Figure 4 The touch sensor panel 115 and touch controller 125 in the example implement this, but the example embodiment is not limited thereto.

[0088] When touch input is sensed (operation S500: Yes), operations S100, S200, S300 and / or S400 can be performed to drive some light sources and obtain and store a first reference image and first environmental information, etc., but the example embodiments are not limited thereto. Figure 6Operations S100, S200, S300 and / or S400 in the reference can be used with reference to Figure 1 The operations S100, S200, S300 and / or S400 are basically the same, but not limited thereto. For example, one or more operations may be performed simultaneously, may be performed in a different order than shown, may be omitted, and additional operations may be performed, etc.

[0089] When no touch input is sensed (Operation S500: No), the process can be terminated without obtaining the first reference image, but the operation is not limited to this. For example, the process can be repeated.

[0090] Reference Figure 7 In the method for obtaining a reference image for optical object recognition according to at least one example embodiment, it may be checked whether the display panel or electronic device is in and / or has an unused state (e.g., off state, inactive state, deactivated state, etc.) (operation S600). An unused state may indicate that the display panel is inactive, disabled and / or deactivated (e.g., not turned on), and may be referred to as an idle state or a power-off state, etc.

[0091] In some example embodiments, the display panel can be inspected by an electronic device to determine whether the display panel or the electronic device is and / or has been in an unused state, for example, based on whether a plurality of light sources included in the display panel are driven. However, the example embodiments are not limited to this. For example, the display panel or electronic device can be determined to be in an unused state when a number of light sources greater than a desired reference number (e.g., a desired threshold number, etc.) does not emit light, does not receive voltage from the display driver, and / or is turned off (e.g., when most of the light sources, except for some light sources set to correspond to an object recognition window, do not emit light and are turned off, etc.). However, the example embodiments are not limited to this. For example, the average number of light sources emitting light and / or being driven, etc., within a desired time period can be calculated, and the average number can be compared with a desired reference number to determine whether the display panel and / or electronic device is in an unused state, etc. In this example, operation S600 can be performed by... Figure 4 The display panel 111 and processor 140 in the example are used for execution, but the example embodiment is not limited thereto.

[0092] In other example embodiments, at least one environmental sensor may be used to check whether the display panel and / or electronic device is in an unused state. For example, the environmental sensor may include an accelerometer, and the state of the display panel and / or electronic device (e.g., whether the display panel or electronic device is in an unused state) may be checked based on acceleration information obtained from the accelerometer. In this example, operation S600 may be performed by... Figure 4The environmental sensor 160 and processor 140 perform this function. As another example, the environmental sensor may include a pressure sensor that detects the amount of pressure applied by the user to the display panel and / or electronic device, and / or the environmental sensor may include a camera that detects whether the user's face and / or eyes are facing the display panel to determine whether the electronic device and / or display panel is in an unused state, etc.

[0093] When the display panel and / or electronic device is in and / or has been in an unused state (operation S1600: Yes), operations S100, S200, S300 and / or S400 may be performed to drive some light sources and acquire and store a first reference image and first environmental information. However, the example embodiment is not limited to this. For example, one or more operations may be performed simultaneously, may be performed in a different order than shown, may be omitted, additional operations may be performed, etc. Typically, the unused state may be or correspond to a low-light environment with a small amount of external light (e.g., when the electronic device is in a pocket, bag, or at night). The reference image acquired in a low-light environment best represents information about the internal structure of the panel, etc., as an interference component. Figure 7 Operations S100, S200, S300, and S400 in the reference can be used as follows: Figure 1 The operations S100, S200, S300 and S400 are basically the same, but are not limited thereto.

[0094] When the display panel or electronic device is not in an unused state (e.g., not in an unused state and / or currently being used, etc.) (Operation S600: No), the process may be terminated without obtaining the first reference image, but the example embodiment is not limited thereto.

[0095] Figure 8A , Figure 8B , Figure 8C , Figure 8D , Figure 8E , Figure 8F and Figure 8G This is used to describe whether a display panel and / or electronic device, according to some example embodiments, has Figure 7 A diagram illustrating operations in an unused state. Figure 8A , Figure 8B , Figure 8C , Figure 8D , Figure 8E , Figure 8F and Figure 8G In the diagram, the horizontal axis represents time, and the vertical axis represents the output value of the accelerometer.

[0096] Figure 8A This displays continuous and / or sequential data obtained from a triaxial accelerometer. For example, Figure 8ALines v1, v2, and v3 in the diagram represent continuous and / or sequential data obtained from the three axes of the triaxial accelerometer, respectively. Figure 8B , Figure 8C , Figure 8D , Figure 8E , Figure 8F and Figure 8G The following are examples of implementations based on... Figure 8A The data in the system determines the current state of the electronic device. Figure 8B This indicates that the electronic device is in a static state (e.g., the electronic device is fixed and / or does not move). Figure 8C The image shows the user of the electronic device sitting down. Figure 8D This shows the state in which the user holds the electronic device in his or her hand. Figure 8E This shows the user of the electronic device is walking. Figure 8F This shows the status of the user going up and / or down a flight of stairs using the electronic device. Figure 8G This indicates that the user of the electronic device is running.

[0097] like Figure 8A , Figure 8B , Figure 8C , Figure 8D , Figure 8E , Figure 8F and Figure 8G As shown, the state in which the user is not using the electronic device can be determined as an unused state by comprehensively and / or systematically analyzing data from the accelerometer obtained based on the movement of the electronic device. Therefore, intermittent and / or periodic acquisition of reference images can be performed without inconveniencing the user in the unused state. In other words, reference images can be obtained by the electronic device even when it is not in use by the user. In some example embodiments, additional sensors (such as optical devices) can be used, if desired, to improve the accuracy of the determination.

[0098] Figure 9 This illustrates the acquisition according to at least one example embodiment. Figure 1 A flowchart of another example of the method using a reference image. (The text will be omitted.) Figure 1 Description of repeated elements.

[0099] Reference Figure 9 In the method for obtaining a reference image for optical object recognition according to at least one example embodiment, Figure 9 Operations S100, S200, S300, and S400 in the reference can be used as follows: Figure 1 The operations S100, S200, S300 and S400 are basically the same, but the example embodiments are not limited thereto.

[0100] According to at least one example embodiment, the electronic device can check whether the first reference image obtained in operation S300 is a contaminated image (operation S700). For example, a contaminated image may be an image that includes objects other than a first object identified as a target object by optical object identification (e.g., a unique pattern of the bag when the electronic device is in the bag), or it may be an image that includes information about an external light source other than the light source of the display device.

[0101] When the first reference image is not a contaminated image (operation S700: No), operation S400 can be performed to store the first reference image and the first environmental information. When the first reference image is a contaminated image (operation S700: Yes), the first reference image can be discarded, deleted, or scrapped without storing it in memory (operation S750), and the first environmental information can also be discarded. For example, as will be described later, an image including objects other than the first object can be removed by analyzing frequency components, and an image including information about an external light source can be removed by analyzing histograms, etc., but the example embodiments are not limited thereto.

[0102] In some example embodiments, the method for obtaining a reference image for optical object recognition can be combined. Figure 6 , Figure 7 and Figure 9 The example implementation may be carried out by at least two of the examples, but the example implementation is not limited to these.

[0103] Figure 10 This illustrates the checking of whether the first reference image is, according to at least one example embodiment. Figure 9 The flowchart shows an example of a contaminated image.

[0104] Reference Figure 9 and Figure 10 When the electronic device checks whether the first reference image is a contaminated image (operation S700), a first value can be obtained by performing spatial signal processing (or object detection processing in the spatial domain) on the first reference image (operation S710), but the example embodiment is not limited to this. For example, spatial signal processing may represent a scheme or method that directly uses pixel values ​​(e.g., pixel color values) or grayscale values, etc. For example, the first value may be a statistical value based on illumination (e.g., variance). However, the example embodiment is not limited to this, and at least one of various techniques may be used.

[0105] The electronic device can determine whether a first reference image is a contaminated image based on a first value and a desired and / or predetermined first threshold (e.g., a spatial domain threshold, a first spatial domain threshold, etc.). For example, when the first value is less than or equal to the first threshold (operation S720: No), the electronic device can determine that the first reference image is a normal reference image (operation S730); however, the example embodiment is not limited thereto. When the first value is greater than the first threshold (operation S720: Yes), the electronic device can determine that the first reference image is a contaminated image (operation S740), etc. For example, a contaminated image may be an image that includes information about an external light source, and an image that includes information about an external light source can be removed by analyzing a histogram, etc. Typically, an image contaminated by an external light source may have relatively high pixel values ​​compared to the pixel values ​​in an image received through an object recognition window after light emitted from a light source of the display device is reflected by the target object. Furthermore, a relatively large number of pixels with pixel values ​​greater than or equal to a desired and / or predetermined threshold may exist in the histogram of the obtained image. Therefore, when multiple pixels with pixel values ​​greater than or equal to the threshold are detected, the electronic device can determine that the obtained image is contaminated by an external light source. However, the example embodiments are not limited to this.

[0106] Figure 11A , Figure 11B , Figure 11C , Figure 11D and Figure 12 This is used to describe checking whether a first reference image is, according to at least one example embodiment. Figure 10 A diagram illustrating the operation of a contaminated image.

[0107] Reference Figure 11A and Figure 11B , Figure 11A A reference image including both noise and interference is shown. Figure 11B The diagram shows the representation of the pair. Figure 11A The image is the result of applying a differential filter and a signal processing filter. It can be seen that because the external light source is clearly captured and observed in the image, therefore... Figure 11A Images and Figure 11B Both images are contaminated by external light sources.

[0108] Reference Figure 11C and Figure 11D , Figure 11C A reference image including both noise and interference is shown. Figure 11D The diagram shows the representation of the pair. Figure 11C The image is the result of applying a differential filter and a signal processing filter. (Compared to...) Figure 11A and Figure 11B Unlike images, external light sources can be blocked and Figure 11C and Figure 11DThese are not observable in the image, therefore it can be seen that only noise and interference are well represented. Figure 11C In the reference image.

[0109] Reference Figure 12 , Figure 12 The examples shown are based on some example embodiments. Figure 11A and Figure 11C Image execution Figure 10 The result of operation S710. In Figure 12 In the graph shown, the horizontal axis FRAME represents the number of frames in the image, and the vertical axis SV represents the first value (e.g., the variance based on illumination). Figure 12 The first frame image F1 and the second frame image F2 in the image represent respectively Figure 11A Images and Figure 11C The image.

[0110] like Figure 12 As shown, the value SV11 obtained by performing spatial signal processing on the first frame image F1 can be greater than the first threshold TH1, therefore Figure 11A The image can be identified as a contaminated image. The value SV12 obtained by performing spatial signal processing on the second frame image F2 can be less than the first threshold TH1, therefore... Figure 11C The image can be identified as a normal reference image.

[0111] Figure 13 This illustrates the checking of whether the first reference image is, according to at least one example embodiment. Figure 9 The flowchart is another example of a contaminated image.

[0112] Reference Figure 9 and Figure 13 When checking whether the first reference image is a contaminated image (operation S700), the electronic device can obtain a second value by performing frequency signal processing (or object detection processing in the frequency domain) on the first reference image (operation S715). For example, frequency signal processing can represent a scheme or method (such as wavelet transform or Fourier transform, etc.) that transforms pixel values ​​(e.g., pixel color values) and / or grayscale values ​​into frequency values ​​(or frequency bands) and uses the transformed frequency values. For example, the second value can be the result of a Fourier transform, etc. However, the example embodiment is not limited thereto, and at least one of various techniques can be used.

[0113] The electronic device can determine whether a first reference image is a contaminated image based on a second value and a desired and / or predetermined second threshold (e.g., a frequency domain threshold, a second frequency domain threshold, etc.). For example, when the second value is less than or equal to the second threshold (operation S725: No), the first reference image can be determined to be a normal reference image (operation S730), but the example embodiment is not limited thereto. When the second value is greater than the second threshold (operation S725: Yes), the electronic device can determine that the first reference image is a contaminated image (operation S740), etc. For example, the second threshold may be different from the first threshold, but is not limited thereto. For example, a contaminated image may be an image that includes objects other than a first object (e.g., a target object), and images that include objects other than the first object can be removed by analyzing frequency components. Generally, when the signal strength in the frequency band is greater than or equal to a desired and / or predetermined threshold, the electronic device can determine that the obtained image is an image that includes objects, and such an image may include the portion of the user's object that is actually measured, and components other than the object signal are preserved in the form of the object.

[0114] Figure 14 This is used to describe checking whether a first reference image is, according to at least one example embodiment. Figure 13 A diagram illustrating the operation of a contaminated image.

[0115] Reference Figure 14 , Figure 14 Showing the Figure 11A and Figure 11C Image execution Figure 13 The result of operation S715 in [the context]. Figure 14 In the graph shown, the horizontal axis FRAME represents the number of frames in multiple images, and the vertical axis FTV represents a second value (e.g., the result of a Fourier transform).

[0116] like Figure 14 As shown, the value FTV11 obtained by performing frequency signal processing on the first frame image F1 can be greater than the second threshold TH2, therefore Figure 11A The image can be identified as a contaminated image by an electronic device. The value FTV12 obtained by performing frequency signal processing on the second frame image F2 can be less than the second threshold TH2, therefore... Figure 11C The image can be identified as a normal reference image by an electronic device.

[0117] although Figure 10 and Figure 13 The examples are described as individual examples, but the example embodiments are not limited thereto, and the methods for obtaining reference images for optical object recognition according to at least one example embodiment can be combined. Figure 10 Examples and Figure 13Examples of both are implemented. For instance, the first value can be obtained by performing spatial signal processing on the first reference image, the second value can be obtained by performing frequency signal processing on the first reference image, and the electronic device can determine whether the first reference image is a contaminated image based on all of the first value, the first threshold, the second value, and the second threshold. For example, the first reference image can be determined to be a contaminated image when the first value is greater than the first threshold or when the second value is greater than the second threshold. The first reference image can be determined to be a normal reference image when the first value is less than or equal to the first threshold and when the second value is less than or equal to the second threshold.

[0118] In some example embodiments, when a reference image is obtained, it can be updated by performing signal processing in the form of Finite Impulse Response (FIR) or Infinite Impulse Response (IIR) instead of using only a single image. In the FIR example, a high-quality (or high-performance) object-free image can be obtained by storing and using a specific number of object-free images. In the IIR example, a high-quality object-free image can be updated and used whenever a new object-free image is input. In the case of the FIR example, object-free images in a specific environment can be collected to obtain images improved and / or optimized for that specific environment. In the case of the IIR example, updates can be performed slowly, but the accumulated information can be widely applied.

[0119] Figure 15 This is a flowchart illustrating a method for obtaining a reference image for optical object recognition according to some example embodiments. (The remaining text is omitted.) Figure 1 Repeated description.

[0120] Reference Figure 15 In the method for obtaining a reference image for optical object recognition according to the example embodiment, operation S2100 may be combined with... Figure 1 The operation S100 is basically the same.

[0121] When several light sources are driven, multiple images are acquired sequentially, periodically, and / or continuously based on reflected light received by an object recognition sensor through an object recognition window (operation S2200). This is in contrast to acquiring only one image at a time. Figure 1 The examples are different, in Figure 15 In the example, the object recognition sensor can sequentially, periodically, and / or continuously acquire some images corresponding to the object recognition window and / or any object based on reflected light.

[0122] At least one of multiple images is obtained as a first reference image for optical object recognition associated with a first object (operation S2300). When some light sources are driven, at least one first environmental information is obtained using at least one environmental sensor (operation S2400). The first environmental information is related to the surrounding environment driving some light sources (e.g., the environment around electronic devices and / or display panels, etc.). The first reference image and the first environmental information for the first reference image are stored together (operation S2500). Therefore, a desired, best, most suitable, and / or optimal reference image can be selected from multiple images. Operations S2400 and S2500 can be respectively connected to... Figure 1 Operations S300 and S400 are basically the same.

[0123] In some example embodiments, the first reference image may be an image that does not include the first object. Alternatively, the plurality of images may include a first image that includes the first object, and the first reference image may be an image obtained immediately before or immediately after the first image from a plurality of images obtained sequentially, periodically, and / or consecutively. Generally, the most suitable reference image obtained in the same environment as the first image may be an image taken immediately before or after the first image, but the example embodiments are not limited thereto.

[0124] In some example embodiments, it is possible to Figure 6 Operation S500 in Figure 7 Operation S600 and Figure 9 At least one of the operations in S700 is added to Figure 15 This is an example, but the example embodiment is not limited to this.

[0125] In a method for obtaining a reference image for optical object recognition according to at least one example embodiment, an object-free image or an image without objects (e.g., a reference image, a calibration image, etc.) for performing optical object recognition can be efficiently obtained. This optical object recognition obtains only object-related pure information by subtracting the object-free image from the image including the object. Furthermore, the object-free image and the image including the object can be selected from multiple images obtained sequentially in the same environment. Therefore, robust object image recovery can be performed, or object-related pure information can be robustly recovered based on the object-free image, and efficient optical object recognition can be supported.

[0126] Figure 16 This is a flowchart illustrating a method for performing optical object recognition according to some example embodiments. (The remaining text is omitted.) Figure 1 Description of repeated elements.

[0127] Reference Figure 16In the method for performing optical object recognition according to some example embodiments, operation S3100 may be combined with Figure 1 The operation S100 is basically the same.

[0128] When driving several light sources, multiple reference images and multiple environmental information corresponding to the multiple reference images are obtained using an object recognition sensor and at least one environmental sensor (operation S3200). Each of the multiple reference images is an image excluding a first object. Operation S3200 may include... Figure 1 Operations S200, S300, and S400 are described in the examples. According to some example embodiments, all of the multiple reference images may be obtained under different environmental conditions, or some of the multiple reference images may be obtained under the same environmental conditions, and / or the reference images obtained under the same environmental conditions may be combined and updated into a single reference image by performing signal processing such as FIR or IIR.

[0129] When driving several light sources, a first image including a first object and corresponding current environmental information are obtained using an object recognition sensor and at least one environmental sensor (operation S3300). A first reference image is selected from multiple reference images based on the current environmental information and multiple environmental information (operation S3400). A first effective image for the first object is obtained based on the first image and the first reference image (operation S3500). Since a reference image among the multiple reference images that is desirable, best, most suitable, etc., for the first image can be selected as the first reference image, efficient optical object recognition can be performed.

[0130] Figure 17 This illustrates a selection based on at least one example embodiment. Figure 16 A flowchart of an example of the first reference image in the diagram.

[0131] Reference Figure 16 and Figure 17 When the first reference image is selected (operation S3400), the electronic device can check whether there is environmental information that matches the current environmental information among the multiple previously obtained environmental information (operation S3410).

[0132] When a first environmental information matching the current environmental information exists (Operation S3410: Yes), the electronic device can select the first environmental information (Operation S3420). When no environmental information that completely matches the current environmental information exists among the previously obtained multiple environmental information (Operation S3410: No), the electronic device can select the second environmental information that is closest to the current environmental information among the multiple environmental information (Operation S3430). A reference image corresponding to the selected environmental information can be selected as the first reference image (Operation S3440).

[0133] although Figure 16 An example including multiple reference images is shown, but the example embodiment is not limited thereto. For example, the example embodiment can also be applied to an example including only one reference image, and Figure 16 Operations such as S3400 can be omitted.

[0134] Figure 18 This illustrates the acquisition according to at least one example embodiment. Figure 16 A flowchart of an example of the first valid image in the diagram.

[0135] Reference Figure 16 and Figure 18 When the first valid image is obtained (operation S3500), the electronic device can obtain the first valid image by subtracting the first reference image from the first image (operation S3510).

[0136] Figure 19A and Figure 19B This is a diagram illustrating a method for performing optical object recognition according to some example embodiments.

[0137] Reference Figure 19A and Figure 19B The image shows an image with differential filters and signal processing filters applied. Figure 19A The results are shown using a reference image obtained in an environment different from the object image (e.g., an environment with a relatively large temperature difference). Figure 19B The results of using reference images obtained in the same environment as the object image according to some example embodiments are shown.

[0138] Although Figure 19A The interference grid pattern is severe, but in Figure 19B Interference and noise are removed, allowing only the fingerprint to be clearly preserved. The enhanced accuracy of the obtained image has the effect of distinguishing and / or improving the ridges and valleys of the fingerprint. Figure 19A In this process, regardless of fingerprint quality, interference and / or noise are generated in the form of multiple lines, and the intersections between these lines and the fingerprint can be incorrectly identified as fingerprint characteristics, thus significantly reducing and / or decreasing the performance and / or accuracy of fingerprint matching, and potentially leading to confusion with other people's fingerprints. Conversely, in Figure 19B In this process, superior and / or improved fingerprint images that are adapted to changes in the environment can be obtained.

[0139] As those skilled in the art will understand, various exemplary embodiments of the inventive concept can be implemented as systems, methods, computer program products, and / or computer program products implemented in one or more non-transitory computer-readable media having computer-readable program code implemented thereon. The computer-readable program code can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. The computer-readable medium can be a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium can be any tangible medium that can contain or store programs for use by or in connection with an instruction execution system, apparatus, or device, such as an optical disk, hard disk drive, solid-state drive, RAM, ROM, etc.

[0140] Figure 20 This is a block diagram illustrating an electronic device according to some example embodiments.

[0141] Reference Figure 20 The electronic device 1000 may include processing circuitry, which includes at least one processor 1010, a memory device 1020, at least one object recognition sensor 1030, etc. The electronic device 100 may also include at least one input / output (I / O) device 1040, a power supply 1050, and / or a display device 1060, etc., but the example embodiment is not limited thereto. The electronic device 100 may also include multiple ports for communicating with video cards, sound cards, memory cards, universal serial bus (USB) devices, other electronic devices, etc.

[0142] Processor 1010 controls the operation of electronic device 1000. Processor 1010 can execute at least one operating system and at least one application to provide an internet browser, game, or video, etc. Memory device 1020 can store data for the operation of electronic device 1000. I / O device 1040 may include input devices (such as keyboard, keypad, mouse, touchpad, touch screen, remote control, etc.) and output devices (such as printer, speaker, display, etc.). Power supply 1050 provides power for the operation of electronic device 1000.

[0143] The display device 1060 includes, but is not limited to, a display panel and / or a touch sensor panel. Figure 17 The display panel, touch sensor panel, object recognition sensor 1030, processor 1010, and memory device 1020 can respectively correspond to Figure 4 The device includes a display panel 111, a touch sensor panel 115, an object recognition sensor 130, a processor 140, and a memory 150, and is capable of performing a method for obtaining a reference image according to at least one example embodiment and a method for performing optical object recognition according to at least one example embodiment.

[0144] Various exemplary embodiments of the inventive concept can be applied to a wide range of electronic devices and systems, including display panels, object recognition sensors, and those performing optical object recognition. For example, at least one exemplary embodiment of the inventive concept can be applied to systems such as mobile phones, smartphones, tablet computers, laptop computers, personal digital assistants (PDAs), portable multimedia players (PMPs), digital cameras, portable game consoles, music players, camcorders, video players, navigation devices, wearable devices, Internet of Things (IoT) devices, Internet of Everything (IoE) devices, e-book readers, virtual reality (VR) devices, augmented reality (AR) devices, robotic devices, drones, and the like.

[0145] The foregoing is illustrative of various exemplary embodiments and should not be construed as limiting the exemplary embodiments. Although some exemplary embodiments have been described, those skilled in the art will readily understand that many modifications are possible in the exemplary embodiments without substantially departing from the novel teachings and advantages of the exemplary embodiments. Therefore, all such modifications are intended to be included within the scope of the exemplary embodiments as defined in the claims. It should be understood that the foregoing is illustrative of various exemplary embodiments and should not be construed as limiting oneself to the specific exemplary embodiments disclosed, and modifications to the disclosed exemplary embodiments, as well as other exemplary embodiments, are intended to be included within the scope of the appended claims.

Claims

1. A method for obtaining a reference image for optical object recognition, the method comprising: The driver includes a subset of light sources from a plurality of light sources in at least one display panel, the subset of light sources corresponding to an object recognition window that is a local area of ​​the display panel; An object recognition sensor is used to receive light reflected through an object recognition window, the light being emitted by a subset of the light source; When a subset of the light sources is driven, multiple reference images are obtained based on reflected light using an object recognition sensor, each of the multiple reference images being an image that does not include a first target object, which is the target of optical object recognition; When a subset of the light sources is driven, at least one environmental sensor is used to obtain multiple environmental information corresponding to the multiple reference images, the multiple environmental information being related to the surrounding environment; When a subset of the light sources is driven, a first target image associated with a first target object is obtained based on the reflected light; When a subset of the light sources is driven, the current environmental information corresponding to the first target image is obtained using the at least one environmental sensor; Based on the current environmental information and the multiple environmental information, a first reference image is selected from the multiple reference images; Store the first reference image and the first environmental information together; and Based on the first target image and the first reference image, a first effective image is obtained for optical object recognition associated with the first target object. The first reference image is an image showing the internal structure of the display panel, and The step of selecting the first reference image includes: selecting first environmental information that matches the current environmental information from among the plurality of environmental information; and selecting the first reference image corresponding to the first environmental information.

2. The method according to claim 1, wherein, The first environmental information includes at least one or any combination of temperature information, humidity information, pressure information, motion information, time information, spatial information, illuminance information, acceleration information, vibration information, external force information, and impact information.

3. The method according to claim 1, further comprising: The user's touch input is sensed via a touch sensor panel; and In response to touch input being sensed: A subset of the light sources is driven, and The first reference image and the first environmental information are obtained and stored.

4. The method according to claim 1, further comprising: Determine whether the display panel or the electronic device including the display panel is inactive; and In response to the display panel or electronic device being inactive, a subset of the light sources is driven, and a first reference image and first environmental information are acquired and stored.

5. The method according to claim 4, wherein, The steps to determine whether a display panel or electronic device is inactive include: Determine whether the plurality of light sources included in the display panel are driven; and Based on the result of determining whether the plurality of light sources included in the display panel are driven, it is determined whether the display panel or electronic device is in an inactive state.

6. The method according to claim 4, wherein, The at least one environmental sensor includes an acceleration sensor; and The step of determining whether a display panel or electronic device is inactive is based on acceleration information obtained from an accelerometer.

7. The method according to claim 1, further comprising: Determine whether the first reference image is a contaminated image; and In response to the first reference image being a contaminated image, the first reference image is discarded without being stored.

8. The method according to claim 7, wherein, The steps to determine whether the first reference image is a contaminated image include: The first value is obtained by performing spatial signal processing on the first reference image; and The first reference image is determined to be a contaminated image based on the first value and the expected first threshold.

9. The method according to claim 8, wherein, The step of determining whether the first reference image is a contaminated image includes: determining that the first reference image is a contaminated image in response to a first value being greater than a first threshold.

10. The method according to claim 7, wherein, The steps to determine whether the first reference image is a contaminated image include: The second value is obtained by performing frequency signal processing on the first reference image; and The first reference image is determined to be contaminated based on the second value and the expected second threshold.

11. The method according to claim 10, wherein, The step of determining whether the first reference image is a contaminated image includes: determining that the first reference image is a contaminated image in response to a second value being greater than a second threshold.

12. The method according to claim 7, wherein, A contaminated image is an image that includes objects other than the first target object, or an image that includes information about external light sources other than the plurality of light sources included in the display panel.

13. The method according to any one of claims 1 to 12, wherein, The display panel includes a first surface on which an image is displayed and a second surface opposite to the first surface; and The object recognition sensor is located below the second surface of the display panel.

14. A method for performing optical object recognition, the method comprising: The driver includes a subset of multiple light sources in the display panel, the subset of light sources corresponding to an object recognition window that is a local area of ​​the display panel; When a subset of the light source is driven, a plurality of reference images are obtained using an object recognition sensor, each of the plurality of reference images being an image that does not include the first target object; When a subset of the light sources is driven, at least one environmental sensor is used to obtain multiple environmental information corresponding to the multiple reference images; When a subset of the light sources is driven, a first target image including a first target object is obtained using an object recognition sensor; When a subset of the light sources is driven, environmental sensors are used to obtain current environmental information corresponding to the first target image; Based on the current environmental information and the multiple environmental information, a first reference image is selected from the multiple reference images; and Based on the first target image and the first reference image, a first effective image for the first target object is obtained. The plurality of reference images are images representing the internal structure of the display panel, and The step of selecting the first reference image includes: selecting first environmental information that matches the current environmental information from among the plurality of environmental information; and selecting the first reference image corresponding to the first environmental information.

15. The method according to claim 14, wherein, The step of selecting the first reference image also includes: Determine whether the environmental information among the plurality of environmental information matches the current environmental information; In response to the absence of environmental information matching the current environmental information, the second environmental information that is closest to the current environmental information is selected from the plurality of environmental information; and The second reference image corresponding to the selected second environmental information is selected as the first reference image.

16. The method according to any one of claims 14 to 15, wherein, The steps to obtain the first valid image include: The first effective image is obtained by subtracting the first reference image from the first target image.

Images