Method for executing half shutter function and method for capturing image using the same

Abstract

A method for performing a half-shutter function using optical object recognition is provided, the method comprising: displaying a shutter button on a display panel by driving a portion of a plurality of light sources included in the display panel, the portion of the plurality of light sources being arranged corresponding to an object recognition window of the display panel; obtaining a first image based on reflected light received by an object recognition sensor through the object recognition window while driving the portion of the plurality of light sources; determining whether the first image includes a first object, the first object having a half-shutter state; and performing a half-shutter process based on the determination that the first image includes the first object having the half-shutter state.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of Korean Patent Application No. 10-2019-0160362 filed in the Korean Intellectual Property Office on December 5, 2019, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Example embodiments relate generally to semiconductor integrated circuits, and more particularly to a method of performing a half-shutter function using optical object recognition, and a method of capturing an image using the method of performing the half-shutter function. Background Art

[0004] Recently, electronic devices dedicated to image capture (e.g., digital single-lens reflex (DSLR) cameras) and electronic devices equipped with various other functions as well as the image capture function (e.g., smart phones) have been widely used. The electronic device dedicated to image capture includes a physical shutter button, and the user performs image capture by fully pressing the shutter button (e.g., fully pressing the shutter button), and performs a half shutter function by lightly pressing the shutter button (e.g., half-pressing the shutter button). In recently released electronic devices such as smart phones, the physical shutter button may not be included to increase the size of the display and reduce the thickness of the electronic device, and image capture is performed using a touch sensor instead of a physical shutter button. In this case, in electronic devices such as smart phones, it is impossible to predict when a full shutter input is provided, and there is a problem that it is difficult to provide a half shutter input that is distinguished from a full shutter input. Summary of the invention

[0005] At least one example embodiment of the present disclosure provides a method of performing a half-shutter function capable of identifying a half-shutter input using optical object recognition without a physical shutter button.

[0006] At least one example embodiment of the present disclosure provides a method of capturing an image using a method of performing a half shutter function.

[0007] According to an aspect of an example embodiment, a method for performing a half-shutter function using optical object recognition is provided. In the method, a shutter button is displayed on a display panel by driving a portion of a plurality of light sources included in the display panel, the portion of the plurality of light sources being arranged corresponding to an object recognition window of the display panel. While driving the portion of the plurality of light sources, a first image is obtained based on reflected light received by an object recognition sensor through the object recognition window. It is determined whether the first image includes a first object, the first object having a half-shutter state. Based on the determination that the first image includes the first object having the half-shutter state, a half-shutter process is performed.

[0008] According to an aspect of an example embodiment, a method for capturing an image is provided. In the method, a shutter button is displayed on a display panel by driving a portion of a plurality of light sources included in the display panel, the portion of the plurality of light sources being set to correspond to an object recognition window of the display panel. While driving the portion of the plurality of light sources, a plurality of images are continuously obtained based on reflected light received by an object recognition sensor through the object recognition window. A half shutter function is performed based on a half shutter input, the half shutter input being detected based on a first image among a plurality of images. An image capture function is performed based on a full shutter input. Executing the half shutter function includes: determining whether a first image includes a first object, the first object having a half shutter state; and executing a half shutter process based on the half shutter state of the first object included in the first image.

[0009] According to one aspect of an exemplary embodiment, a method for performing a half shutter function using optical object recognition is provided. In the method, an image capture application providing an image capture function is run and displayed on a display panel, the display panel including a plurality of light sources. A shutter button is displayed on the display panel by driving a portion of the plurality of light sources, the portion of the plurality of light sources being set to correspond to an object recognition window of the display panel. Before receiving a full shutter input, a plurality of images are continuously obtained based on reflected light received by an object recognition sensor through the object recognition window while driving the portion of the plurality of light sources. It is determined that a first image among the plurality of images includes a first object having a half shutter state. A half shutter process is performed based on the half shutter state of the first object included in the first image. It is determined that the first image includes a first object having a half shutter state based on a result of performing at least one of spatial signal processing and frequency signal processing on the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Illustrative, non-limiting example embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0011] Figure 1 is a flowchart illustrating a method of performing a half-shutter function using optical object recognition according to example embodiments.

[0012] Figure 2 is a plan view of an electronic device according to example embodiments.

[0013] Figure 3A and Figure 3B is along Figure 2 A cross-sectional view of an example of an electronic device taken along line AA'.

[0014] Figure 4 and Figure 5 It is shown Figure 2 A block diagram of an example of an electronic device.

[0015] Figure 6 It is shown Figure 1 Flowchart of an example of operations S300 and S400 in FIG.

[0016] Fig. 7A , Figure 7B , Figure 7C , Figure 8 and Fig. 9 Is used to describe Figure 6 Graph of the operation.

[0017] Fig.10 and Fig.11 It is shown Figure 1 Flowchart of an example of operations S300 and S400 in FIG.

[0018] Fig.12 and Fig.13 Is used to describe Fig.11 Graph of the operation.

[0019] Fig.14 and Fig.15 It is shown Figure 1 Flowchart of an example of operations S300 and S400 in FIG.

[0020] Fig.16 Is used to describe Fig.15 Graph of the operation.

[0021] Fig.17 It is shown Figure 1 Flowchart of an example of operations S300 and S400 in FIG.

[0022] Fig.18 and Fig.19 is a flowchart illustrating a method of performing a half-shutter function using optical object recognition according to example embodiments.

[0023] Fig. 20 is a flowchart illustrating a method of capturing an image according to example embodiments.

[0024] Fig.21A , Fig. 21B , Fig. 21C and Fig. 22 It is used to describe the capture Fig. 20 Figure 4. Method of image processing.

[0025] Fig.23 is a block diagram illustrating an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0026] Various example embodiments will be described more fully with reference to the accompanying drawings showing example embodiments. However, the present disclosure can be embodied in many different forms and should not be construed as being limited to the example embodiments set forth herein. Throughout this application, like reference numerals refer to like elements.

[0027] Figure 1 is a flowchart illustrating a method of performing a half-shutter function using optical object recognition according to example embodiments.

[0028] refer to Figure 1 The method of performing a half shutter function using optical object recognition according to an example embodiment may be performed by an electronic device including a display panel and an object recognition sensor, and when the electronic device captures or shoots an image, the half shutter function may be performed. The display panel may include a plurality of light sources, and the object recognition sensor may perform optical object recognition by using light provided by the plurality of light sources. Figures 2 to 5 Describe the detailed configuration of the electronic device.

[0029] In a method for performing a half shutter function using optical object recognition according to an exemplary embodiment, a shutter button may be displayed on a display panel by driving some of a plurality of light sources (or a portion of a plurality of light sources) included in the display panel (operation S100). A portion of the plurality of light sources is set to correspond to an object recognition window which is a partial area of ​​the display panel. According to an exemplary embodiment, all of the light sources in a portion of the plurality of light sources may emit light to have the same grayscale value, or a portion of the plurality of light sources may emit light to have different grayscale values. The shutter button is displayed on a partial area of ​​the display panel (e.g., on the object recognition window). For example, the shutter button may be displayed together with an image capture application which provides an image capture function when executed. In an exemplary embodiment, as will be described with reference to FIG. Fig. 20 As described, an operation of executing an image capturing application may be performed before operation S100.

[0030] While driving some of the plurality of light sources, a first image may be obtained or acquired based on the reflected light received by the object recognition sensor through the object recognition window (operation S200). According to example embodiments, only one first image may be obtained, or after a plurality of images are obtained continuously, one of the plurality of images may be selected as the first image, such as referring to Fig. 20For example, all light sources of a portion of the plurality of light sources may be turned on substantially simultaneously or concurrently. Light generated from a portion of the plurality of light sources may be emitted through the object recognition window and may be reflected by any object (or any object) positioned corresponding to the object recognition window, and the reflected light may be provided to the object recognition sensor. Therefore, the object recognition sensor may obtain an image corresponding to the object recognition window and / or the object based on the reflected light.

[0031] It is determined whether the first image includes a first object that provides a full shutter input (or shutter input) or a half shutter input (operation S300). When it is determined that the first image includes the first object (operation S300: yes), it is determined whether the first object in the first image has (or is in) a half shutter state for providing a half shutter input (operation S400). When it is determined that the first object in the first image has a half shutter state (operation S400: yes), a half shutter process is driven based on the half shutter state (operation S500).

[0032] In some example embodiments, the full shutter input may be an input for performing image capture (e.g., an image capture function or process), and the half shutter input may be an input for performing a half shutter process associated with image capture before performing image capture. However, example embodiments are not limited thereto, and the half shutter input may include any input received before receiving the full shutter input as long as a half shutter state distinguished from a full shutter state is recognized.

[0033] In some example embodiments, the half shutter process may include at least one of focus adjustment, exposure adjustment, white balance adjustment, gain adjustment, low power consumption, and control / command for continuous image capture. However, example embodiments are not limited thereto, and the half shutter process may include at least one of various functions performed before shooting (or capturing) an image, and achieve zero shutter lag with low power consumption. For example, the various functions included in the half shutter process may be functions that can not only improve the image quality of an image to be captured by a subsequent shutter input by performing a preparatory operation for image capture, but also reduce power consumption by performing a preparatory operation with low power consumption.

[0034] In some example embodiments, the first object may be a finger of a user including a fingerprint. In this example, the object recognition window and the object recognition sensor may be a fingerprint recognition window and a fingerprint recognition sensor, respectively. However, example embodiments are not limited thereto, and the first object may be one of various objects for which full shutter input and half shutter input may be distinguished, identified, and / or recognized, such as a gloved finger, an object including a specific pattern, and the like.

[0035] When the first image does not include the first object (operation S300: No), or when the first object in the first image does not have (or is not in) a half-shutter state (operation S400: No), the half-shutter process may not be driven, and operations S200, S300, and S400 may be repeatedly performed until an image including the first object having a half-shutter state is obtained.

[0036] In the method for performing the half shutter function according to the example embodiment, a half shutter input for performing the half shutter process can be provided or recognized before pressing the shutter button by detecting a specific situation based on the object recognition sensor and the optical object recognition and by recognizing the specific situation as a half shutter state. Therefore, the half shutter input can be effectively recognized without the input of the physical shutter button, and zero shutter lag can be effectively achieved with low power consumption by performing various functions (e.g., focus adjustment, exposure adjustment, gain adjustment, control / command for continuous image capture, etc.) before capturing an image, without reducing the size of the display in the electronic device and / or increasing the thickness of the electronic device in order to provide a physical shutter button. Therefore, the half shutter function according to the example embodiment can be provided while the size of the display is increased and the thickness of the device is reduced.

[0037] Figure 2 is a plan view of an electronic device according to example embodiments.

[0038] refer to Figure 2 , the electronic device 100 includes a panel 110 to interface with a user. The user of the electronic device 100 can view information output from the electronic device 100 through the panel 110. The user of the electronic device 100 can input a signal or command to the electronic device 100 through the panel 110. For example, the panel 110 may include a display panel configured to output visual information to the user, a touch sensor panel configured to sense a touch input of the user, etc.

[0039] An object recognition window ORW may be provided on the panel 110. Figure 3A and Figure 3B As described above, the object recognition sensor for detecting the object may be disposed at a position spatially corresponding to the position of the object recognition window ORW. Figure 2 The object recognition window ORW is shown as a circle in FIG. 1 or as a rectangle in other figures, but the shape of the object recognition window ORW may be variously changed according to example embodiments.

[0040] In addition, the electronic device 100 may include an image capture module 102 (e.g., a camera module or a camera). When an image capture application 104 (e.g., a camera application) is executed, image information of an object (e.g., a person) received through the image capture module 102 may be displayed on the panel 110 (e.g., on the display panel), and a shutter button SB may be displayed at a position corresponding to the position of the object recognition window ORW. Figure 2 It is illustrated that one image capturing module 102 is disposed on the front surface of the electronic device 100 , but example embodiments are not limited thereto, and a plurality of image capturing modules may be disposed on at least one of the front and rear surfaces of the electronic device 100 .

[0041] In some example embodiments, the reference may be provided or received based on a scheme of sensing a user's touch input to the shutter button SB using the panel 110 (e.g., a touch sensor panel) or a scheme of recognizing an object on the object recognition window ORW using an object recognition sensor. Figure 1 Describes the full shutter input.

[0042] In some example embodiments, the reference may be provided or received based on a scheme of recognizing an object on the object recognition window ORW using an object recognition sensor. Figure 1 The half shutter input described. The half shutter input may represent an action performed before providing or receiving a full shutter input. The user uses the shutter button SB to provide a full shutter input, which may be similar to the user experience when using an existing camera shutter button (e.g., a physical camera shutter button), and may be effective. Since the shutter button SB is illuminated to indicate its position on the display panel, the user may experience the same experience as an existing camera interface.

[0043] In some example embodiments, the electronic device 100 may include any mobile system, such as a mobile phone, a smart phone, a tablet computer, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a digital camera, a portable game console, a music player, a portable camcorder, a video player, a navigation device, a wearable device, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, an e-book reader, a virtual reality (VR) device, an augmented reality (AR) device, a robotic device, a drone, or the like.

[0044] Example embodiments may provide an interface that can be used to detect an object. For example, when a user contacts or approaches panel 110, a function of detecting a finger or fingerprint may be performed. According to example embodiments, an interface for object detection and an object recognition sensor interface may share an area on electronic device 100 with panel 110 (e.g., an area on panel 110 may be used as an interface and an object recognition sensor), so the interface and object recognition sensor may not require an additional area on electronic device 100. Therefore, the size of electronic device 100 may be reduced, or the free area may be used for other purposes.

[0045] Figure 3A and Figure 3B is along Figure 2 A cross-sectional view of an example of an electronic device taken along line AA'.

[0046] refer to Figure 3A and Figure 3B , in the object recognition mode, the object recognition window ORW may be displayed on a partial area (or portion) of the panel 110. For example, as shown in FIG. Figure 2 As described, when the image capturing application 104 is executed, the object recognition window ORW for recognizing the full shutter input and / or the half shutter input may be displayed, and the shutter button SB may be displayed together at a position corresponding to the position of the object recognition window ORW.

[0047] The panel 110 may include a display panel 111 and a touch sensor panel 115 .

[0048] The display panel 111 may include a plurality of light sources 112. Figure 4 and Figure 5 As described above, the plurality of light sources 112 may be included in the plurality of pixels included in the display panel 111. Among the plurality of light sources 112, only a portion 113 of the plurality of light sources disposed to correspond to the object recognition window ORW may be driven substantially simultaneously in the object recognition mode. Figure 3A and Figure 3B In FIG. 1 , a portion 113 of the plurality of light sources that are driven and emit light is indicated by a shaded portion.

[0049] The object recognition sensor 130 may be disposed below the panel 110 so that the object recognition sensor 130 may overlap the object recognition window ORW in a vertical direction. In other words, the panel 110 may include a first surface on which an image is displayed and a second surface opposite to the first surface, and the object recognition sensor 130 may be disposed below the second surface of the panel 110.

[0050] The object recognition sensor 130 may include a lens 132 and an image sensor 134. The lens 132 may be disposed below the panel 110 (e.g., between the panel 110 and the image sensor 134), and may concentrate reflected light received through the object recognition window ORW on the image sensor 134. The image sensor 134 may be disposed below the lens 132, and may generate an image signal for an object on a partial area based on the reflected light concentrated by the lens 132. In some example embodiments, the lens 132 may be omitted in the object recognition sensor 130.

[0051] For example, in which Figure 3A In the case of fingerprint detection where the user places the finger 10 on the object recognition window ORW as shown, the light generated from the part 113 of the plurality of light sources within the object recognition window ORW may be reflected by the fingerprint of the finger 10, and the reflected light of the fingerprint may be provided to the object recognition sensor 130. The object recognition sensor 130 may capture an image signal of the fingerprint or information associated with the shape of the fingerprint (e.g., a fingerprint image) based on the reflected light of the fingerprint received through the object recognition window ORW.

[0052] In another example, where Figure 3A In the first case where the user directly touches the finger 10 to the object recognition window ORW, or in the case where Figure 3B As shown in the second case where the finger 10 is spaced a certain distance from the object recognition window ORW, the reflected light generated from the part 113 of the multiple light sources and reflected by the finger 10 in the first case may be different from the reflected light generated from the part 113 of the multiple light sources and reflected by the finger 10 in the second case, and thus the image signal of the finger 10 captured by the object recognition sensor 130 in the first case may be different from the image signal of the finger 10 captured by the object recognition sensor 130 in the second case. By recognizing the above differences, full shutter input and half shutter input can be distinguished and identified. For example, Figure 3A The first case can be identified as a full shutter input, and Figure 3B The second situation can be recognized as half shutter input.

[0053] Despite Figure 3A and Figure 3B Although not shown in the figure, the object recognition sensor 130 may further include a filter for adjusting the frequency characteristic and / or polarization characteristic of the reflected light to be provided to the image sensor 134.

[0054] Figure 4 and Figure 5 It is shown Figure 2 A block diagram of an example of an electronic device.

[0055] refer to Figure 4 , the electronic device 100 includes a panel 110 and an object recognition sensor 130. The panel 110 may include a display panel 111 and a touch sensor panel 115. The electronic device 100 may further include a display driver 120, a touch controller 125, a processor 140, a memory 150, a camera processor 160, and a camera 170.

[0056] The touch sensor panel 115 may sense contact or proximity of an object (e.g., a user's finger) to the touch sensor panel 115. The touch sensor panel 115 may generate a sensing signal in response to the contact or proximity of the object. The touch sensor panel 115 may include a plurality of sensing capacitors formed along rows and columns. Figure 4 An example sensing capacitor CS is shown. The capacitance value of the sensing capacitor may change in response to contact or proximity of an object.

[0057] The touch controller 125 may control the operation of the touch sensor panel 115. The touch controller 125 may process an operation associated with contact or proximity of an object based on a sensing signal output from the touch sensor panel 115. For example, the touch controller 125 may recognize contact or proximity of an object based on a change in the capacitance value of a sensing capacitor. For example, when the sensing signal is associated with the operation or running of a specific application, the touch controller 125 may output a command to the processor 140 so that the specific application is run or operated.

[0058] The display panel 111 outputs visual information to a user. The display panel 111 may include a plurality of pixels arranged along rows and columns to display an image. Figure 4 One example pixel PX is shown. Each pixel may be configured to emit light of a particular color that forms an image. When multiple pixels emit light together, the display panel 111 may display a desired or intended image.

[0059] In some example embodiments, the display panel 111 may be an electroluminescent display panel. The electroluminescent display panel may be driven with a fast response speed and low power consumption using a light emitting diode (LED) or an organic light emitting diode (OLED) that generates light through the recombination of electrons and holes. Compared with a liquid crystal display panel using a backlight unit, the pixels of the electroluminescent display panel may emit light by themselves, and the reflected light received through the object recognition window ORW on the touch sensor panel 115 (or the object recognition window ORW' on the display panel 111) may be provided to the object recognition sensor 130 below the display panel 111 through the space (or interval) between the pixels. Therefore, according to an example embodiment, the light emitting diode or organic light emitting diode included in the pixel may correspond to the light source included in the display panel. However, the example embodiment is not limited thereto, and the display panel 111 may be any display panel having a structure in which the reflected light received through the object recognition window ORW or ORW' may be provided to the object recognition sensor 130.

[0060] The display driver 120 may control the operation of the display panel 111 and may drive the display panel 111. For example, the display driver 120 may appropriately drive each pixel of the display panel 111 in response to a command of the processor 140 so that a desired or intended image is displayed on the display panel 111. For example, the display driver 120 may partially drive the display panel 111 so that the pixels corresponding to the object recognition window ORW′ emit light. Figure 4 Although not shown in the figure, the display driver 120 may include a data driver, a scan driver, a timing controller, a gamma circuit, etc.

[0061] The coordinates on the touch sensor panel 115 may be matched with the corresponding coordinates on the display panel 111. For example, the display panel 111 may display interface information on a specific area P. A user may contact or approach a specific area Q on the touch sensor panel 115 to input a command through the displayed interface information. Here, the coordinates of the specific area Q may be matched with the coordinates of the specific area P. Therefore, contact or approach to the specific area Q may be processed in association with the interface information displayed on the specific area P.

[0062] In some example embodiments, the touch sensor panel 115 may be implemented separately from the display panel 111. For example, Figure 4 As shown, the touch sensor panel 115 may be placed on or above the display panel 111. However, example embodiments are not limited thereto. Figure 4 Unlike shown in FIG. 1 , the display panel 111 may be placed on or over the touch sensor panel 115. Alternatively, the touch sensor panel 115 and the display panel 111 may be implemented in one single panel.

[0063] The object recognition sensor 130 may be used to detect an object. The object recognition sensor 130 may generate / output an image signal associated with an object on the object recognition window ORW. For example, the object recognition sensor 130 may be operated to obtain an image signal associated with a finger (or fingerprint) contacting or approaching the object recognition window ORW. Figure 3A and Figure 3B As described above, the object recognition sensor 130 may include a lens 132 and an image sensor 134. In some example embodiments, the lens 132 may be omitted.

[0064] The object recognition sensor 130 may provide a function of optical object recognition or optical-based object detection. For example, the image sensor 134 included in the object recognition sensor 130 may include a photodiode capable of generating a current in response to light.

[0065] As reference Figure 2 As described above, an object recognition window ORW may be provided on the panel 110, for example, on the touch sensor panel 115. In addition, an object recognition window ORW' may be provided on the display panel 111 to correspond to the object recognition window ORW. The object recognition sensor 130 may be disposed below the display panel 111 to spatially correspond to the position of the object recognition window ORW and the position of the object recognition window ORW'.

[0066] In some example embodiments, the position of the object recognition window ORW may be associated with coordinates on the touch sensor panel 115, and the position of the object recognition window ORW′ may be associated with coordinates on the display panel 111. In addition, the position and size of each of the object recognition windows ORW and ORW′ may be modified or changed depending on the arrangement of the object recognition sensor 130.

[0067] The processor 140 may control the overall operation of the electronic device 100. The processor 140 may process / perform various arithmetic / logical operations to provide the functions of the electronic device 100.

[0068] The processor 140 may communicate with the display driver 120, the touch controller 125, the object recognition sensor 130, and the memory 150. The processor 140 may control the operations of the display driver 120, the touch controller 125, the object recognition sensor 130, and the memory 150. The processor 140 may process commands, requests, responses, etc. associated with the operations of the display driver 120, the touch controller 125, the object recognition sensor 130, and the memory 150.

[0069] For example, the processor 140 may process a command received from the touch controller 125 according to a user command input through the touch sensor panel 115. For example, the processor 140 may provide various information to the display driver 120 to display a desired or wanted image on the display panel 111. For example, the processor 140 may control the operation timing and / or sequence of the display panel 111 and the object recognition sensor 130 so that the object recognition sensor 130 generates a signal associated with an object image. For example, the processor 140 may generate and analyze information associated with an object image based on a signal output from the object recognition sensor 130. For example, the processor 140 may store the associated data in the memory 150 or may load the associated data from the memory 150.

[0070] In some example embodiments, the processor 140 may include one or more dedicated circuits (e.g., field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.) to perform various operations. For example, the processor 140 may include one or more processor cores capable of performing various operations. For example, the processor 140 may be implemented with a general-purpose processor, a dedicated processor, or an application processor.

[0071] The memory 150 may store data associated or related to the operation of the electronic device 100. For example, the memory 150 may store various data for performing a method of performing a half shutter function according to an example embodiment.

[0072] In some example embodiments, the memory 150 may include at least one of various volatile memories and / or at least one of various nonvolatile memories, the various volatile memories such as dynamic random access memory (DRAM), static random access memory (SRAM), etc., and the various nonvolatile memories such as flash memory, phase change random access memory (PRAM), resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), nano floating gate memory (NFGM), polymer random access memory (PoRAM), etc.

[0073] The camera 170 may correspond to Figure 2 The image capture module 102 in the image capture module 102 may perform an image capture function. For example, the camera 170 may include an image sensor, a lens, etc. The camera processor 160 may control the operation of the camera 170. For example, the camera processor 160 may perform a half shutter process when a half shutter input is received.

[0074] In some example embodiments, the display driver 120, the touch controller 125, the object recognition sensor 130, the processor 140, the memory 150, the camera processor 160, and the camera 170 may be implemented with separate circuits / modules / chips, respectively. In other example embodiments, based on the functions, some of the display driver 120, the touch controller 125, the object recognition sensor 130, the processor 140, the memory 150, the camera processor 160, and the camera 170 may be combined into one circuit / module / chip, or may be further divided into a plurality of circuits / modules / chips.

[0075] The electronic device 100 may execute the Figure 1 The method of performing the half shutter function of the exemplary embodiment described above. For example, the display panel 111 and the display driver 120 may perform Figure 1 In operation S100, the object recognition sensor 130 may perform Figure 1 In operation S200, the processor 140 may execute Figure 1 and the camera processor 160 may perform operations S300 and S400 in FIG. Figure 1 In addition, the electronic device 100 may perform the operation S500 of the reference Fig.18 and Fig.19 The method of performing the half shutter function described herein and / or referring to Fig. 20 Describes the method of capturing images.

[0076] refer to Figure 5 , except that the electronic device 100a includes a processor 140a, Figure 5 The electronic device 100a can be used with Figure 4 The electronic device 100 is substantially the same as Figure 4 The processor 140 and the camera processor 160 are integrated or merged in the processor 140a.

[0077] Figure 6 It is shown Figure 1 Flowchart of an example of operations S300 and S400 in FIG.

[0078] refer to Figure 1 and Figure 6 , when determining whether the first image includes the first object (operation S300), the first value may be obtained by performing spatial signal processing (or object detection processing in the spatial domain) on the first image (operation S310). For example, spatial signal processing may refer to a scheme or method of directly using pixel values ​​or grayscale values. For example, each of the plurality of first values ​​may be a statistical value (e.g., variance) based on illumination. However, example embodiments are not limited thereto, and at least one of various techniques may be used.

[0079] Whether the first image includes the first object may be determined based on the first value and a predetermined first threshold value. For example, when the first value is less than or equal to the first threshold value (operation S320: No), it may be determined that the first image includes the first object (operation S330). When the first value is greater than the first threshold value (operation S320: Yes), it may be determined that the first image does not include the first object (operation S340).

[0080] When it is determined that the first image includes the first object (operation S330), it may be determined whether the first object in the first image has a half-shutter state (operation S400).

[0081] When determining whether the first object in the first image has a half shutter state (operation S400), the second value may be obtained by performing frequency signal processing (or object detection processing in the frequency domain) on the first image (operation S410). For example, frequency signal processing may refer to a scheme or method of transforming a pixel value or a grayscale value into a frequency value (or a frequency band) using a wavelet transform or a Fourier transform, and using the transformed frequency value. For example, each of the plurality of second values ​​may be a result of a Fourier transform. However, example embodiments are not limited thereto, and at least one of various techniques may be used.

[0082] It may be determined whether the first object in the first image has a half-shutter state based on the second value and a predetermined second threshold value. For example, when the second value is greater than the second threshold value (operation S420: yes), it may be determined that the first object in the first image has a half-shutter state (operation S430). When the second value is less than or equal to the second threshold value (operation S420: no), it may be determined that the first object in the first image does not have a half-shutter state (operation S440). For example, the second threshold value in operation S420 may be different from the first threshold value in operation S320.

[0083] When it is determined that the first object in the first image has a half-shutter state (operation S430), the Figure 1 The half shutter process is performed as shown in operation S500 of the first image. When it is determined that the first object in the first image does not have the half shutter state (operation S440), the method according to the example embodiment may be terminated without performing the half shutter process. For example, when the first object in the first image does not have the half shutter state, the first object may have a full shutter state for providing a full shutter input, or may have another state.

[0084] When it is determined that the first image does not include the first object (operation S340), the method according to the example embodiment may be terminated without determining whether the first object in the first image has a half-shutter state. For example, when the first image does not include the first object, the first image may be a contaminated (or stained) image. For example, the contaminated image may be an image including information of an external light source other than the light source of the display device, or may be an image including an object other than the first object as a target object for optical object recognition (e.g., a unique pattern of the bag when the electronic device is in a bag). For example, the image including information of the external light source may be removed by analyzing a histogram. Generally, an image contaminated by an external light source may have a relatively high pixel value compared to a pixel value of a light source of the display device reflected by an object. Therefore, in the histogram of the obtained image, there may be a relatively large number of pixels having a pixel value greater than or equal to a predetermined threshold, and when a plurality of pixels having a pixel value greater than or equal to the threshold are detected, it may be determined that the obtained image is contaminated by an external light source.

[0085] Fig. 7A , Figure 7B , Figure 7C , Figure 8 and Fig. 9 Is used to describe Figure 6 Graph of the operation.

[0086] refer to Fig. 7A , the external light source is clearly photographed or filmed, and Fig. 7A An image may represent an image contaminated by external light sources.

[0087] refer to Figure 7B and Figure 7C , showing a case where the first object is a user's finger including a fingerprint. Figure 7B As shown, the fingerprint is photographed relatively blurry, and objects other than the fingerprint (e.g., external light sources) are partially photographed at the edge of the image, so Figure 7B The image may represent a case where the user's finger is spaced a certain distance from the object recognition window ORW (for example, when a half shutter input is provided). Figure 7C As shown, the fingerprint is photographed relatively clearly, and no other objects are photographed, so Figure 7C The image may represent a case where the user's finger is in direct contact with the object recognition window ORW (eg, when a full shutter input is provided).

[0088] In some example embodiments, the image may be obtained by subtracting an image without an object or an image without an object (eg, a reference image) from an image including an object. Figure 7B and Figure 7CThe image shown in the figure is obtained, thereby obtaining only pure information associated with the object. Generally, the panel including the display panel and the touch sensor panel has a complex internal structure, which includes a pattern of multilayer wiring, electrodes, etc. When any object (or arbitrary object) is placed on the object recognition window while illuminating the light source corresponding to the object recognition window, the reflected light received by the object recognition window includes information of the object and information of the internal structure of the panel (e.g., bottom view). Therefore, in order to obtain only the information of the object, only the information about the internal structure of the panel as an interference component can be obtained first, and then compensation for removing the interference component from the image signal obtained by the object can be performed. The reference image can represent the interference component and can be referred to as a calibration image or calibration data.

[0089] refer to Figure 8 , showing the Fig. 7A , Figure 7B and Figure 7C Image execution Figure 6 Example results of operation S310 in FIG. Figure 8 In the graph, the horizontal axis FRAME (frame) represents the frame number of the image, and the vertical axis SV represents the frame number of the image. Figure 6 The first frame image F1, the second frame image F2 and the third frame image F3 respectively represent Fig. 7A images, Figure 7B Images and Figure 7C of the image.

[0090] like Figure 8 As shown, the value SV1 obtained by performing spatial signal processing on the first frame image F1 may be greater than the first threshold TH11, so Fig. 7A The image may be determined as an image that does not include the user's finger (eg, a contaminated image). Each of the values ​​SV2 and SV3 obtained by performing spatial signal processing on the second frame image F2 and the third frame image F3 may be less than the first threshold value TH11, so Figure 7B and Figure 7C Each of the images may be determined to be an image including a finger of the user.

[0091] refer to Fig. 9 , showing the Figure 7B and Figure 7C Image execution Figure 6 Example result of operation S410 in FIG. Fig. 7A The image does not include the user's finger, so it can be wrong Fig. 7A The image is used to perform operation S410. Fig. 9 In the graph, the horizontal axis FRAME (frame) represents the frame number of the image, and the vertical axis FTV represents the frame number of the image. Figure 6 A second value obtained by frequency signal processing in (eg, a result of Fourier transform).

[0092] like Fig. 9 As shown, the value FTV12 obtained by performing frequency signal processing on the second frame image F2 may be greater than the second threshold TH12, and thus it may be determined that Figure 7B The user's finger in the image has or is at a half shutter level. The value FTV13 obtained by performing frequency signal processing on the third frame image F3 may be less than the second threshold TH12, and thus it may be determined that Figure 7C The user's finger in the image does not have or is not in a half-shutter state (e.g., has or is in a full-shutter state).

[0093] Fig.10 and Fig.11 It is shown Figure 1 The flowchart of the example of operation S300 and S400 in FIG. Figure 6 Duplicate description.

[0094] refer to Fig.10 , except that operation S305 is added to Fig.10 Examples and Figure 6 Operations S310 and S410 in are respectively changed to Fig.10 In addition to operations S310a and S410a, Fig.10 The example can be compared with Figure 6 The examples are basically the same.

[0095] A first key area (e.g., a main area or an important area) may be detected in the first image (operation S305). Typically, an object recognition sensor may be equipped for the purpose of touch sensing, and since the photographing range and purpose of the object recognition sensor do not match the photographing range and purpose for detecting the half shutter input, an image other than the target for detecting the half shutter input may be obtained. According to an example embodiment, the half shutter input detection efficiency may be improved by determining a portion corresponding to the target and performing a detection process only on the determined portion (e.g., a key area). Key area detection may be a concept including spatial domain detection and frequency domain detection, and a wider range of candidates may be detected. The key area may be referred to as a region of interest (ROI).

[0096] Thereafter, the first key area in the first image may be subjected to Fig.10 The spatial signal processing in operation S310a and Fig.10 The frequency signal is processed in operation S410a.

[0097] refer to Figure 1 and Fig.11, when determining whether the first image includes the first object (operation S300), a first value may be obtained by performing first frequency signal processing on the first image (operation S312). Fig.11 The first frequency signal processing in operation S312 may be performed with Figure 6 The frequency signal processing in operation S410 is substantially the same or different.

[0098] Whether the first image includes the first object may be determined based on the first value and a predetermined first threshold. For example, when the first value is greater than the first threshold (operation S322: yes), it may be determined that the first image includes the first object (operation S330). When the first value is less than or equal to the first threshold (operation S322: no), it may be determined that the first image does not include the first object (operation S340). For example, Fig.11 The first threshold in operation S322 may be different from Figure 6 The first threshold in operation S320.

[0099] When determining whether the first object in the first image has a half shutter state (operation S400), a second value may be obtained by performing a second frequency signal processing on the first image (operation S412). The second frequency signal processing in operation S412 may be at least partially different from the first frequency signal processing in operation S312. For example, algorithms, implementation schemes, etc. may be different from each other, or the accuracy, resolution, etc. of the processing results may be different from each other. For example, Fig.11 The second frequency signal processing in operation S412 may be performed with Figure 6 The frequency signal processing in operation S410 is substantially the same or different.

[0100] Whether the first object in the first image has a half-shutter state can be determined based on the second value and a predetermined second threshold. For example, when the second value is greater than the second threshold (operation S422: yes), it can be determined that the first object in the first image has a half-shutter state (operation S430). When the second value is less than or equal to the second threshold (operation S422: no), it can be determined that the first object in the first image does not have a half-shutter state (operation S440). For example, the second threshold in operation S422 can be different from the first threshold in operation S322. For example, Fig.11 The second threshold value in operation S422 may be Figure 6 The second threshold in operation S420 is substantially the same as or different from that in operation S420.

[0101] and Figure 6 Unlike an example in which each of the spatial signal processing and the frequency signal processing is performed once, in Fig.11 In the example shown in FIG. 1 , only the frequency signal processing can be performed twice.

[0102] Fig.12 and Fig.13 Is used to describe Fig.11 The operation diagram of will be omitted. Figure 8 and Fig. 9 Duplicate description.

[0103] refer to Fig.12 , showing the Fig. 7A , Figure 7B and Figure 7C Image execution Fig.11 Example result of operation S312 in FIG. Fig.12 In the graph, the horizontal axis FRAME (frame) represents the frame number of the image, and the vertical axis FTVA represents the frame number of the image. Fig.11 The first value is obtained by processing the first frequency signal in.

[0104] like Fig.12 As shown, the value FTV21a obtained by performing the first frequency signal processing on the first frame image F1 may be smaller than the first threshold TH21, so Fig. 7A The image may be determined as an image that does not include the user's finger (eg, a contaminated image). Each of the values ​​FTV22a and FTV23a obtained by performing the first frequency signal processing on the second frame image F2 and the third frame image F3 may be greater than the first threshold value TH21, so Figure 7B and Figure 7C Each of the images may be determined as an image including a finger of the user.

[0105] refer to Fig.13 , showing the Figure 7B and Figure 7C Image execution Fig.11 Example result of operation S412 in FIG. Fig. 7A The image does not include the user's finger, so it can be wrong Fig. 7A The image is processed by performing operation S412. Fig.13 In the graph, the horizontal axis FRAME (frame) represents the frame number of the image, and the vertical axis FTVB represents the frame number of the image. Fig.11 The second value is obtained by processing the second frequency signal in.

[0106] like Fig.13 As shown, the value FTV22b obtained by performing the second frequency signal processing on the second frame image F2 may be greater than the second threshold TH22, and thus it may be determined that Figure 7B The user's finger in the image has a half shutter state. The value FTV23b obtained by performing the second frequency signal processing on the third frame image F3 may be less than the second threshold TH22, and thus it may be determined that Figure 7CThe user's finger in the image does not have a half shutter state (eg, has a full shutter state).

[0107] Fig.14 and Fig.15 It is shown Figure 1 The flowchart of the example of operation S300 and S400 in FIG. Figure 6 , Fig.10 and Fig.11 Duplicate description.

[0108] refer to Fig.14 , except that operation S305 is added to Fig.14 Examples and Fig.11 Operations S312 and S412 in are respectively changed to Fig.14 In addition to operations S312a and S412a, Fig.14 The example can be compared with Fig.11 The examples are basically the same. Fig.14 Operation S305 in the above example can be combined with Fig.10 The operation S305 in FIG. 1 is substantially the same as that in FIG. 1 . The operation S305 in FIG. 1 may be performed only on the first key area in the first image. Fig.14 The first frequency signal processing in operation S312a and Fig.14 The second frequency signal is processed in operation S412a.

[0109] refer to Figure 1 and Fig.15 , when determining whether the first image includes the first object (operation S300), a first value may be obtained by performing frequency signal processing on the first image (operation S314). Fig.15 The frequency signal processing in operation S314 may be performed with Figure 6 The frequency signal processing in operation S410, Fig.11 The first frequency signal processing in operation S312 and / or Fig.11 The second frequency signal processing in operation S412 is substantially the same as or different from that in operation S414.

[0110] It may be determined whether the first image includes the first object based on the first value and a predetermined first threshold value (operation S324). For example, when the first value is greater than the first threshold value (operation S324: yes), it may be determined that the first image includes the first object (operation S330). When the first value is less than or equal to the first threshold value (operation S324: no), it may be determined that the first image does not include the first object (operation S340). For example, Fig.15 The first threshold in operation S324 may be Fig.11 The first threshold in operation S322 is substantially the same or different.

[0111] When determining whether the first object in the first image has a half-shutter state (operation S400), it can be determined whether the first object in the first image has a half-shutter state based on the first value and a predetermined second threshold. For example, when the first value is greater than the second threshold (operation S424: yes), it can be determined that the first object in the first image has a half-shutter state (operation S430). When the first value is less than or equal to the second threshold (operation S424: no), it can be determined that the first object in the first image does not have a half-shutter state (operation S440). For example, the second threshold in operation S424 can be different from the first threshold in operation S324. For example, Fig.15 The second threshold value of operation S424 may be Fig.11 The second threshold in operation S422 is substantially the same as or different from that in operation S422.

[0112] and Figure 6 An example in which each of spatial signal processing and frequency signal processing is performed once and Fig.11 The example in which the frequency signal processing is performed twice is different in Fig.15 In the example, frequency signal processing may be performed only once.

[0113] Fig.16 Is used to describe Fig.15 The operation diagram of will be omitted. Figure 8 , Fig. 9 , Fig.12 and Fig.13 Duplicate description.

[0114] refer to Fig.16 , showing the Fig. 7A , Figure 7B and Figure 7C Image execution Fig.15 Example result of operation S314 in FIG. Fig.16 In the graph, the horizontal axis FRAME (frame) represents the frame number of the image, and the vertical axis FTVC represents the frame number of the image. Fig.15 The first value obtained by processing the frequency signal in.

[0115] like Fig.16 As shown, the value FTV31 obtained by performing frequency signal processing on the first frame image F1 may be smaller than the first threshold TH31, so Fig. 7A The image may be determined as an image that does not include the user's finger (eg, a contaminated image). The value FTV32 obtained by performing frequency signal processing on the second frame image F2 may be greater than the first threshold TH31 and greater than the second threshold TH32, so Figure 7BThe image may be determined as an image including a user's finger in a half shutter state. The value FTV33 obtained by performing frequency signal processing on the third frame image F3 may be greater than the first threshold TH31 and less than the second threshold TH32, so Figure 7C The image may be determined as an image including the user's finger that does not have a half shutter state (eg, has a full shutter state).

[0116] Fig.17 It is shown Figure 1 The flowchart of the example of operation S300 and S400 in FIG. Figure 6 , Fig.10 , Fig.11 and Fig.15 Duplicate description.

[0117] refer to Fig.17 , except that operation S305 is added to Fig.17 Examples and Fig.15 Operation S314 in is changed to Fig.17 In addition to operation S314a, Fig.17 The example can be compared with Fig.15 The examples are basically the same. Fig.17 Operation S305 in the above example can be combined with Fig.10 The operation S305 in FIG. 1 is substantially the same as that in FIG. 1 . The operation S305 in FIG. 1 may be performed only on the first key area in the first image. Fig.17 The frequency signal is processed in operation S314a.

[0118] Fig.18 and Fig.19 is a flowchart illustrating a method of performing a half shutter function using optical object recognition according to an example embodiment. Figure 1 Duplicate description.

[0119] refer to Fig.18 , Fig.18 The operations S100, S200, S300, S400 and S500 in the embodiment can be respectively Figure 1 The operations S100, S200, S300, S400 and S500 in FIG. 1 are substantially the same.

[0120] It may be determined whether a full shutter input is received (operation S600). Only when before receiving the full shutter input, for example, when the full shutter input has not been received (operation S600: No), operations S200, S300, S400, and S500 may be performed to obtain a first image, determine whether the first image includes a first object, determine whether the first object in the first image has a half shutter state, and drive a half shutter process. When after receiving the full shutter input, for example, when the full shutter input has been received (operation S600: Yes), the next process may be waited for while the shutter button is displayed.

[0121] According to the present disclosure, a specific state (or situation) before a full shutter input is detected as a half shutter state and is used to perform a half shutter function, and the half shutter function may not be performed after receiving a full shutter input. However, the example embodiments are not limited thereto. For example, in addition to the preparation operation, the example embodiments may also be applied to post-processing of an image after receiving a full shutter input (e.g., a touch input). For example, when finger information exists together with the shutter input, the finger information may be used as a half shutter or post-processing command by reflecting the touch sensor input information.

[0122] In some example embodiments, Fig.18 The full shutter input in operation S600 may be input by a user via a touch sensor panel (eg, Figure 4 The touch sensor panel 115 is provided to provide touch input.

[0123] refer to Fig.19 , Fig.19 The operations S100, S200 and S500 in the embodiment can be respectively Figure 1 The operations S100, S200 and S500 in FIG. 1 are substantially the same.

[0124] A first comparative image may be obtained based on the first image and at least one second image that is pre-stored or stored in advance (operation S700). Figure 1 Unlike the example of , the operation of determining whether the first object is included (operation S300a) and the operation of determining whether the first object in the first image has a half-shutter state (operation S400a) may be performed based on the first comparison image instead of the first image.

[0125] In some example embodiments, as will be referred to Fig. 20 As described, a plurality of images including a first image may be continuously obtained based on reflected light received by the object recognition sensor through the object recognition window, and at least one second image may be an image obtained before the first image is obtained among the plurality of images obtained continuously. By continuously capturing or photographing, the previously captured images may be stored in a storage device (e.g., Figure 4The image captured by the current shutter input may be compared with the image currently captured, and thus the accuracy and efficiency of half shutter input detection may be improved or enhanced by using the changes in the currently captured image that were not present in the previously captured image. Here, continuous capturing or shooting may be one of the examples of using the similarity of temporal and spatial information among various information.

[0126] In some example embodiments, the first comparison image may be obtained or updated by performing signal processing in the form of a finite impulse response (FIR) or an infinite impulse response (IIR) instead of using only one image. In FIR, one object-free image with high quality (or performance) may be obtained by storing and using a specific number of object-free images. In IIR, one object-free image with high quality may be updated and used whenever a new object-free image is input. In the case of FIR, object-free images in a specific environment may be collected to obtain an image optimized for the specific environment. In the case of IIR, updates may be performed slowly, but may be widely applied using accumulated information.

[0127] As described above, the half-shutter input detection algorithm according to the example embodiments may have several advantages. In terms of space, spatial domain detection and frequency domain detection may be used to detect half-shutter input. In terms of time, a comparison image may be used or detected not only based on the current image but also based on a previous image stored in advance to improve or enhance the detection accuracy and efficiency of the half-shutter input. In addition, with respect to the object used in the half-shutter input detection, the half-shutter state may be identified by detecting various objects and situations as well as fingers to identify the situation before the shutter button is pressed as a half-shutter input. In some example embodiments, the previous image may not be limited to being used to detect differences from the current image, and may be used to detect various types of shutter inputs and / or ready states (e.g., image capture preparation states) by using image information obtained when the shutter button is input.

[0128] Fig. 20 is a flowchart illustrating a method of capturing an image according to an example embodiment. Figure 1 Duplicate description.

[0129] refer to Fig. 20 In the method of capturing an image according to an exemplary embodiment, an image capturing application providing an image capturing function is displayed on a display panel including a plurality of light sources by running the image capturing application (operation S1100). A shutter button is displayed by driving a portion of a plurality of light sources among the plurality of light sources included in the display panel (operation S1200). Fig. 20 Operation S1200 in Figure 1 The operation S100 in is basically the same.

[0130] While driving parts of the plurality of light sources, a plurality of images are continuously obtained or acquired based on reflected light received by the object recognition sensor through the object recognition window (operation S1300 ). Fig. 20 Operation S1300 in may be similar to Figure 1 Operation S200 in. For example, the object recognition sensor may continuously or sequentially obtain a number of images corresponding to the object recognition window and / or the object based on the reflected light. Here, continuous capture or shooting may be one of the examples of using the similarity of time and space information among various information.

[0131] When a first image among the plurality of images provides a half-shutter input, a half-shutter function is performed (operation S1400). Fig. 20 Operation S1400 may include Figure 1 Operations S300, S400 and S500 in FIG.

[0132] When a full shutter input is received after the half shutter function is executed, an image capturing function is performed (operation S1500).

[0133] In some example embodiments, by touching a sensor panel (e.g., Figure 4 The touch input received by the user of the touch sensor panel 115 in the display may provide a full shutter input. For example, when the user's finger directly contacts the object recognition window, the touch sensor panel may sense the full shutter input.

[0134] In other example embodiments, a second image obtained after a first image in the plurality of images may provide a full shutter input. For example, when a first object in the second image has a full shutter state for providing a full shutter input, an image capture function may be performed based on the full shutter state. In this example, even if the touch sensor panel does not function properly (or if the touch sensor panel does not detect a touch input), the object recognition sensor may be used to recognize a full shutter input and an image capture function may be performed.

[0135] Fig.21A , Fig. 21B , Fig. 21C and Fig. 22 Is used to describe Fig. 20 FIG. 1 is a diagram of a method of capturing an image.

[0136] refer to Fig.21A , Fig. 21B , Fig. 21C , images captured continuously before and after the image including the object can be obtained. Fig.21A and Fig. 21C shows an image captured indoors, and Fig. 21B An image captured outdoors is shown. Fig.21A and Fig. 21B shows the case where the user's finger is used directly, and Fig. 21C The case where a gloved finger is used is shown.

[0137] For example, according to the image sequence (or image order) when continuously capturing images with half shutter and full shutter inputs, an image contaminated by an external light source may be captured first. Next, when a finger approaches the panel, the external light source may be blocked or masked by the finger, and an image in which the finger is separated from the panel by a certain distance may be captured. Subsequently, an image including a finger (e.g., a fingerprint) may be captured. Afterwards, when the finger is moved away, an image in which the finger is separated from the panel by a certain distance may be captured again. Finally, an image contaminated by an external light source may be captured again. In an environment without an external light source, there may be no image contaminated by an external light source.

[0138] exist Fig.21A In the example, there may be 40 images from No. 1 to No. 40. The image may be captured by changing the position of the user's finger and the device in an indoor environment where there are multiple external light sources. Fig.21A Image. Fig.21A , images No. 1 to No. 10 are arranged in the first row in the left-to-right direction (i.e., image No. 1 is the leftmost position and image No. 10 is the rightmost position in the first row), images No. 11 to No. 20 are arranged in the second row (image No. 11 is the leftmost position and image No. 20 is the rightmost position in the second row), images No. 21 to No. 30 are arranged in the third row (image No. 21 is the leftmost position and image No. 30 is the rightmost position in the third row), and images No. 31 to No. 40 are arranged in the fourth row (image No. 31 is the leftmost position and image No. 40 is the rightmost position in the fourth row).

[0139] Images No. 1 to No. 8 may represent images contaminated by an external light source, images No. 9 to No. 14 may represent images in which a finger approaches a shutter button in preparation for image capture, images No. 15 to No. 18 may represent images in which the shutter button is touched, images No. 19 to No. 26 may represent images in which the finger is removed after being touched (e.g., after image capture), and images No. 27 to No. 40 may represent images contaminated by an external light source. A portion including images No. 12 to No. 18 may represent a half shutter state or a full shutter state recognized according to an example embodiment.

[0140] exist Fig. 21B In the example, there may be 60 images from No. 1 to No. 60, 10 images are arranged in each row from the first row to the sixth row, and the image numbers increase in the left-to-right direction, similar to Fig.21A. Images No. 1 to No. 7 may represent images in which a finger is not close to the panel, images No. 8 to No. 13 may represent images in which a finger is close to the panel, images No. 14 to No. 16 may represent images in which a shutter button is touched, images No. 17 to No. 36 may represent images in which a finger is moved away after being touched, images No. 37 to No. 44 may represent images in which a finger is not close to the panel, images No. 45 to No. 58 may represent images in which a finger is close to the panel again, and images No. 59 to No. 60 may represent images in which a finger is not close to the panel. A portion including images No. 8 to No. 13, No. 17 to No. 36, and No. 45 to No. 58 may represent a half-shutter state recognized according to an example embodiment.

[0141] exist Fig. 21C In the example, there may be 50 images from No. 1 to No. 50, 10 images are arranged in each row from the first row to the fifth row, and the image numbers increase in the left-to-right direction, similar to Fig.21A . Images No. 1 to No. 3, No. 23 to No. 29, and No. 41 to No. 47 may represent contaminated images, images No. 4 to No. 14, No. 21 to No. 22, No. 30 to No. 34, No. 40, and No. 48 to No. 50 may represent images immediately before or after a touch input, and images No. 15 to No. 20 and No. 35 to No. 39 may represent images with a touch input. A portion including images No. 4 to No. 12, No. 21 to No. 22, No. 30, No. 40, and No. 48 to No. 50 may represent a half-shutter state identified according to an example embodiment. During a touch input, a fabric pattern of a glove may be captured instead of a fingerprint.

[0142] refer to Fig. 22 , shows the result of recognizing the half shutter state based on the method of executing the half shutter function according to an example embodiment. Fig. 22 In the figure, the horizontal axis FRAME (frame) represents the frame number of the image, and the vertical axis DV represents the frame number of the image. Fig. 21B The half shutter state as a result of the determination may be represented as a value of 100, and images No. 8 to No. 13, No. 17 to No. 36, and No. 45 to No. 58 having a value of 100 may be recognized as images having the half shutter state.

[0143] As will be appreciated by those skilled in the art, the present invention may be embodied as a system, method, computer program product, and / or a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon. The computer-readable program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. For example, a computer-readable medium may be a non-transitory computer-readable medium.

[0144] Fig.23 is a block diagram illustrating an electronic device according to an example embodiment.

[0145] refer to Fig.23 , the electronic device 1000 may include a processor 1010, a storage device 1020, an object recognition sensor 1030, an input / output (I / O) device 1040, a power supply 1050, and a display device 1060. The electronic device 1000 may also include a plurality of ports for communicating with a video card, a sound card, a memory card, a universal serial bus (USB) device, other electronic devices, and the like.

[0146] The processor 1010 controls the operation of the electronic device 1000. The processor 1010 may run an operating system and at least one application to provide an Internet browser, a game, a video, etc. The memory device 1020 may store data for the operation of the electronic device 1000. The I / O device 1040 may include an input device and an output device, the input device being a camera, a keyboard, a keypad, a mouse, a touch pad, a touch screen, a remote controller, etc., and an output device being a printer, a speaker, a display, etc. The power supply 1050 may provide power for the operation of the electronic device 1000.

[0147] The display device 1060 includes a display panel. Fig.23 The display device 1060, the object recognition sensor 1030, the processor 1010 and the storage device 1020 in the embodiment may correspond to Figure 4 The display panel 111, the object recognition sensor 130, the processor 140, and the memory 150 in FIG. 1 may perform a method of performing a half shutter function according to an example embodiment and a method of capturing an image according to an example embodiment.

[0148] In a method for performing a half-shutter function using optical object recognition and a method for capturing an image according to an example embodiment, a half-shutter input for performing a half-shutter process can be provided or recognized by detecting a situation before a shutter button is pressed based on an object recognition sensor and optical object recognition and by recognizing the situation as a half-shutter state. Therefore, a half-shutter input can be effectively recognized without a physical shutter button, and zero shutter lag can be effectively achieved with low power consumption by performing various functions before capturing an image without reducing the size of a display in an electronic device and increasing the thickness of the electronic device.

[0149] The present invention can be applied to various electronic devices and systems including a display panel and an object recognition sensor and performing a half shutter function when capturing an image. For example, the present invention can be applied to systems such as mobile phones, smart phones, tablet computers, laptop computers, personal digital assistants (PDAs), portable multimedia players (PMPs), digital cameras, portable game consoles, music players, portable camcorders, video players, navigation devices, wearable devices, Internet of Things (IoT) devices, Internet of Everything (IoE) devices, e-book readers, virtual reality (VR) devices, augmented reality (AR) devices, robotic devices, drones, etc.

[0150] According to example embodiments, at least one of the components, elements, modules or units described herein may be embodied as various numbers of hardware, software and / or firmware structures for running the above-mentioned corresponding functions. For example, at least one of these components, elements or units may use a direct circuit structure, such as a memory, a processor, a logic circuit, a lookup table, etc., which may be controlled by one or more microprocessors or other control devices to run the corresponding functions. Moreover, at least one of these components, elements or units may be specifically embodied by a part of a module, a program or a code containing one or more executable instructions for performing a specified logical function, and may be run by one or more microprocessors or other control devices. Moreover, at least one of these components, elements or units may further include a processor or be implemented by a processor, such as a central processing unit (CPU), a microprocessor, etc., that performs the corresponding functions. Two or more of these components, elements or units may be combined into a single component, element or unit that performs all operations or functions of the two or more components, elements or units combined. Moreover, at least a portion of the functions of at least one of these components, elements or units may be performed by another of these components, elements or units. In addition, although the bus is not shown in the block diagram, the communication between components, elements or units may be performed by the bus. The functional aspects of the above exemplary embodiments can be implemented in algorithms running on one or more processors. In addition, the components, elements or units represented by the blocks or processing operations can adopt any number of related technologies for electronic configuration, signal processing and / or control, data processing, etc.

[0151] The foregoing is illustrative of example embodiments and should not be construed as limiting thereof. Although some example embodiments have been described, those skilled in the art will readily appreciate that many modifications may be made in the example embodiments without materially departing from the novel teachings and advantages of the example embodiments. Therefore, all such modifications are intended to be included within the scope of the example embodiments as defined in the claims. Therefore, it is to be understood that the foregoing is illustrative of various example embodiments and should not be construed as being limited to the specific example embodiments disclosed, and modifications to the disclosed example embodiments as well as other example embodiments are intended to be included within the scope of the appended claims.

Claims

1. A method for performing a half shutter function using optical object recognition, the method include: displaying a shutter button on a display panel by driving a portion of a plurality of light sources included in the display panel, the portion of the plurality of light sources being disposed corresponding to an object recognition window of the display panel; while driving the portion of the plurality of light sources, obtaining a first image based on reflected light received by an object recognition sensor through the object recognition window; determining whether the first image includes a first object based on blur in the first image, the first object having a half-shutter state, the first image indicating that the first object is a predetermined distance from the object recognition window; as well as Based on a determination that the first image includes the first object having the half-shutter state, a half-shutter process is performed.

2. The method according to claim 1, in, The half-shutter state indicates an input for performing a half-shutter process associated with image capturing before performing image capturing.

3. The method according to claim 2, in, The half shutter process includes at least one of focus adjustment, exposure adjustment, white balance adjustment, gain adjustment, low power consumption, and control for continuous image capture.

4. The method according to claim 1, in, Determining whether the first image includes the first object includes: obtaining a first value by performing spatial signal processing on the first image; and It is determined whether the first image includes the first object based on the first value and a predetermined first threshold.

5. The method according to claim 4, in, Determining whether the first image includes the first object further includes: obtaining a second value by performing frequency signal processing on the first image; and Whether the first object in the first image has the half-shutter state is determined based on the second value and a predetermined second threshold.

6. The method according to claim 5, in, Determining whether the first image includes the first object further includes: detecting a first key region in the first image, and The spatial signal processing and the frequency signal processing are performed only on the first key area.

7. The method according to claim 1, in, Determining whether the first image includes the first object includes: obtaining a first value by performing a first frequency signal processing on the first image; and It is determined whether the first image includes the first object based on the first value and a predetermined first threshold.

8. The method according to claim 7, in, Determining whether the first image includes the first object further includes: obtaining a second value by performing a second frequency signal processing on the first image; and Whether the first object in the first image has the half-shutter state is determined based on the second value and a predetermined second threshold.

9. The method according to claim 1, in, Determining whether the first image includes the first object includes: obtaining a first value by performing frequency signal processing on the first image; and It is determined whether the first image includes the first object based on the first value and a predetermined first threshold.

10. The method according to claim 9, in, Determining whether the first image includes the first object further includes: Whether the first object in the first image has the half-shutter state is determined based on the first value and a predetermined second threshold.

11. The method according to claim 1, further comprising: include: Determine if full shutter input is received, When it is determined that the full shutter input is not received, determining whether the first image includes the first object having the half shutter state is performed.

12. The method according to claim 11, in, The full shutter input is provided through a touch sensor panel.

13. The method according to claim 1, further comprising: include: obtaining a first comparison image based on the first image and at least one second image stored in advance, Wherein, determining whether the first image includes the first object having the half-shutter state is performed based on the first comparison image.

14. The method according to claim 13, in, continuously obtaining a plurality of images including a first image based on the reflected light received by the object recognition sensor through the object recognition window, and The at least one second image is an image among the plurality of images obtained before the first image.

15. The method according to claim 1, in, The first object includes a user's finger.

16. A method for capturing an image, the method include: displaying a shutter button on the display panel by driving a portion of a plurality of light sources included in the display panel, the portion of the plurality of light sources being disposed to correspond to an object recognition window of the display panel; while driving the portion of the plurality of light sources, continuously obtaining a plurality of images based on reflected light received by an object recognition sensor through the object recognition window; performing a half shutter function based on a half shutter input detected based on a first image of the plurality of images; as well as Perform image capture based on full shutter input, Wherein, executing the half shutter function includes: determining whether the first image includes a first object based on blur in the first image, the first object having a half-shutter state, the first image indicating that the first object is a predetermined distance from the object recognition window; and A half-shutter process is performed based on the half-shutter state of the first object included in the first image.

17. The method according to claim 16, in, The full shutter input is detected based on a second image of the plurality of images obtained after the first image.

18. The method according to claim 16, in, The full shutter input is detected based on a user's touch input received through the touch sensor panel.

19. The method according to claim 16, further comprising: include: An image capture application providing the image capture functionality is run.

20. A method for performing a half shutter function using optical object recognition, the method include: running an image capture application providing an image capture function, and displaying the image capture application on a display panel, the display panel comprising a plurality of light sources; displaying a shutter button on the display panel by driving a portion of the plurality of light sources, the portion of the plurality of light sources being disposed to correspond to an object recognition window of the display panel; before receiving a full shutter input, continuously obtaining a plurality of images based on reflected light received by an object recognition sensor through an object recognition window while driving the portion of the plurality of light sources, and determining that the first image includes a first object having a half shutter state based on blur in a first image among the plurality of images, the first image indicating that the first object is a predetermined distance from the object recognition window; as well as performing a half-shutter process based on the half-shutter state of the first object included in the first image, Wherein, based on a result of performing at least one of spatial signal processing and frequency signal processing on the first image, it is determined that the first image includes the first object having the half shutter state.

Images