Electronic devices and their operating methods

Abstract

This application discloses an electronic device and its operating method. An electronic device includes: an image sensor for acquiring an image including normal pixel values ​​sensed during a first exposure time and short-exposure pixel values ​​sensed during a second exposure time shorter than the first exposure time; and a processor configured to use an exposure ratio of the first exposure time to the second exposure time and using the short-exposure pixel values ​​and normal pixel values ​​included in a selected region among a plurality of regions of the image to acquire flag information indicating that the selected region is one of a motion region and a normal region, and configured to output a restored image based on the flag information, including restored pixel values ​​corrected from the short-exposure pixel values ​​included in the selected region.

Description

Technical Field

[0001] This disclosure generally relates to electronic devices and methods of operating the same, and more specifically, to an electronic device for performing image processing and a method of operating the electronic device. Background Technology

[0002] Recently, with the development of electronic technology, the demand for image sensors has been increasing in various electronic devices, including smartphones, digital cameras, game consoles, the Internet of Things, robots, surveillance cameras, medical cameras, and autonomous vehicles. Image sensors generate images by converting light into electrical signals.

[0003] Typically, image processing methods that compare multiple images acquired sequentially over time can be used to detect regions in an image where an object is moving. However, this method requires separate storage space for storing previously acquired images and is difficult to apply in environments requiring low-power technology because power consumption occurs each time an image is acquired due to image storage. Therefore, techniques to address these issues are needed. Summary of the Invention

[0004] According to one aspect of this disclosure, an electronic device is provided, comprising: an image sensor configured to acquire an image including normal pixel values ​​sensed during a first exposure time and short exposure pixel values ​​sensed during a second exposure time shorter than the first exposure time; and a processor configured to acquire flag information indicating that the selected region is one of a motion region and a normal region by using an exposure ratio of the first exposure time to the second exposure time and by using the short exposure pixel values ​​and normal pixel values ​​included in a selected region among a plurality of regions of the image, and configured to output a restored image including restored pixel values ​​corrected from the short exposure pixel values ​​included in the selected region based on the flag information.

[0005] According to another aspect of this disclosure, a method for operating an electronic device is provided, the method comprising the steps of: acquiring an image including normal pixel values ​​sensed during a first exposure time and short exposure pixel values ​​sensed during a second exposure time shorter than the first exposure time; acquiring flag information indicating that the selected region is one of a motion region and a normal region by using an exposure ratio of the first exposure time to the second exposure time and by using the short exposure pixel values ​​and normal pixel values ​​in a selected region included in a plurality of regions of the image; and outputting a restored image including restored pixel values ​​corrected from the short exposure pixel values ​​included in the selected region based on the flag information.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: a communication interface configured to receive an image from an external device, the image including normal pixel values ​​and short exposure pixel values ​​sensed during different exposure times; and a processor configured to determine whether a selected region is a motion region or a normal region by using an exposure ratio of the exposure time and by using short exposure pixel values ​​and normal pixel values ​​included in a plurality of regions of the image, and when the selected region is a motion region, to obtain a first recovered pixel value corrected from the short exposure pixel values ​​by using peripheral pixel values ​​of the short exposure pixel values, and when the selected region is a normal region, to obtain a second recovered pixel value corrected from the short exposure pixel values ​​using an exposure ratio. Attached Figure Description

[0007] Exemplary embodiments will now be described more fully below with reference to the accompanying drawings; however, they may be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the exemplary embodiments to those skilled in the art.

[0008] In the accompanying drawings, dimensions may be exaggerated for clarity. It will be understood that when an element is referred to as being "between" two elements, it can be the only element between those two elements, or there may be one or more intermediate elements. Similar designations always indicate similar elements.

[0009] Figure 1A This is a diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0010] Figure 1B This is a diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0011] Figure 2 This is a diagram illustrating an image sensor according to an embodiment of the present disclosure.

[0012] Figure 3 This is a diagram illustrating the image processing operation of a processor according to an embodiment of the present disclosure.

[0013] Figure 4 This is a diagram illustrating a detailed configuration of a processor according to an embodiment of the present disclosure.

[0014] Figure 5A This is a diagram illustrating an embodiment according to the present disclosure.

[0015] Figure 5B This is a diagram illustrating the exposure time according to an embodiment of the present disclosure.

[0016] Figure 5CThis is a diagram showing an image obtained according to an embodiment of the present disclosure based on exposure time.

[0017] Figure 5D This is a diagram illustrating an embodiment according to the present disclosure.

[0018] Figure 6 This is a diagram illustrating the operation of an exposure corrector according to an embodiment of the present disclosure.

[0019] Figure 7A This is a diagram illustrating the operation of the motion ratio calculator and motion area detector according to embodiments of the present disclosure.

[0020] Figure 7B This is a diagram illustrating the form of marking information according to embodiments of the present disclosure.

[0021] Figure 8A This is a diagram illustrating the operation of a motion region detector according to an embodiment of the present disclosure.

[0022] Figure 8B This is a diagram illustrating the operation of a motion region detector according to an embodiment of the present disclosure.

[0023] Figure 8C This is a diagram illustrating the operation of a motion region detector according to an embodiment of the present disclosure.

[0024] Figure 9A This is a diagram illustrating the operation of a noise filter according to an embodiment of the present disclosure.

[0025] Figure 9B This is a diagram illustrating the operation of a noise filter according to an embodiment of the present disclosure.

[0026] Figure 9C This is a diagram illustrating the operation of a noise filter according to an embodiment of the present disclosure.

[0027] Figure 10A This is a diagram illustrating the operation of a pixel corrector according to an embodiment of the present disclosure.

[0028] Figure 10B This is a diagram illustrating the operation of a pixel corrector according to an embodiment of the present disclosure.

[0029] Figure 10C This is a diagram illustrating the operation of a pixel corrector according to an embodiment of the present disclosure.

[0030] Figure 11 This is a diagram illustrating a method for compressing an image using marking information according to an embodiment of the present disclosure.

[0031] Figure 12This is a diagram illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0032] Figure 13 This is a diagram illustrating an implementation example of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0033] The specific structural or functional descriptions disclosed herein are merely illustrative in order to describe embodiments based on the concepts of this disclosure. Embodiments based on the concepts of this disclosure may be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0034] The embodiments provide an electronic device capable of detecting motion regions included in an image and a method for operating the electronic device.

[0035] Figure 1A This is a diagram illustrating an electronic device according to an embodiment of the present disclosure. Figure 1B This is a diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0036] Reference Figure 1A The electronic device may include an image sensor 110 and a processor 130.

[0037] Electronic device 10 can be an image acquisition device, digital camera, camcorder, closed-circuit television (CCTV), webcam, security camera, industrial vision camera, mobile device, smartphone, personal computer (PC), tablet PC, laptop computer, personal digital assistant (PDA), enterprise digital assistant (EDA), portable multimedia player (PMP), wearable device, black box, robot, autonomous vehicle, vehicle vision camera, set-top box, game console, electronic dictionary, e-book reader, desktop computer, server, MP3 player, smart medical device, television, digital video disc (DVD) player, audio equipment, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air purifier, smart mirror, smart window, electronic key, electronic photo frame, digital billboard, security control panel, etc. Wearable devices can be smartwatches, rings, bracelets, anklets, necklaces, glasses, contact lenses, head-mounted devices (HMDs), skin pads, electronic tattoos, or bio-implanted circuits, etc.

[0038] Electronic device 10 can be implemented as a packaged module, component, etc. That is, electronic device 10 can operate as a component included in a computing system. For example, the computing system can be implemented as various devices, including digital cameras, mobile devices, smartphones, personal computers (PCs), tablet PCs, laptop computers, personal digital assistants (PDAs), enterprise digital assistants (EDAs), portable multimedia players (PMPs), wearable devices, black boxes, robots, autonomous vehicles, etc.

[0039] Image sensor 110 acquires images by sensing optical signals. Therefore, image sensor 110 can be implemented as a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.

[0040] The image may include pixel data. Pixel data can be acquired by multiple pixels arranged along rows and columns. A pixel data may include information associated with a pixel for which the corresponding pixel data is acquired. Specifically, pixel data may include information about pixel value, color, position, and exposure time. The pixel value may represent the brightness of the optical signal sensed by the corresponding pixel. The pixel value may have a value within a range of allocated data bits. For example, when the data bits correspond to 8 bits, the pixel value may have a natural value in the range of 0 to 255. Position may represent the row and column in which the corresponding pixel is arranged. Color may represent the color of the optical signal sensed by the corresponding pixel. For example, the color may be one of red, green, and blue. However, this is only one implementation, and the color is not limited to the colors described above. Exposure time may represent the time it takes for the corresponding pixel to sense the optical signal.

[0041] Image sensor 110 can acquire images including normal pixel values ​​and short-exposure pixel values. Normal pixel values ​​can be pixel values ​​sensed during a first exposure time, and short-exposure pixel values ​​can be pixel values ​​sensed during a second exposure time. The second exposure time can be a shorter time than the first exposure time. That is, image sensor 110 can acquire images including pixel values ​​sensed during different exposure times. However, this is only one implementation. The type and number of exposure times are not limited to this and can be modified and implemented differently.

[0042] The processor 130 can control the overall operation of the electronic device 100. For this purpose, the processor 130 may include at least one processing device. For example, the processor 130 may include at least one of an image signal processor (ISP), a digital signal processor (DSP), an application processor (AP), a graphics processing unit (GPU), a central processing unit (CPU), a controller, etc.

[0043] Upon receiving a user request, processor 130 can control image sensor 110 to acquire an image. Specifically, processor 130 can receive the user request via an input interface. For example, the input interface can be implemented as a touch-sensing panel, keyboard, mouse, button, microphone, etc. Alternatively, processor 130 can receive the user request from an external device via a communication interface. Processor 130 can send a command to image sensor 110 instructing image sensor 110 to acquire an image. In addition, processor 130 can receive the image acquired from image sensor 110. The image may include normal pixel values ​​sensed during a first exposure time and short-exposure pixel values ​​sensed during a second exposure time shorter than the first exposure time.

[0044] Additionally, upon receiving an image, the processor 130 can use short-exposure pixel values ​​and normal pixel values ​​included in the image to determine each region of the image as a moving region or a normal region. The processor 130 can acquire flag information indicating the result obtained by determining each region of the image as a moving region or a normal region. A moving region can represent a region among multiple regions of the image where object movement occurs. A normal region can represent a region among multiple regions of the image where object movement does not occur.

[0045] Additionally, processor 130 can output a restored image corrected from short-exposure pixel values ​​included in the image. Specifically, processor 130 can acquire restored pixel values ​​corrected from short-exposure pixel values ​​included in the image. Processor 130 can acquire a restored image including normal pixel values ​​and restored pixel values ​​included in the image. For example, processor 130 can use peripheral pixel values ​​of short-exposure pixel values ​​to acquire a first restored pixel value. The first restored pixel value can be a value used to correct short-exposure pixel values ​​included in motion regions. For example, short-exposure pixel values ​​included in motion regions can be corrected to a first restored value. Processor 130 can use an exposure ratio to acquire a second restored pixel value. The second restored pixel value can be a value used to correct short-exposure pixel values ​​included in normal regions. For example, short-exposure pixel values ​​included in normal regions can be corrected to a second restored value. Processor 130 can output a restored image including either the first restored pixel value included in motion regions or the second restored pixel value included in normal regions. Differences between normal pixel values ​​and pixel values ​​with different exposure times appear in short-exposure pixel values, and noise is likely to appear in short-exposure pixel values. Therefore, short-exposure pixel values ​​are corrected, thereby solving this problem.

[0046] Reference Figure 1B The electronic device 100 may include a communication interface 120 and a processor 130.

[0047] The communication interface 120 can send / receive information or signals via communication according to various communication schemes. For this purpose, the communication interface 120 may include circuitry for performing wired or wireless communication. For example, the communication interface 120 may include at least one of the following: Bluetooth module, Wi-Fi module, cellular communication module, near field communication (NFC) module, infrared communication module, Zigbee communication module, ultra-wideband (UWB) module, ultrasound module, Ethernet module, local area network (LAN) module, mobile high-definition link (MHL), universal serial bus (USB), display port (DP), high-definition multimedia interface (HDMI), digital visual interface (DVI), lightning module, optical communication module, satellite communication module, and components.

[0048] Electronic device 100 can receive images from an external device via communication interface 120. The external device may be a device including the image sensor 110 described above. Electronic device 100 may be a server. However, this is only one embodiment; electronic device 100 may be one of the various devices described above, such as a PC, set-top box, smart speaker, and television.

[0049] The image received through the communication interface 120 may include normal pixel values ​​and short-exposure pixel values ​​sensed during different exposure times. The image may include normal pixel values ​​sensed during a first exposure time and short-exposure pixel values ​​sensed during a second exposure time shorter than the first exposure time.

[0050] The processor 130 can receive images via the communication interface 120. Furthermore, upon receiving an image, the processor 130 can use the short-exposure pixel values ​​and normal pixel values ​​included in the image to determine various regions of the image as moving or normal regions. Additionally, the processor 130 can output a restored image corrected from the short-exposure pixel values ​​included in the image. As described above, the electronic device 100 can use images acquired from the image sensor 110 installed therein or from an external device to detect moving regions and can output a restored image corrected from the short-exposure pixel values ​​included in the image.

[0051] As described above, the electronic device 100 can detect a moving region using a single image without any prior image. Therefore, the memory space required to store previous images and the power consumption can be reduced. Furthermore, compared to an image comprising all simultaneously sensed pixel values, the electronic device 100 can acquire a restored image with undegraded or improved image quality.

[0052] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

[0053] Figure 2 This is a diagram illustrating an image sensor according to an embodiment of the present disclosure.

[0054] Reference Figure 2 The image sensor 110 may include an optical lens LS, a pixel array 111, a line decoder 112, a timing generator 113, a signal converter 114, and an output buffer 115.

[0055] An optical lens LS refracts optical signals. The optical signals refracted by the optical lens LS can be transmitted to the pixel array 111. For this purpose, the optical lens LS can be a component of microlenses arranged on the pixel array 111. In addition, the optical signals can include information about the object, such as the object's size, shape, position, color, etc. The object can be an external subject being photographed, such as a person, object, animal, or plant.

[0056] Pixel array 111 may include multiple pixels. These pixels may be arranged in both row and column directions. Each pixel in pixel array 111 may correspond to a pixel value in an image IMG. The pixels in pixel array 111 may be pixels arranged in a physical area, while the pixel values ​​in the image IMG may be pixel values ​​arranged in a digital area. The pixels in pixel array 111 and the pixel values ​​in the image IMG may have a correspondence relative to the same arrangement position.

[0057] Each pixel of the pixel array 111 may include a color filter and sensing circuitry.

[0058] A color filter can be positioned on top of the sensing circuit. Optical signals refracted by the optical lens LS can reach the sensing circuit while passing through the color filter. The color filter can be used to allow optical signals of a specific color or wavelength to pass through while blocking optical signals of other colors or wavelengths. The color filter can have a color corresponding to one of red, green, and blue. Depending on the color of the color filter included in the pixel, the pixel can be designated as a red pixel, a green pixel, or a blue pixel. However, this is only one implementation method, and the color of the color filter can be varied.

[0059] The pixels of pixel array 111 can be arranged according to a Bayer pattern. Specifically, groups of pixels, each having the same color, can be arranged alternately. That is, pixels of the same color can be included in a group, and pixels in a group can be arranged in an m×n pattern. Here, m and n are natural numbers. For example, in the case of a quaternary Bayer pattern, each group can include pixels of the same color arranged in a 2×2 pattern.

[0060] A sensing circuit may be disposed at the bottom of the color filter. The sensing circuit may include a light-sensing element. The light-sensing element can generate an electrical pixel signal from an optical signal received using the photoelectric effect. For example, when an optical signal is received by the light-sensing element for the duration of exposure, a charge corresponding to the optical signal can accumulate in the sensing circuit, and a pixel signal corresponding to the charge accumulated in the sensing circuit can be generated. For this purpose, the light-sensing element can be implemented as various semiconductor devices such as pn junction diodes, PIN photodiodes, avalanche photodiodes (APDs), and phototransistors.

[0061] The line decoder 112 can select pixels for reading pixel data under the control of the timing generator 113. For example, the line decoder 112 can select pixels located on the row corresponding to the address output from the timing generator 113 from among a plurality of pixels included in the pixel array 111 in response to address and control signals.

[0062] Timing generator 113 can control line decoder 112 and signal converter 114 to read pixel data from pixels located in a specific row of pixel array 111. For example, timing generator 113 can sequentially output row addresses representing pixels located in a specific row to line decoder 112. Timing generator 113 can also output commands to line decoder 112 for adjusting the exposure time of pixels located in a specific column.

[0063] Signal converter 114 can acquire pixel data using pixel signals received from pixel array 111 and row addresses received from timing generator 113. Additionally, signal converter 114 can transmit pixel data to output buffer 115.

[0064] Specifically, signal converter 114 can be connected to the pixels of pixel array 111 via multiple column lines. The number of pixels is proportional to the number of rows connected to a column line. Signal converter 114 can receive pixel signals about pixels located in a row via each column line. That is, signal converter 114 can receive pixel signals row by row. Signal converter 114 can acquire pixel values ​​corresponding to the level of the pixel signals received from pixel array 111. For example, the pixel signal can be a voltage. For this purpose, signal converter 114 may include an analog-to-digital converter. Additionally, signal converter 114 can acquire information about the color of pixels located in the row corresponding to the row address received from timing generator 113 based on the color information of pixels mapped to each row address.

[0065] Output buffer 115 can be connected to signal converter 114 via multiple column lines. Output buffer 115 can store pixel values ​​received from signal converter 114 row by row. When the pixel values ​​for all rows have been stored, output buffer 115 can output an image IMG including the stored pixel values.

[0066] Figure 3 This is a diagram illustrating the image processing operation of a processor according to an embodiment of the present disclosure.

[0067] Reference Figure 3 The processor 130 can use the short-exposure pixel value S and normal pixel value, as well as the exposure ratio, included in the selected area of ​​the image to acquire the marking information included in the motion graph 430. The marking information may be information indicating whether the selected area is a motion area or a normal area.

[0068] Specifically, the processor 130 may receive the image 410 via the image sensor 110 or the communication interface 120. The image 410 may include normal pixel values ​​sensed during a first exposure time and short exposure pixel values ​​S sensed during a second exposure time shorter than the first exposure time.

[0069] Additionally, processor 130 can acquire a corrected image 420 having a corrected pixel value C corrected from the short-exposure pixel value S included in image 410. Corrected image 420 may include the normal pixel values ​​and the corrected pixel value C included in image 410. That is, corrected image 420 may be an image in which the short-exposure pixel value S included in image 410 is replaced with the corrected pixel value C. Regions of image 410 may correspond to regions of corrected image 420. The region units described in this disclosure may be regions having pixel values ​​of the same color and arranged continuously among the pixel values ​​included in image 410 or corrected image 420. The size of each region may represent the number of pixel values ​​arranged in the row and column directions. For example, the size of each region may be 2×2, 3×3, etc.

[0070] Additionally, the processor 130 can use the normal pixel values ​​and corrected pixel values ​​C included in the selected area to obtain flag information indicating whether the selected area of ​​the corrected image 420 is a motion area or a normal area. The flag information can be included in the motion image 430. For example, when the flag information has a value of 1, this indicates that the selected area is a motion area. When the flag information has a value of 0, this indicates that the selected area is a normal area.

[0071] Additionally, the processor 130 can use flag information about selected regions included in the motion graph 430 to obtain flag information about empty regions to be included in the final motion graph 440. The motion graph 430 may include multiple regions corresponding to multiple regions of the image 410 or the corrected image 420, respectively. Selected regions may represent regions of the multiple regions of the image 410 that include short exposure pixel values ​​S or regions of the multiple regions of the corrected image 420 that include corrected pixel values ​​C. Empty regions may represent regions of the multiple regions of the image 410 that do not have short exposure pixel values ​​S or regions of the multiple regions of the corrected image 420 that do not have corrected pixel values ​​C.

[0072] Additionally, processor 130 can use image 410 and final motion map 440 to output restored image 450. Corrected image 420 can be used in place of image 410. Motion map 430 can be used in place of final motion map 440.

[0073] For example, processor 130 can use the marker information included in the final motion graph 440 to determine whether a selected region of image 410 is a moving region or a normal region.

[0074] The processor 130 can acquire a first recovered pixel value I using short-exposure pixel values ​​S included in the motion region of the image 410, according to a first method. The first method is a method of interpolating the peripheral pixel values ​​of the short-exposure values ​​S included in the motion region.

[0075] Alternatively, the processor 130 can obtain a second restored pixel value C using the short-exposure pixel value S included in the normal area of ​​the image, according to a second method. The second method can be a method of calculating a value obtained by multiplying the short-exposure pixel value S included in the normal area by the exposure ratio. That is, the second restored pixel value C included in the normal area can be equal to the value of the corrected pixel value C included in the normal area of ​​the corrected image 420.

[0076] Additionally, the processor 130 can acquire a restored image 450 having a first restored pixel value I of a motion region or a second restored pixel value C of a normal region. The restored image 450 may include the first restored pixel value I or the second restored pixel value C, as well as normal pixel values ​​included in image 410. That is, the restored image 450 may be an image in image 410 where the short exposure pixel value S is replaced with the first restored pixel value I or the second restored pixel value C. Alternatively, the restored image 450 may be an image in corrected image 420 where the corrected pixel value C in the motion region is replaced with the first restored pixel value I.

[0077] Figure 4 This is a diagram illustrating a detailed configuration of a processor according to an embodiment of the present disclosure.

[0078] Reference Figure 4 The processor 130 may include at least one of an exposure corrector 131, a motion ratio calculator 132, a motion area detector 133, a noise filter 134, and a pixel corrector 135.

[0079] In one embodiment, each of the exposure corrector 131, motion ratio calculator 132, motion area detector 133, noise filter 134, and pixel corrector 135 may be implemented as a software module including instructions, programs, etc., executed by the processor 130. The processor 130 or electronic device 100 may also include memory storing the software module. In another embodiment, each of the exposure corrector 131, motion ratio calculator 132, motion area detector 133, noise filter 134, and pixel corrector 135 may be implemented as hardware such as logic circuitry, components, ISP, DSP, AP, CPU, or GPU.

[0080] Exposure corrector 131 can obtain a corrected pixel value using a short-exposure pixel value and an exposure ratio. The exposure ratio can be the ratio of a first exposure time for a normal pixel value to a second exposure time for a short-exposure pixel value. Exposure corrector 131 can obtain the corrected pixel value as the value obtained by multiplying the short-exposure pixel value by the exposure ratio. Additionally, exposure corrector 131 can obtain a corrected image including both the corrected pixel value and the normal pixel value. This will be referred to... Figure 6 To describe in more detail.

[0081] The motion ratio calculator 132 can obtain the motion ratio with respect to each of a plurality of regions of a corrected image. For example, the motion ratio calculator 132 can select any one of the plurality of regions of the corrected image as the selected region. The selected region can be a region of the plurality of regions of the corrected image that includes short-exposure pixel values. The motion ratio calculator 132 can use normal pixel values ​​and at least one short-exposure pixel value included in the selected region to obtain the motion ratio with respect to the selected region.

[0082] In one implementation, the motion ratio calculator 132 can obtain a motion ratio that represents the ratio of the average of normal pixel values ​​included in the selected area to the average of corrected pixel values ​​included in the selected area.

[0083] The motion region detector 133 can determine whether a selected region of the corrected image, among multiple regions of the corrected image, is a moving region or a normal region based on the motion ratio of the selected region. The selected region can be a region among the multiple regions of the corrected image that includes short-exposure pixel values. The motion region detector 133 can acquire flag information indicating whether the selected region among the multiple regions of the corrected image is a moving region or a normal region based on the motion ratio of the selected region. The set of flag information for each selected region among the multiple regions can be... Figure 3 The motion diagram shown is 430.

[0084] In one implementation, when the difference between the motion ratio and the first reference value exceeds a second reference value, the motion region detector 133 acquires flag information with a first value indicating that the selected region is a motion region. In another implementation, when the difference between the motion ratio and the first reference value is equal to or less than the second reference value, the motion region detector 133 acquires flag information with a second value indicating that the selected region is a normal region. The first value can be 1, and the second value can be 0. However, this is only one implementation; the first and second values ​​can be modified and implemented as various values. This will be referred to... Figure 7A and Figure 7B To describe in more detail.

[0085] Furthermore, the first and second reference values ​​may be pre-stored in the memory included in the processor 130 or electronic device 100. For example, the first reference value may be 1, and the second reference value may be 0.3. However, this is only one implementation method, and the first and second reference values ​​may be modified and specifically implemented into various values.

[0086] The motion region detector 133 can acquire labeling information about the empty region using labeling information about the surrounding regions of the empty region. The empty region can be a region among multiple regions that lacks any labeling information. That is, the empty region can be a region among multiple regions of image 410 that does not have any short-exposure pixel values. The set of labeling information about the corresponding multiple regions can be... Figure 3 The final motion diagram shown is 440.

[0087] Specifically, the motion region detector 133 can allow the central region of the kernel to correspond to an empty region. For example, the motion region detector 133 can align the kernel such that the central region of the kernel lies on an empty region. The kernel may include multiple regions arranged according to a predetermined arrangement. For example, the kernel may include peripheral regions and a central region arranged in a cross arrangement.

[0088] The motion region detector 133 can obtain flag information indicating whether an empty region is a motion region or a normal region based on the values ​​of flag information included in a region corresponding to the peripheral region of the central region included in the kernel. This will refer to... Figures 8A to 8C To describe in more detail.

[0089] Noise filter 134 can be used Figure 3 The kernel in the final motion diagram 440 shown is used to remove noise. The kernel may include multiple regions arranged according to a predetermined pattern. For example, the kernel may include peripheral regions and a central region arranged in a cross pattern.

[0090] Specifically, noise filter 134 allows the central region of the kernel to correspond to flag information indicating that a selected region among multiple regions is a moving region. Based on the value of the flag information included in the moving flag region, indicating the region among multiple regions corresponding to the peripheral region of the kernel and the moving region, noise filter 134 can maintain the flag information included in the moving flag region or change the flag information included in the moving flag region to flag information indicating that the selected region is a normal region. This will refer to... Figures 9A to 9C To describe in more detail.

[0091] The pixel corrector 135 can acquire restored pixel values ​​for correcting short-exposure pixel values. The pixel corrector 135 can acquire and output a restored image including the restored pixel values.

[0092] Specifically, when the flag information indicates that the selected area is a moving area, the pixel corrector 135 can use the peripheral pixel values ​​of the short exposure pixel values ​​to obtain the first recovered pixel value for correcting the short exposure pixel values.

[0093] In one implementation, when the flag information indicates that the selected area is a motion area, the pixel corrector 135 may select a normal pixel value from the normal pixel values ​​that has the same color as the short-exposure pixel value included in the selected area. The pixel corrector 135 may obtain a value as a first restored pixel value based on the distance between the short-exposure pixel value and the peripheral normal pixel value that is closest to the short-exposure pixel value in multiple directions, and the weight of the peripheral normal pixel value.

[0094] Multiple directions can include up, down, left, and right.

[0095] In an implementation, when a first pixel value that is closest to the short exposure pixel value in any of the selected normal pixel values ​​in any of the multiple directions does not exist, the pixel corrector 135 may obtain a second pixel value that is closest to the short exposure pixel value in the opposite direction as the first pixel value.

[0096] Furthermore, when the flag information indicates that the selected area is a normal area, the pixel corrector 135 can obtain a second restored pixel value by multiplying the short-exposure pixel values ​​included in the selected area by the exposure ratio. The operation of the pixel corrector 135 will refer to Figures 10A to 10C To describe in more detail.

[0097] Figure 5A This is a diagram illustrating an embodiment according to the present disclosure.

[0098] Reference Figure 5AImage 510 may include multiple pixel values. These pixel values ​​can be categorized into normal pixel values ​​and short-exposure pixel values ​​based on exposure time. They can also be categorized by color into Gr pixel values ​​Gr1, Gr2, Gr3, and Gr4; R pixel values ​​R1, R2, R3, and R4; B pixel values ​​B1, B2, B3, and B4; and Gb pixel values ​​Gb1, Gb2, Gb3, and Gb4. The arrangement of these pixel values ​​in image 510 can be determined based on the pixel arrangement of image sensor 110.

[0099] Image 510 may include multiple Bayer regions. These multiple Bayer regions may be arranged repeatedly according to the number of pixel values ​​included in the row and column directions of the image 510. Any one Bayer region 10 will be described, as the same description can be applied to each of the multiple Bayer regions.

[0100] The Bayer region 10 may include multiple regions. Each of the multiple regions may be a region with pixel values ​​of the same color. For example, the Bayer region 10 may include first to fourth regions. The first region may include Gr pixel values ​​Gr1, Gr2, Gr3, and Gr4 representing green. The second region may include R pixel values ​​R1, R2, R3, and R4 representing red. The third region may include B pixel values ​​B1, B2, B3, and B4 representing blue. The fourth region may include Gb pixel values ​​Gb1, Gb2, Gb3, and Gb4 representing green. The first region having Gr pixel values ​​Gr1, Gr2, Gr3, and Gr4 representing green and the fourth region having Gb pixel values ​​Gb1, Gb2, Gb3, and Gb4 representing green may be located diagonally.

[0101] The Bayer region 10 may include normal pixel values ​​and short-exposure pixel values. Normal pixel values ​​may be pixel values ​​sensed during a first exposure time ET_N, and short-exposure pixel values ​​may be pixel values ​​sensed during a second exposure time ET_S, which is shorter than the first exposure time ET_N.

[0102] In an implementation, the number of short-exposure pixel values ​​ET_S included in image 510 may be less than the number of normal pixel values ​​ET_N included in image 510. For example, any region 11 included in a plurality of regions within a Bayer region 10 may include three normal pixel values ​​11n and one short-exposure pixel value 11s. This is to reduce noise by limiting the number of short-exposure pixel values ​​ET_S, since noise is likely to occur due to the short exposure time of the short-exposure pixel values ​​ET_S.

[0103] In one implementation, the first and fourth regions, among the multiple regions included in image 510, may include at least one short-exposure pixel value ET_S. The first and fourth regions may include multiple normal pixel values ​​arranged at locations other than the short-exposure pixel value ET_S. The short-exposure pixel value ET_S may be arranged at the same location in both the first and fourth regions. However, this is merely one implementation, and the number and arrangement of the short-exposure pixel values ​​ET_S may be modified and implemented differently.

[0104] Figure 5B This is a diagram illustrating the exposure time according to an embodiment of the present disclosure. Figure 5C This is a diagram showing an image obtained according to an embodiment of the present disclosure based on exposure time.

[0105] Reference Figure 5B According to embodiments of this disclosure, the first exposure time ET_N can be longer than the second exposure time ET_S. At least a portion of the first exposure time ET_N and at least a portion of the second exposure time ET_S can overlap.

[0106] Reference Figure 5B (1) According to the embodiment, the first exposure time ET_N can be a time period from the first time t1 to the third time t3. The first time t1 can be the time when the first pixel, which sets the first exposure time ET_N, begins to sense the optical signal, and the third time t3 can be the time when the first pixel ends the sensing operation.

[0107] The second exposure time ET_S can be a time interval from the second time t2 to the third time t3. The second time t2 can be the time when the second pixel, for which the second exposure time ET_S is set, begins its sensing operation to detect the optical signal, and the third time t3 can be the time when the second pixel ends its sensing operation. That is, the first pixel can begin its sensing operation earlier than the second pixel, and then the second pixel can begin its sensing operation. Subsequently, the first pixel and the second pixel can end their sensing operations simultaneously.

[0108] Reference Figure 5B (2) According to the implementation method, the first exposure time ET_N can be a time period from the first time t1 to the third time t3. The second exposure time ET_S can be a time period from the first time t1 to the second time t2. That is, the first pixel and the second pixel can start the sensing operation simultaneously. Subsequently, the second pixel can end the sensing operation earlier than the first pixel. Subsequently, the first pixel can end the sensing operation. In addition, the above implementation method is only an implementation method, and the start time and end time of the sensing operation can be modified and implemented differently.

[0109] According to embodiments of this disclosure, image sensor 110 can acquire an image including normal pixel values ​​sensed up to a first exposure time ET_N and short exposure pixel values ​​sensed up to a second exposure time ET_S. Processor 130 can acquire a normal image with respect to the first exposure time ET_N by inputting the normal pixel values ​​into a deep learning model. Additionally, processor 130 can acquire a short exposure image with respect to the second exposure time ET_S by inputting the short exposure pixel values ​​into a deep learning model. The deep learning model can be an artificial intelligence model that learns to output an image using pixel values.

[0110] For example, refer to Figure 5B (1) Assume the first exposure time ET_N is the time interval from the first time t1 to the third time t3, the second exposure time ET_S is the time interval from the second time t2 to the third time t3, and the object moves from left to right from the first time t1 to the third time t3. (Refer to...) Figure 5C (1) and (2), the short exposure image 530 and the normal image 550 can be obtained from an image through a deep learning model.

[0111] The short exposure image 530 may include a first region 531, a second region 533, a third region 535, and a fourth region 537. The third region 535 may represent the area where the object is located during the second exposure time ET_S from the second time t2 to the third time t3. That is, the third region 535 may include pixel values ​​representing the object. The first region 531, the second region 533, and the fourth region 537 may include pixel values ​​representing another object sensed during the second exposure time ET_S.

[0112] The normal image 550 may include a first region 551, a second region 553, a third region 555, and a fourth region 557. The first region 551 to the third region 555 may represent the region where the object ET_N is located during the first exposure time ET_N from the first time t1 to the third time t3. That is, the first region 551 to the third region 555 may include pixel values ​​representing the object.

[0113] Processor 130 can determine the region where the object is located during a specific time period based on the difference between the regions corresponding to the short exposure image 530 and the normal image 550. The specific time period can be from a first time t1 to a second time t2, where the first exposure time ET_N and the second exposure time ET_S do not overlap. Processor 130 can determine the region where the object is located during the time period from the second time t2 to the third time t3 using the short exposure image 530. Processor 130 can determine the direction of object movement by each change in the region where the object is located. That is, processor 130 can determine the direction of movement of the object included in the current image. The direction of movement of the object included in the current image can be used to determine the region and direction in which the object to move in the next image. The current image and the next image can be consecutive images among a plurality of images acquired sequentially according to time, such as motion images. As described above, the accuracy of detecting the region where the object is located or the region where the object's movement occurs in the next image can be improved.

[0114] Figure 5D This is a diagram illustrating an embodiment according to the present disclosure.

[0115] Figure 5D (1) and (2) show the Bayer region of the image. (See reference...) Figure 5D According to embodiments of the present disclosure, in (1) and (2), an image may include multiple regions. Each of the multiple regions may include pixel values ​​having the same color.

[0116] Reference Figure 5D (1) According to an embodiment of the present disclosure, each region of the image may include 2×2 pixel values. Each region of the image may include pixel values ​​sensed during the first exposure time ET_1 to the fourth exposure time ET_4. For example, in each region of the image, the pixel value at position (1,1) may correspond to the value sensed up to the first exposure time ET_1, the pixel value at position (1,2) may correspond to the value sensed up to the second exposure time ET_2, the pixel value at position (2,1) may correspond to the value sensed up to the third exposure time ET_3, and the pixel value at position (2,2) may correspond to the value sensed up to the fourth exposure time ET_4. The first exposure time ET_1 to the fourth exposure time ET_4 may all be different times, or some of the first exposure time ET_1 to the fourth exposure time ET_4 may be different times.

[0117] Reference Figure 5D(2) According to embodiments of the present disclosure, each region of the image may include 3×3 pixel values. Each region of the image may include pixel values ​​sensed during the first exposure time ET_1 to the ninth exposure time ET_9. For example, in each region of the image, the pixel value at position (1,1) to the pixel value at position (3,3) may correspond to the values ​​sensed up to the first exposure time ET_1 to the ninth exposure time ET_9. The first exposure time ET_1 to the ninth exposure time ET_9 may all be different times, or some of the first exposure time ET_1 to the ninth exposure time ET_9 may be different times. Furthermore, the above embodiments are merely embodiments, and may be modified and implemented differently depending on the arrangement of exposure times and the color of pixel values.

[0118] Figure 6 This is a diagram illustrating the operation of an exposure corrector according to an embodiment of the present disclosure. Figure 6 An image 610 received from an image sensor 110 or a communication interface 120 is shown.

[0119] Reference Figure 6 In (1) and (2), the exposure corrector 131 can acquire a corrected image 620 corrected from image 610 using an exposure ratio. Image 610 may include Bayer regions arranged repeatedly along the row and column directions. The Bayer regions may include a first region to a fourth region. The first region may include pixel values ​​at (1,1) to (2,2), the second region may include pixel values ​​at (1,3) to (2,4), the third region may include pixel values ​​at (3,1) to (4,2), and the fourth region may include pixel values ​​at (3,3) to (4,4). The Bayer regions may include short-exposure pixel values ​​at (2,2), short-exposure pixel values ​​at (4,4), and normal pixel values ​​at other locations.

[0120] Exposure corrector 131 can acquire corrected pixel values ​​621C and 622C, corrected using exposure ratios from short-exposure pixel values ​​611S and 612S included in image 610. Additionally, exposure corrector 131 can acquire a corrected image 620 having corrected pixel values ​​621C and 622C, as well as normal pixel values.

[0121] For example, exposure corrector 131 can acquire corrected pixel values ​​621C and 622C as values ​​obtained by multiplying short exposure pixel values ​​611S and 612S by the exposure ratio. The exposure ratio can be the ratio of a first exposure time ET_N to a second exposure time ET_S, where the second exposure time ET_S is shorter than the first exposure time ET_N. For example, as... Figure 6 As shown in (2), the exposure ratio can be obtained by dividing the first exposure time ET_N by the second exposure time ET_S.

[0122] Figure 7A This is a diagram illustrating the operation of the motion ratio calculator and motion area detector according to embodiments of the present disclosure.

[0123] Reference Figure 7A In (1) and (2), the motion ratio calculator 132 can obtain the motion ratio 721R or 722R of a selected region 711C or 712C among multiple regions of the corrected image 710.

[0124] Specifically, the motion ratio calculator 132 can use the normal pixel value PV and the corrected pixel value cPV included in the selected region 711C or 712C to obtain the motion ratio 721R or 722R with respect to the selected region 711C or 712C. For this purpose, the motion ratio calculator 132 can select a region including the corrected pixel value from among a plurality of regions of the corrected image 710 as the selected region 711C or 712C.

[0125] In an implementation, the motion ratio calculator 132 can obtain the ratio of the average of the normal pixel values ​​PV included in the selected region 711C or 712C to the average of the corrected pixel values ​​cPV included in the selected region 711C or 712C as the motion ratio. For example, as Figure 7A As shown in (2), the motion ratio 721R or 722R can be a value obtained by dividing the average of the normal pixel values ​​PV included in the selected region 711C or 712C by the average of the corrected pixel values ​​cPV included in the selected region 711C or 712C.

[0126] In another embodiment, the motion ratio calculator 132 may obtain the ratio of the median of the normal pixel values ​​PV included in the selected region 711C or 712C to the median of the corrected pixel values ​​cPV included in the selected region 711C or 712C as the motion ratio 721R or 722R. When multiple values ​​are arranged in ascending order, the median may refer to the value that is located at the center of the multiple values. In another embodiment, the motion ratio calculator 132 may obtain the ratio of the maximum frequency value of the normal pixel values ​​PV included in the selected region 711C or 712C to the maximum frequency value of the corrected pixel values ​​cPV included in the selected region 711C or 712C as the motion ratio 721R or 722R.

[0127] Furthermore, the motion ratio calculator 132 can generate a dataset 720 using motion ratios 721R or 722R with respect to selected regions 711C or 712C. Multiple regions included in the dataset 720 and multiple regions included in the corrected image can correspond to each other at the same location. Additionally, the dataset 720 can include null values ​​for unselected regions. Null values ​​can indicate a state where no motion ratio exists. Unselected regions can be regions among multiple regions of the corrected image 710 that do not have a corrected pixel value cPV.

[0128] Reference Figure 7A In (2) and (3), the motion region detector 133 can determine whether the selected region 711C or 712C of the corrected image 710 is a motion region or a normal region based on the motion ratio 721R or 722R with respect to the selected region 711C or 712C of the corrected image 710, and acquire flag information 731F or 732F indicating that the selected region 711C or 712C is one of the motion region and the normal region. The flag information 731F or 732F may include one of a first value and a second value. The first value may be 1, indicating a motion region. The second value may be 0, indicating a normal region. However, this is only one implementation, and the first value and the second value may be modified and implemented differently.

[0129] In an implementation, when the difference between the motion ratio 721R of the first selected region 711C and the first reference value is equal to or less than the second reference value, the motion region detector 133 can acquire flag information 731F with a second value indicating that the first selected region 711C is a normal region.

[0130] In an implementation, when the difference between the motion ratio 722R of the second selected region 712C and the first reference value exceeds the second reference value, the motion region detector 133 can acquire flag information 732F with a first value indicating that the second selected region 712C is a motion region.

[0131] For example, the first reference value can be 1, and the second reference value can be 0.3 or 0.5. However, this is only one implementation method, and the first and second reference values ​​can be modified and specifically implemented into various values.

[0132] Furthermore, the motion region detector 133 can generate a motion map 730 using the flag information 731F and 732F regarding the selected regions 711C and 712C. Multiple regions included in the motion map 730 and multiple regions included in the corrected image 710 can correspond to each other at the same location. Additionally, the motion map 730 can include null values ​​for unselected regions. Null values ​​can indicate a state where flag information is absent. An unselected region can be a region among the multiple regions of the corrected image 710 that has no corrected pixel value cPV. Regions among the multiple regions of the motion map 730 that correspond to unselected regions and include null values ​​can be designated as empty regions.

[0133] Figure 7B This is a diagram illustrating the form of marking information according to embodiments of the present disclosure.

[0134] Reference Figure 7B (1) According to an embodiment, motion graph 730 may include a marker information corresponding to each region having pixel values ​​of the same color. The marker information included in the corresponding region may be applied collectively to the pixel values ​​included in the corresponding region. See also... Figure 7B (2) According to the embodiment, the motion graph 735 may include a flag information corresponding to each pixel value. That is, the flag information may exist in units of regions or in units of pixel values.

[0135] Figures 8A to 8C This is a diagram illustrating the operation of a motion region detector according to an embodiment of the present disclosure.

[0136] Reference Figure 8A The motion region detector 133 can use the first kernel 830 to generate a motion map 850 from the motion map 810, which has marking information about the empty region 811c. The motion map 850 can be a map in which the empty region 811c of the motion map 810 is filled with marking information. That is, even regions among multiple regions in the image that do not have any short exposure pixel values ​​can be filled with marking information indicating whether the corresponding region is a moving region or a normal region.

[0137] Specifically, the motion region detector 133 may allow the central region 830c of the first kernel 830 to correspond to the empty region 811c of the motion map 810. For example, the motion region detector 133 may align the first kernel 830 such that the empty region 811c of the motion map 810 and the central region 830c of the first kernel 830 overlap each other.

[0138] The first core 830 may include a peripheral region and a central region 830c arranged according to a predetermined configuration. In one embodiment, the peripheral region of the first core 830 may include the region closest to the central region 830c in the upward, downward, leftward, and rightward directions. The upward and downward directions may be row directions, and the left and right directions may be column directions. That is, the first core 830 may include multiple regions arranged in a cross configuration. However, this is only one embodiment, and the arrangement and number of peripheral regions may be modified and implemented differently.

[0139] The motion region detector 133 can align the central region 830c of the first core 830 with the empty region 811c of the motion map 810. The empty region 811c may correspond to the central region 830c of the first core 830, and the peripheral region of the empty region 811c may correspond to the peripheral region of the first core 830. The empty region 811c may be a region without any marking information, and the peripheral region of the empty region 811c may include a region with marking information.

[0140] The motion region detector 133 can obtain flag information indicating that the empty region 811c of the motion map 810 is one of the motion region and the normal region, based on the value of the flag information included in the region corresponding to the peripheral region of the central region 830c of the first core 830.

[0141] For example, the peripheral region of the central region 830c of the first core 830 in the upward direction can correspond to the region of the empty region 811c in the upward direction of the motion diagram 810. The peripheral region of the central region 830c of the first core 830 in the downward direction can correspond to the region of the empty region 811c in the downward direction of the motion diagram 810. The peripheral region of the central region 830c of the first core 830 in the left direction can correspond to the region of the empty region 811c in the left direction of the motion diagram 810. The peripheral region of the central region 830c of the first core 830 in the right direction can correspond to the region of the empty region 811c in the right direction of the motion diagram 810.

[0142] According to the implementation, when the number of first values ​​of the flag information included in the region corresponding to the peripheral region of the central region 830c of the first core 830 of the motion map 810 exceeds a reference number, the motion region detector 133 can acquire flag information indicating that the empty region 811c of the motion map 810 is a motion region.

[0143] According to the implementation, when the number of first values ​​of the flag information included in the region corresponding to the peripheral region of the central region 830c of the first core 830 of the motion map 810 is equal to or less than the reference number, the motion region detector 133 can acquire flag information indicating that the empty region 811c of the motion map 810 is a normal region.

[0144] The first value of the flag information can indicate that the corresponding area of ​​motion graph 810 is a motion area, and the second value of the flag information can indicate that the corresponding area of ​​motion graph 810 is a normal area. That is, when the number of first values ​​of flag information included in the area of ​​motion graph 810 corresponding to the peripheral area of ​​the central area 830c of the first core 830 exceeds a reference number, motion region detector 133 can use the first values ​​to fill the flag information of empty area 811c. In addition, when the number of first values ​​of flag information included in the area of ​​motion graph 810 corresponding to the peripheral area of ​​the central area 830c of the first core 830 is equal to or less than the reference number, motion region detector 133 can use the second value to fill the flag information of empty area 811c of motion graph 810.

[0145] Reference Figure 8B For example, the base number can be 1. For example, referencing Figure 8B In (1) to (3), when the number of first values ​​of flag information included in the region of motion graph 810 corresponding to the peripheral region of the first core 831, 832 or 833 exceeds the reference number 1, the first values ​​of flag information can be filled into the empty region 811c of motion graph 810 corresponding to the central region of the first core 831, 832 or 833. In another example, referring to Figure 8B In (4) and (5), when the number of first values ​​of the flag information included in the region corresponding to the peripheral region of the first core 834 or 835 of the motion graph 810 is equal to or less than the reference number 1, the second value of the flag information can be filled in the empty region 811c corresponding to the central region of the first core 834 or 835 of the motion graph 810.

[0146] However, this is only one implementation method, and the base number can be modified differently and specifically implemented as 2, 3, etc. Furthermore, the base value can be determined based on the number of regions included in the second core 930.

[0147] Reference Figure 8C According to the implementation, when the central region 830c of the first core 830 is aligned with the region located at the edge of the motion map 810, there may be cases where some peripheral regions of the first core 830 do not overlap with any of the multiple regions of the motion map 810. Marking information included in the region of the motion map 810 corresponding to another peripheral region of the first core 830 can be used.

[0148] In this implementation, the motion region detector 133 can align the central region 830c of the core 830 with the empty region 813c of the motion map 810. When any of the peripheral regions of the core 830 does not correspond to one of the multiple regions of the motion map 810, the motion region detector 133 can select a peripheral region of the core 830 located in the opposite direction to that peripheral region. For example, when the peripheral region of the core 830 outside the motion map 810 is the upper peripheral region located in the upward direction with respect to the central region 830c of the core 830, the motion region detector 133 can select the lower peripheral region located in the downward direction with respect to the central region 830c of the core 830.

[0149] The motion region detector 133 can acquire the flag information included in the lower peripheral region 813b corresponding to the lower peripheral region of the motion graph 810, as the flag information for the upper peripheral region 813t corresponding to the upper peripheral region of the kernel 830 in the motion graph 810. That is, when the upper peripheral region 813t corresponding to the empty region 813 of the central region 830c of the kernel 830 in the upward direction does not exist, the motion region detector 133 can copy the flag information of the lower peripheral region 813b relative to the empty region 813c in the downward direction as the flag information for the upper peripheral region 813t.

[0150] Subsequently, the motion region detector 133 can obtain flag information indicating that the empty region 813c of the motion map 810 is one of the motion region and the normal region, based on the value of the flag information included in the region corresponding to the peripheral region of the central region 830c of the first core 830.

[0151] In addition, return to reference Figure 8A The motion map 810 may include a first region 810N and a second region 810M. Assume that the first region 810N includes areas where no actual motion of the object occurs, and the second region 810M includes areas where actual motion of the object occurs. The first region 810N may include regions with flag information indicating that the region is a normal region and empty regions without flag information. The second region 810M may include regions with flag information indicating that the region is a moving region and empty regions without flag information. Because the signal-to-noise ratio (SNR) of short-exposure pixel values ​​is relatively lower than that of normal pixel values, the flag information obtained using short-exposure pixel values ​​may be inaccurate. Alternatively, the motion map 810 may include a noise region 815 with flag information indicating that a noise region 815 is a moving region. (See reference...) Figures 9A to 9C Describe in detail the method for removing noise region 815.

[0152] Figures 9A to 9CThis is a diagram illustrating the operation of a noise filter according to an embodiment of the present disclosure.

[0153] Reference Figures 9A to 9C According to embodiments of this disclosure, noise filter 134 can use a second kernel 930 to remove noise from motion graph 910. Motion graph 910 may include a first region 910N and a second region 910M. Assume that the first region 910N may include a region where no actual movement of the object occurs, and the second region 910M includes a region where movement of the object actually occurs. Assume that even if no actual movement of the object occurs, the first region 910N includes a noise region 915 having flag information indicating that the noise region 915 is a moving region. Noise filter 134 can use the second kernel 930 to obtain a final motion graph 950 corrected from the flag information regarding the noise region 915 of motion graph 910. To this end, noise filter 134 can determine whether a motion flag region is a noise region, and when it is determined that a motion flag region is a noise region, correct the motion flag region to a normal flag region. That is, when it is determined that a region with a first value of flag information is a noise region, the value of the flag information for the corresponding region can be corrected to a second value.

[0154] Specifically, the noise filter 134 allows the central region 930c of the second core 930 to correspond to a motion marker region 911c among the multiple regions of the motion graph 910. The motion marker region 911c can be a region with marker information indicating that the region is a motion region. For example, the motion region detector 133 can align the second core 930 such that the central region 930c of the second core 930 overlaps with the motion marker region 911c among the multiple regions of the motion graph 910.

[0155] The second core 930 may include a peripheral region and a central region 930c arranged according to a predetermined configuration. In one embodiment, the peripheral region included in the second core 930 may include the region that is closest to the central region 930c in the upward, downward, leftward, and rightward directions. That is, the second core 930 may include multiple regions arranged in a cross configuration. However, this is only one embodiment, and the arrangement and number of peripheral regions may be modified and implemented differently.

[0156] Based on the value of the flag information included in the peripheral region of the motion flag region 911c, which corresponds to the peripheral region of the central region 930c of the second core 930, among the multiple regions of the motion graph 910, the noise filter 134 can maintain the flag information included in the motion flag region 911c or change the flag information included in the motion flag region 911c to flag information indicating that the flag region 911c is a normal region.

[0157] According to the embodiment, when the number of first values ​​of the flag information indicating that the outer region of the motion marker region 911c is a motion region in the outer region corresponding to the outer region of the central region 930c of the second core 930 among the plurality of regions of the motion map 910 exceeds a reference number, the noise filter 134 can maintain the flag information included in the motion marker region 911c. The flag information included in the motion marker region 911c can be maintained at the first value.

[0158] According to the embodiment, when the number of first values ​​of flag information indicating that the outer region of motion marker region 911c is a motion region included in the outer region of motion marker region 911c, which corresponds to the outer region of the central region 930c of the second core 930, among the plurality of regions of motion map 910, is equal to or less than a reference number, noise filter 134 can change the flag information included in motion marker region 911c to flag information indicating that motion marker region 911c is a normal region. The flag information included in motion marker region 911c can be changed from a first value to a second value.

[0159] Reference Figure 9B For example, the base number can be 1. For example, referencing Figure 9B In (1) to (3), when the number of first values ​​of the flag information included in the peripheral region corresponding to the peripheral region of the second core 931, 932 or 933 of the motion graph 910 exceeds the reference number 1, the first values ​​of the flag information included in the motion flag region corresponding to the central region of the second core 941, 942 or 943 can remain equal. In another example, referring to Figure 9B In (4) and (5), when the number of first values ​​of the flag information included in the region corresponding to the peripheral region of the second core 934 or 935 of the motion graph 910 is equal to or less than the reference number 1, the first value of the flag information included in the motion flag region corresponding to the central region of the second core 944 or 945 can be changed to a second value.

[0160] However, this is only one implementation method, and the base number can be modified and specifically implemented as 2, 3, etc. Furthermore, the base number can be determined based on the number of regions included in the second core 930.

[0161] Reference Figure 9C According to the implementation, when the central region 930c of the second core 930 is aligned with the region located at the edge of the motion map 910, there may be cases where some peripheral regions of the second core 930 do not overlap with any of the multiple regions of the motion map 910. The marking information included in the region of the motion map 910 corresponding to another peripheral region of the second core 930 can be used.

[0162] In one implementation, the noise filter 134 can align the central region 930c of the second core 930 with the motion marker region 913c of the motion map 910. When any of the peripheral regions of the second core 930 does not correspond to one of the multiple regions of the motion map 910, the noise filter 134 can select a peripheral region of the second core 930 located in the opposite direction to that peripheral region. For example, when the peripheral region of the second core 930 outside the motion map 910 is a left peripheral region located to the left of the central region 930c of the second core 930, the noise filter 134 can select a right peripheral region located to the right of the central region 930c of the second core 930.

[0163] The noise filter 134 can acquire the flag information included in the right peripheral region 913r of motion map 910 corresponding to the right peripheral region of the second core 930 as the flag information on the left peripheral region 913l of motion map 910 corresponding to the left peripheral region of the second core 930. That is, when the left peripheral region 913l in the left direction relative to the motion flag region 913c corresponding to the central region 930c of the second core 930 does not exist, the noise filter 134 can copy the flag information of the right peripheral region 913r in the right direction relative to the motion flag region 913c as the flag information of the left peripheral region 913l.

[0164] Subsequently, based on the value of the flag information included in the peripheral region corresponding to the peripheral region of the central region 930c of the second core 930 in the motion graph 910, the noise filter 134 may maintain the flag information included in the motion flag region 911c or change the flag information included in the motion flag region 911c to flag information indicating that the motion flag region 911 is a normal region.

[0165] Figures 10A to 10C This is a diagram illustrating the operation of a pixel corrector according to an embodiment of the present disclosure.

[0166] Reference Figure 10A and Figure 10B According to embodiments of the present disclosure, the pixel corrector 125 can use the image 1010 and the motion map 1020 to acquire the restored image 1030.

[0167] Image 1010 can be an image in a state with a short exposure pixel value sPV or an image with a corrected pixel value corrected from the short exposure pixel value sPV. Motion map 1020 can be a motion map in a state with empty regions where no marker information exists, a state with no empty regions, or a state with noise-removed regions. Hereinafter, it is assumed and described that image 1010 is an image in a state with a short exposure pixel value sPV, and motion map 1020 is a motion map in a state where marker information is filled in empty regions (i.e., a state with no empty regions). Furthermore, Figure 10A The motion graph 1020 shown in (2) represents a state with corresponding marker information in units of pixel values. However, this is only one implementation method, and the motion graph 1020 can be implemented in the form of marker information in units of regions. The restored image 1030 may include normal pixel values ​​and restored pixel values. The restored image 1030 may be an image in which short exposure pixel values ​​in the image are replaced with restored pixel values.

[0168] Specifically, pixel corrector 135 may acquire recovered pixel values ​​iPV or cPV corrected from short-exposure pixel values ​​sPV included in image 1010 based on marker information included in motion graph 1020. For example, pixel corrector 135 may acquire recovered pixel values ​​iPV or cPV corrected from short-exposure pixel values ​​sPV according to a first or second method based on marker information having the same location as the short-exposure pixel value sPV. Alternatively, pixel corrector 135 may acquire recovered pixel values ​​iPV or cPV corrected from short-exposure pixel values ​​sPV included in a selected area according to a first or second method based on marker information regarding a region that is the same as a selected area having short-exposure pixel values ​​sPV.

[0169] Image 1010 may include multiple regions. For example, each region may include pixel values ​​arranged in a 2×2 pattern. Each region may include pixel values ​​of the same color.

[0170] The pixel corrector 135 can select a region with a short exposure pixel value sPV as the selected region. For example, a region with a first short exposure pixel value 1011C can be selected as the first selected region, and a region with a second short exposure pixel value 1013C can be selected as the second selected region. For example, the first selected region can be the region with pixel values ​​arranged at positions (3,3) to (4,4), and the second selected region can be the region with pixel values ​​arranged at positions (5,5) to (6,6).

[0171] In an implementation, when the flag information 1021C about the first selected region has a first value indicating the motion region, the pixel corrector 135 can use the peripheral pixel values ​​of the first short exposure pixel value 1011C to obtain a first recovered pixel value 1031C corrected from the first short exposure pixel value 1011C included in the first selected region.

[0172] In a particular implementation, when the flag information 1021C regarding the first selected region has a first value indicating the motion region, the pixel corrector 135 may select a normal pixel value from the normal pixel values ​​PV that has the same color as the first short exposure pixel value 1011C. For example, when the first short exposure pixel value 1011C is green, a normal pixel with green can be selected.

[0173] Additionally, the pixel corrector 135 can obtain a value as the first restored pixel value 1031C based on the distance between the peripheral normal pixel value that is closest to the first short exposure pixel value 1011C in multiple directions and the weight of the peripheral normal pixel value.

[0174] For example, refer to Figure 10A and Figure 10B The peripheral normal pixel values ​​may include the upper pixel value 1031T in the upward direction, the lower pixel value 1031B in the downward direction, the left pixel value 1031L in the left direction, and the right pixel value 1031R in the right direction with respect to the first short exposure pixel value 1011C.

[0175] Pixel corrector 135 can determine a first weight value for the upper pixel value 1031T based on the distance between the upper pixel value 1031T and the first short-exposure pixel value 1011C. Pixel corrector 135 can determine a second weight value for the lower pixel value 1031B based on the distance between the lower pixel value 1031B and the first short-exposure pixel value 1011C. Pixel corrector 135 can determine a third weight value for the left pixel value 1031L based on the distance between the left pixel value 1031L and the first short-exposure pixel value 1011C. Pixel corrector 135 can determine a fourth weight value for the right pixel value 1031R based on the distance between the right pixel value 1031R and the first short-exposure pixel value 1011C. Each weight can be determined to be a value that decreases as the distance increases. Additionally, as... Figure 10B As shown in (3), the pixel corrector 135 can obtain the sum of the values ​​obtained by multiplying the upper pixel value 1031T by a first weight value, the lower pixel value 1031B by a second weight value, the left pixel value 1031L by a third weight value, and the right pixel value 1031R by a fourth weight value as the first restored pixel value 1031C.

[0176] As mentioned above, since the motion region is a blurred region due to the movement of the object, interpolating the short exposure pixel values ​​included in the motion region using the peripheral pixel values ​​of the short exposure pixel values ​​will not affect the degradation of image quality.

[0177] In addition, such as Figure 10B As shown in (2), when the flag information 1023C regarding the second selected region has a second value indicating a normal region, the pixel corrector 135 can use the value obtained by multiplying the second short-exposure pixel value 1013C by the exposure ratio to obtain a second recovered pixel value 1033C corrected from the second short-exposure pixel value 1013 included in the second selected region. The exposure ratio can be the ratio of the first exposure time ET_N for sensing the normal pixel value to the second exposure time ET_S for sensing the short-exposure pixel value.

[0178] Reference Figure 10A and Figure 10C According to the implementation, when the flag information regarding the third selected region with the third short exposure pixel value indicates that the third selected region is a moving region, and the third selected region is a region located at the edge of image 1010, there may be a situation where any normal pixel value is absent in a particular direction among multiple directions regarding the third short exposure pixel value. A peripheral normal pixel value in the opposite direction to the specific direction can be used as a peripheral normal pixel value in the specific direction. The multiple directions may include an upward direction, a downward direction, a leftward direction, and a rightward direction.

[0179] In one implementation, when the flag information 1021C regarding the third selected region has a first value indicating the motion region, the pixel corrector 135 can select a normal pixel value from a plurality of pixel values ​​that has the same color as the third short-exposure pixel value included in the third selected region. For example, the third short-exposure pixel value and the selected normal pixel value could be green pixel values.

[0180] Furthermore, when any first pixel value among the selected normal pixel values ​​that is closest to the third short exposure pixel value in any of the multiple directions does not exist, the pixel corrector 135 may acquire the second pixel value that is closest to the third short exposure pixel value in the opposite direction of that direction as the first pixel value.

[0181] For example, suppose the third selected region includes pixel values ​​located at (7,7) to (8,8), and the short-exposure pixel value included in the third selected region is located at (8,8). Peripheral pixel values ​​having the same color as the short-exposure pixel value may include the upper pixel value 1043T located at (7,8) in the upward direction and the left pixel value 1043L located at (8,7) in the left direction. Additionally, the lower pixel value 1043B located in the downward direction and the right pixel value 1043R located in the right direction of the short-exposure pixel value may not exist. Pixel corrector 135 can acquire the upper pixel value 1043T in the opposite direction to the direction of the lower pixel value 1043B in the third selected region as the lower pixel value 1043B. Additionally, pixel corrector 135 can acquire the left pixel value 1043L in the opposite direction to the direction of the right pixel value 1043R in the third selected region as the right pixel value 1043R.

[0182] In addition, such as Figure 10C As shown in (2), the pixel corrector 135 can determine each weight value based on the distance between each of the upper pixel value 1043T, lower pixel value 1043B, left pixel value 1043L and right pixel value 1043R and the third short exposure pixel value, and obtain the value of the sum of the weights of the weight values ​​corresponding to the upper pixel value 1043T, lower pixel value 1043B, left pixel value 1043L and right pixel value 1043R respectively as the third restored pixel value 1043C.

[0183] Moving and normal regions can be detected simultaneously using short-exposure pixel values, which can then be used to distinguish them from each other. Furthermore, the short-exposure pixel values ​​can be recovered to accurate pixel values ​​using another method, depending on whether the region with the short-exposure pixel value is a moving or normal region. Therefore, the image can be recovered without image quality degradation due to pixel values ​​differing between exposure times.

[0184] Figure 11 This is a diagram illustrating a method for compressing an image using marking information according to an embodiment of the present disclosure.

[0185] Reference Figure 11According to embodiments of this disclosure, the image sensor 110 can sequentially acquire a plurality of images 1110, 1120, and 1130 over time. Each of the plurality of images 1110, 1120, and 1130 may include a region 1111, 1121, or 1131 representing an object and a region 1115, 1125, or 1135 representing the background. The object may be in a state where the object is moving in the right direction during the time period acquired by each of the images 1110, 1120, and 1130. In addition, each of the images 1110, 1120, and 1130 may include normal pixel values ​​sensed during a first exposure time and short-exposure pixel values ​​sensed during a second exposure time.

[0186] Processor 130 can use short-exposure pixel values ​​and normal pixel values ​​included in each of images 1110, 1120, and 1130 to generate a motion map of each of images 1110, 1120, and 1130. The motion map may include flag information indicating whether each of a plurality of regions of images 1110, 1120, and 1130 is a moving region or a normal region.

[0187] Processor 130 can generate video data by compressing images 1110, 1120, and 1130 using marker information included in the motion map for each of images 1110, 1120, and 1130. For example, processor 130 can determine whether each of the multiple regions of images 1110, 1120, and 1130 is a moving region or a normal region based on the value of the marker information for each of the multiple regions. Additionally, processor 130 can generate video data by removing pixel values ​​from images 1110, 1120, and 1130 other than representative pixel values ​​included in the normal regions. As described above, accurate detection of moving and normal regions improves image quality while simultaneously improving data compression ratio.

[0188] Figure 12 This is a diagram illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0189] The operation method of the electronic device 100 may include: step S1210, acquiring an image including normal pixel values ​​sensed during a first exposure time and short exposure pixel values ​​sensed during a second exposure time shorter than the first exposure time; step S1220, using the exposure ratio of the first exposure time and the second exposure time and using the short exposure pixel values ​​and normal pixel values ​​included in a selected region among a plurality of regions of the image to acquire flag information indicating that the selected region is one of a motion region and a normal region; and step S1230, outputting a restored image including restored pixel values ​​corrected from the short exposure pixel values ​​included in the selected region based on the flag information.

[0190] Specifically, an image can be acquired, which includes normal pixel values ​​and short-exposure pixel values ​​(S1210). Normal pixel values ​​can be pixel values ​​sensed during the first exposure time. Short-exposure pixel values ​​can be pixel values ​​sensed during a second exposure time shorter than the first exposure time.

[0191] Additionally, the exposure ratio of the first exposure time and the second exposure time, as well as the short exposure pixel value and normal pixel value in the selected area included in the multiple areas of the image, indicate that the selected area is one of the moving area and the normal area (S1220).

[0192] Each of the multiple regions in an image can be an area of ​​the same size. Each of the multiple regions can include the same number of pixel values. The selected region can be one of the multiple regions that includes at least one short-exposure pixel value. A motion region can represent an area where the pixel value changes significantly over time due to the movement of an object. A normal region can represent an area where the pixel value changes less over time because there is no movement of an object.

[0193] In one implementation, a corrected image may be obtained, comprising corrected pixel values ​​and normal pixel values ​​corrected from short-exposure pixel values ​​using an exposure ratio. The corrected image may be an image with some pixel values ​​changed. Alternatively, the corrected image may be an image generated separately from the original image. Multiple regions of the corrected image and multiple regions of the original image may correspond to each other.

[0194] Additionally, a motion ratio can be obtained, which represents the ratio of the average value of normal pixels in the region corresponding to the selected region among multiple regions of the calibrated image to the average value of calibrated pixels in the region. Furthermore, flag information can be obtained based on the motion ratio, indicating whether the selected region is a moving region or a normal region.

[0195] In this implementation, when the difference between the motion ratio and the first reference value exceeds a second reference value, flag information can be acquired, including a first value indicating that the selected area is a motion area. Additionally, when the difference between the motion ratio and the first reference value is equal to or less than the second reference value, flag information can be acquired, including a second value indicating that the selected area is a normal area. For example, the first reference value may be 1, and the second reference value may be 0.3. However, this is only one implementation method, and the first and second reference values ​​can be modified and implemented with various values.

[0196] Additionally, a restored image, including restored pixel values ​​corrected from the short exposure pixel values ​​included in the selected area, can be output based on the flag information (S1230).

[0197] In the implementation, when the flag information indicates that the selected area is a moving area, the recovered pixel value can be obtained using the short exposure pixel value according to the first method.

[0198] In this implementation, a normal pixel value that has the same color as the short-exposure pixel value included in the selected area can be selected from the normal pixel values. The recovered pixel value can be obtained as a sum of the distances between the short-exposure pixel value and the peripheral normal pixel value that is closest to the short-exposure pixel value in multiple directions, and the weights of the peripheral normal pixel values.

[0199] Furthermore, if the first peripheral normal pixel value, which is closest to the short exposure pixel value in any of the selected normal pixel values ​​in multiple directions, does not exist, the second peripheral normal pixel value, which is closest in the opposite direction of that direction, can be obtained as the first peripheral normal pixel value.

[0200] In this implementation, when the flag information indicates that the selected area is a normal area, the recovered pixel value can be obtained using the short exposure pixel value according to the second method. The recovered pixel value can be obtained by multiplying the short exposure pixel value included in the selected area by the exposure ratio.

[0201] In the implementation, the central region of the kernel may correspond to an empty region among multiple regions where no flag information exists, and flag information indicating whether the empty region is a moving region or a normal region may be obtained based on the number of flag information included in the region corresponding to the peripheral region of the central region of the kernel among multiple regions.

[0202] The central region of the kernel may correspond to a motion flag region among multiple regions, which includes flag information indicating that the region is a motion region. Depending on the number of regions corresponding to the peripheral region of the kernel among the multiple regions and the number of flag information of the motion region included in the motion flag region, the flag information included in the motion flag region may be maintained or changed to flag information indicating that the region among the multiple regions is a normal region.

[0203] Figure 13 This is a diagram illustrating an implementation example of an electronic device according to an embodiment of the present disclosure.

[0204] Reference Figure 13 The electronic device 10 can be implemented as a computing system 2000. The computing system 2000 may include an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input / output (I / O) device 2050, and a display device 2060. Although Figure 13As not shown, the computing system 2000 may also include ports that can communicate with storage devices 2030, memory devices 2040, I / O devices 2050, display devices 2060, etc., or with external devices.

[0205] Image sensor 2010 can acquire images comprising multiple pixels to which exposure values ​​are individually applied. Image sensor 2010 can be connected to processor 2020 via address bus, control bus, and data bus, or via a separate communication line, to perform communication.

[0206] Image sensor 2010 can be implemented using various types of packages. For example, at least some components of image sensor 2010 can be implemented using packages such as stacked package (PoP), ball grid array (BGA), chip-scale package (CSP), plastic leaded chip carrier (PLCC), plastic dual in-line package (PDIP), die in waffle package, die in wafer form, chip on board (COB), ceramic dual in-line package (CERDIP), plastic metric quad flat package (MQFP), thin quad flat package (TQFP), small outline integrated circuit (SOIC), shrink small outline package (SSOP), thin small outline package (TSOP), system-in-package (SIP), multi-chip package (MCP), wafer-level fabrication package (WFP), or wafer-level processing stacked package (WSP). In some embodiments, image sensor 2010 can be integrated with processor 2020 on a single chip, or image sensor 2010 and processor 2020 can be integrated on different chips.

[0207] The processor 2020 may include at least one of a central processing unit (CPU), an application processing unit (APU), a graphics processing unit (GPU), etc.

[0208] The processor 2020 can be connected to the storage device 2030, memory device 2040, and I / O device 2050 via the address bus, control bus, and data bus to perform communication. According to embodiments of this disclosure, the processor 2020 can also be connected to an expansion bus such as a peripheral component interconnect (PCI) bus.

[0209] Storage device 2030 can store data including images, motion graphics, etc. The data stored in storage device 2030 can be preserved not only when computing system 2000 is powered on, but also when computing system 2000 is not powered on. For example, storage device 2030 can be configured using at least one of all types of non-volatile memory devices such as flash memory devices, solid-state drives (SSDs), hard disk drives (HDDs), and optical discs.

[0210] Memory device 2040 can store data including images, motion graphics, etc. Memory device 2040 can temporarily store data to be processed by processor 2020 or temporarily store data already processed by processor 2020. Data stored in memory device 2040 can only be saved when computing system 2000 is powered on. Alternatively, data stored in memory device 2040 can be saved not only when computing system 2000 is powered on, but also when computing system 2000 is not powered on. For example, memory device 2040 can include volatile memory devices such as dynamic random access memory (DRAM) and static random access memory (SRAM), and non-volatile memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory.

[0211] The I / O device 2050 may include input devices and output devices. Input devices are devices capable of interactively inputting user commands and can be implemented as, for example, a keyboard, keypad, mouse, microphone, etc. Output devices are devices capable of outputting data and can be implemented as printers, speakers, etc.

[0212] Display device 2060 is a device for visually outputting images. For this purpose, display device 2060 can be implemented using various types of displays, such as liquid crystal displays (LCDs) that use a separate backlight unit (e.g., a light-emitting diode (LED) or the like) as a light source to control the molecular arrangement of liquid crystals, thereby adjusting the degree (brightness or intensity) of light transmitted through the liquid crystals from the backlight unit; and displays that use self-emissive elements (e.g., miniature LEDs with a size of 100 μm to 200 μm, micro-LEDs with a size of 100 μm or smaller, organic LEDs (OLEDs), quantum dot LEDs (QLEDs), etc.) as a light source. Display device 2060 can emit red, green, and blue light corresponding to the output image.

[0213] According to this disclosure, an electronic device capable of detecting a moving region included in an image and a method for operating the electronic device may be provided.

[0214] According to this disclosure, moving regions included in the current image can be detected without any previous images. Therefore, the memory space and power consumption required to store previous images can be reduced. Furthermore, noise can be reduced, resulting in more accurate detection of moving regions.

[0215] Although this disclosure has been shown and described with reference to certain exemplary embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents. Therefore, the scope of this disclosure should not be limited to the exemplary embodiments described above, but should be determined not only by the appended claims but also by their equivalents.

[0216] In the above embodiments, all steps may be performed selectively, or some steps may be omitted. In various embodiments, the steps may not be performed in the described order and may be rearranged. The embodiments disclosed in this specification and accompanying drawings are merely examples for illustrative purposes and are not intended to limit the scope of this disclosure. That is, it will be apparent to those skilled in the art that various modifications can be made based on the technical scope of this disclosure.

[0217] Furthermore, exemplary embodiments of this disclosure are described in the accompanying drawings and specification. Although specific terminology is used herein, these are merely illustrative of embodiments of this disclosure. Therefore, this disclosure is not limited to the embodiments described above, and many variations can be made within the spirit and scope of this disclosure. It will be apparent to those skilled in the art that various modifications can be made based on the technical scope of this disclosure in addition to the embodiments disclosed herein.

[0218] Cross-references to related applications

[0219] This application claims priority to Korean Patent Application No. 10-2021-0091844 filed on July 13, 2021 and Korean Patent Application No. 10-2021-0126344 filed on September 24, 2021, the full disclosure of which is incorporated herein by reference.

Claims

1. An electronic device comprising: An image sensor that acquires an image including normal pixel values ​​sensed during a first exposure time and short-exposure pixel values ​​sensed during a second exposure time shorter than the first exposure time; as well as The processor, the processor: The corrected pixel value is obtained by multiplying the short exposure pixel value included in a selected region among multiple regions of the image by the exposure ratio of the first exposure time to the second exposure time. Based on the normal pixel values ​​and the corrected pixel values ​​included in the selected region, flag information indicating that the selected region is either a motion region or a normal region is obtained; and Based on the flag information, output a restored image including restored pixel values ​​corrected from the short exposure pixel values ​​included in the selected region.

2. The electronic device according to claim 1, wherein, The processor includes an exposure corrector that acquires a corrected image comprising corrected pixel values ​​obtained by multiplying the short exposure pixel values ​​by the exposure ratio and the normal pixel values.

3. The electronic device according to claim 2, wherein, The processor includes: A motion ratio calculator that acquires a motion ratio representing the ratio of the average of the normal pixel values ​​included in the selected region to the average of the corrected pixel values; and A motion region detector that acquires the marker information based on the motion ratio.

4. The electronic device according to claim 3, wherein, The motion region detector: When the difference between the motion ratio and the first reference value exceeds the second reference value, the flag information having the first value indicating that the selected region is the motion region is acquired; and When the size value is equal to or less than the second reference value, the flag information having a second value indicating that the selected region is the normal region is obtained.

5. The electronic device according to claim 1, wherein, The processor includes a pixel corrector, which: When the flag information indicates that the selected area is the motion area, a normal pixel value of the same color as the short-exposure pixel value included in the selected area is selected from the normal pixel values, and The recovered pixel value is obtained based on the distance between each peripheral normal pixel value that is closest to the short exposure pixel value in multiple directions among the selected normal pixel values ​​and the short exposure pixel value, and the weight of the peripheral normal pixel value.

6. The electronic device according to claim 5, wherein, The multiple directions include the up direction, down direction, left direction, and right direction, and Wherein, if the first pixel value among the selected normal pixel values ​​that is closest to the short exposure pixel value in any of the multiple directions does not exist, the pixel corrector obtains the second pixel value that is closest to the short exposure pixel value in the opposite direction of the selected normal pixel value as the first pixel value.

7. The electronic device according to claim 5, wherein, When the flag information indicates that the selected area is the normal area, the pixel corrector obtains the corrected pixel value as the restored pixel value.

8. The electronic device according to claim 1, wherein, The processor includes a motion region detector, which: Allowing the central region among the plurality of regions included in the first kernel to overlap with empty regions lacking the flag information, and Based on the number of first values ​​of the flag information included in the peripheral region of the empty region corresponding to the peripheral region of the central region among the plurality of regions included in the first kernel, flag information indicating that the empty region is one of the motion region and the normal region is obtained.

9. The electronic device according to claim 8, wherein, The processor includes a noise filter, which: Allowing the central region among the plurality of regions included in the second kernel to overlap with a motion marker region having marker information indicating that the selected region is the motion region, and Based on the number of first values ​​of the flag information included in the peripheral region of the motion flag region corresponding to the peripheral region of the central region among the plurality of regions included in the second kernel, the flag information included in the motion flag region is maintained or the flag information included in the motion flag region is changed to flag information indicating that the selected region is the normal region.

10. The electronic device according to claim 8, wherein, The peripheral region included in the first kernel or the second kernel includes the regions located in the upward, downward, leftward, and rightward directions of the central region of the first kernel or the second kernel.

11. A method of operating an electronic device, the method comprising the following steps: An image is acquired, which includes normal pixel values ​​sensed during a first exposure time and short-exposure pixel values ​​sensed during a second exposure time shorter than the first exposure time; The corrected pixel value is obtained by multiplying the short exposure pixel value included in a selected region among multiple regions of the image by the exposure ratio of the first exposure time to the second exposure time. Based on the normal pixel values ​​and the corrected pixel values ​​included in the selected region, obtain flag information indicating that the selected region is either a motion region or a normal region; and Based on the flag information, output a restored image including restored pixel values ​​corrected from the short exposure pixel values ​​included in the selected region.

12. The method according to claim 11, wherein, The steps for obtaining the flag information include: Obtain a corrected image comprising the corrected pixel values ​​obtained by multiplying the short-exposure pixel values ​​by the exposure ratio and the normal pixel values; Obtain the motion ratio, which indicates the ratio of the average of the normal pixel values ​​included in the selected region to the average of the corrected pixel values; and The flag information is obtained based on the said motion ratio.

13. The method according to claim 12, wherein, The steps for obtaining the flag information include: When the difference between the motion ratio and the first reference value exceeds the second reference value, acquire the flag information having the first value indicating that the selected region is the motion region; and When the size value is equal to or less than the second reference value, the flag information having a second value indicating that the selected region is the normal region is obtained.

14. The method according to claim 11, wherein, The steps for outputting the restored image include: When the flag information indicates that the selected area is the motion area, the recovered pixel value is obtained using the short-exposure pixel value according to the first method; and When the flag information indicates that the selected area is the normal area, the recovered pixel value is obtained by using the short exposure pixel value according to the second method.

15. The method according to claim 14, wherein, The step of obtaining the recovered pixel value using the short exposure pixel value according to the first method includes: Select the normal pixel value from the normal pixel values ​​that has the same color as the short-exposure pixel value included in the selected area; and The recovered pixel value is obtained based on the distance between each peripheral normal pixel value that is closest to the short exposure pixel value in multiple directions among the selected normal pixel values ​​and the short exposure pixel value, and the weight of the peripheral normal pixel value.

16. The method of claim 14, further comprising the step of: If the first pixel value, which is closest to the short exposure pixel value in any of the multiple directions among the selected normal pixel values, does not exist, the second pixel value, which is closest to the short exposure pixel value in the opposite direction of the selected direction, is obtained as the first pixel value.

17. The method according to claim 14, wherein, In the step of obtaining the recovered pixel value using the short exposure pixel value according to the second method, the corrected pixel value obtained by multiplying the short exposure pixel value included in the selected area by the exposure ratio is obtained as the recovered pixel value.

18. The method of claim 11, further comprising the step of: Allow the central region among the plurality of regions included in the kernel to correspond to an empty region where the flag information does not exist, and obtain flag information indicating that the empty region is one of the moving region and the normal region based on the value of the flag information included in the region among the plurality of regions that corresponds to the peripheral region of the central region of the kernel; as well as The central region is allowed to correspond to a motion marker region among the plurality of regions, which has marker information indicating that the selected region is the motion region. Based on the value of the marker information indicating that the selected region is the motion region included in the region corresponding to the peripheral region of the motion marker region among the plurality of regions, the marker information included in the motion marker region is maintained or the marker information included in the motion marker region is changed to marker information indicating that the selected region is the normal region.

19. An electronic device comprising: A communication interface that receives images from an external device, the images including normal pixel values ​​and short exposure pixel values ​​sensed during different exposure times; as well as The processor, the processor: A first corrected pixel value is obtained by multiplying the short-exposure pixel value included in a selected region among a plurality of regions of the image by the exposure ratio of the different exposure times; Based on the first corrected pixel value and normal pixel value included in the selected region among the plurality of regions of the image, it is determined whether the selected region is a moving region or a normal region; When the selected area is the normal area, the short exposure pixel value is corrected to the first corrected pixel value; and When the selected region is the motion region, the short exposure pixel value is corrected to a second corrected pixel value, which is obtained based on the weighted sum of the surrounding pixel values ​​of the short exposure pixel value.

20. The electronic device according to claim 19, wherein, The number of short-exposure pixel values ​​is less than the number of normal pixel values, and The color of the short-exposure pixel value is green.

Images