Electronic device and method of operating the same

Abstract

The present application discloses an electronic device and a method for operating the same. The electronic device according to the present technology may include: an image sensor including a plurality of pixel groups corresponding to a plurality of exposure values, respectively; and a controller configured to select a reference scale image having a minimum exposure value and a target scale image having an exposure value different from the minimum exposure value among a plurality of scale images obtained based on the plurality of pixel groups, and output an output image obtained by using a motion map indicating a position change of an object commonly included in the reference scale image and the target scale image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Application No. 10-2021-0115123, filed on August 30, 2021, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an electronic device, and more particularly, to an electronic device including an image sensor and a method of operating the electronic device. Background Art

[0004] Recently, with the development of the computer industry and the communication industry, the demand for image sensors in various electronic devices such as smartphones, digital cameras, gaming devices, the Internet of Things, robots, security cameras, medical cameras, and autonomous vehicles has continued to increase.

[0005] In particular, various technologies for synthesizing a plurality of original images with different exposure times into one image are being studied recently to improve image quality such as dynamic range (DR). However, when the object or the image sensor itself moves, the position of the object included in each original image appears differently depending on the exposure time. Therefore, there is a problem of a so-called ghosting phenomenon in which the same object is repeatedly displayed at different positions in one synthesized image or the movement process of the object is displayed. A technology for solving this ghosting phenomenon is needed. Summary of the invention

[0006] According to an embodiment of the present disclosure, an electronic device may include: an image sensor, which includes multiple pixel groups corresponding to multiple exposure values ​​respectively; and a controller, which is configured to select a reference scale image with a minimum exposure value and a target scale image with an exposure value different from the minimum exposure value among multiple scale images obtained based on the multiple pixel groups, and output an output image obtained using a motion map indicating position changes of objects commonly included in the reference scale image and the target scale image.

[0007] According to an embodiment of the present disclosure, a method for operating an electronic device may include: obtaining multiple images from multiple pixel groups sensing pixel values ​​during different exposure times; obtaining multiple scale images obtained by correcting the multiple images using the exposure time of each of the multiple images; generating a motion map indicating position changes of an object based on a reference scale image having a minimum exposure time among the multiple scale images and a selected scale image having an exposure time different from the minimum exposure time; and outputting an output image synthesized by using the motion map as a weight for the reference scale image and the selected scale image. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0009] Figure 2A is a diagram showing an image sensor according to an embodiment of the present disclosure.

[0010] Figure 2B is a diagram showing a controller according to an embodiment of the present disclosure.

[0011] Figure 2C is a diagram showing a detailed configuration of a scaler, a preprocessor, and a motion map generator according to an embodiment of the present disclosure.

[0012] Figure 3 is a diagram illustrating a method of operating an electronic device according to an embodiment of the present disclosure.

[0013] FIG. 4A to FIG. 4D is a diagram showing an image according to an embodiment of the present disclosure.

[0014] Figure 5A and Figure 5B is a diagram showing a scale image according to an embodiment of the present disclosure.

[0015] Fig. 6A is a diagram showing a scaled image converted into grayscale according to an embodiment of the present disclosure.

[0016] Figure 6B is a diagram showing blocks according to an embodiment of the present disclosure.

[0017] FIG. 7A to FIG. 7C is a diagram illustrating a motion graph according to an embodiment of the present disclosure.

[0018] FIG. 8A to FIG. 8C is a diagram showing an output image according to an embodiment of the present disclosure.

[0019] Fig. 9 is a diagram showing an implementation example of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The specific structural or functional description of the embodiments of the concepts disclosed in this specification or application is only used to describe the embodiments of the concepts according to the present disclosure. The embodiments of the concepts according to the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments described in this specification or application.

[0021] Embodiments of the present disclosure provide an electronic device and an operating method thereof for outputting an image with improved image quality while preventing a ghost phenomenon according to a motion of an object from occurring.

[0022] The present technology can provide an electronic device and an operating method thereof for outputting an image with improved image quality while preventing a ghost phenomenon from occurring. According to the present technology, the occurrence of a ghost phenomenon in an image obtained by synthesizing images having different exposure times can be minimized.

[0023] Figure 1 FIG. 1 is a diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0024] See also Figure 1 According to the embodiment, the electronic device 100 can obtain an image. In addition, the electronic device 100 can store, display or output an output image obtained by processing the image to an external device. According to the embodiment, the electronic device 100 can output the output image to the host 200 according to the request of the host 200.

[0025] In one embodiment, the electronic device 100 may be implemented in the form of a package module, a component, etc. In this case, the electronic device 100 may be installed on a host 200. The host 200 may be implemented as various electronic devices. For example, the host 200 may be implemented as a digital camera, a mobile device, a smart phone, a personal computer (PC), a tablet PC, a portable computer, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a portable multimedia player (PMP), a wearable device, a black box, a robot, an autonomous driving vehicle, etc.

[0026] In another embodiment, the electronic device 100 may be implemented as an electronic device separated from the host 200. For example, the electronic device 100 may be an imaging device, a digital camera, a video camera, a closed-circuit television (CCTV), a webcam, a security camera, an industrial visual camera, a mobile device, a smart phone, a personal computer (PC), a tablet PC, a portable computer, a PDA, an EDA, a PMP, a wearable device, a black box, a robot, an autonomous vehicle, a vehicle-mounted visual camera, a set-top box, a game terminal, an electronic dictionary, an e-book reader, a desktop computer, a server, an MP3 player, an intelligent medical device, a television, a digital video disc (DVD) player, a stereo, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, an air purifier, a smart mirror, a smart window, an electronic key, an electronic photo frame, a digital bulletin board, a security control panel, etc. Here, the wearable device may be a smart watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, a head-mounted device (HMD), a skin pad, a tattoo, a bio-implanted circuit, etc.

[0027] The electronic device 100 may include an image sensor 110 and a controller 120 .

[0028] The image sensor 110 may obtain a raw image. Specifically, when a command requesting to obtain an image is received from the host 200 or the controller 120, the image sensor 110 may obtain a raw image. To this end, the image sensor 110 may be implemented as a charge coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, etc. In addition, the image sensor 110 may send the raw image to the controller 120.

[0029] The image sensor 110 may include a plurality of pixels. The plurality of pixels may be arranged in a matrix form. The plurality of pixels may be grouped into a plurality of pixel groups according to exposure values. That is, the image sensor 110 may include a plurality of pixel groups. The plurality of pixel groups may correspond to a plurality of exposure values, respectively. That is, the plurality of pixel groups may have different exposure values. Here, the exposure value may represent the exposure time of the pixel sensing the pixel value. For example, the first pixel group may include a pixel having a first exposure value among the plurality of pixels. The second pixel group may include a pixel having a second exposure value among the plurality of pixels.

[0030] The image sensor 110 may obtain pixel data through a plurality of pixels. The pixel data may include at least one of a position, a color, and an exposure value of the pixel.

[0031] The position of a pixel may represent the position where the corresponding pixel is arranged among a plurality of pixels. A plurality of pixels may be distinguished according to the position of the pixel. A pixel at a corresponding position may be selected by an address indicating the position of the pixel.

[0032] The color of a pixel may represent the color of light for the corresponding pixel. For example, the color of each of the plurality of pixels may be one of red, blue, and green.

[0033] The exposure value may represent an exposure time. The exposure time may represent a period of time during which a pixel is exposed to light, or a period of time during which a pixel senses light to obtain a pixel value. Generally, when the exposure time decreases, the pixel value may decrease due to a decrease in the amount of incident light during the exposure time, and when the exposure time increases, the pixel value may increase due to an increase in the amount of incident light during the exposure time. According to an embodiment, a separate exposure value may be set for each pixel. In this case, the pixel value of each pixel may be a value obtained during the exposure time corresponding to the exposure value set for each pixel.

[0034] The controller 120 may control the overall operation of the electronic device 100. For example, the controller 120 may control the image sensor 110 to perform an operation of adjusting an exposure time or obtaining an image.

[0035] In one embodiment, the controller 120 may obtain a plurality of scale images based on a plurality of pixel groups included in the image sensor 110. For example, the controller 120 may generate a first scale image based on pixel values ​​obtained by a first pixel group included in the image sensor 110, and generate a second scale image based on pixel values ​​obtained by a second pixel group included in the image sensor 110. Here, the first pixel group may correspond to a first exposure value, and the second pixel group may correspond to a second exposure value. For example, the first pixel group may include pixels having a first exposure value set among a plurality of pixels included in the image sensor 110. The second pixel group may include pixels having a second exposure value set among a plurality of pixels included in the image sensor 110. Here, the first exposure value and the second exposure value may have different values. For example, the second exposure value may be greater than the first exposure value.

[0036] Specifically, the controller 120 may obtain an original image from the image sensor 110. The original image may include a plurality of pixel values ​​obtained by a plurality of pixels. The controller 120 may obtain a plurality of images from one original image. Each of the plurality of images may be an image with a different exposure value. For example, the first image may include pixel values ​​obtained during a first exposure time indicated by a first exposure value. The second image may include pixel values ​​obtained during a second exposure time indicated by a second exposure value. Here, assuming that the second exposure value is greater than the first exposure value, the second exposure time may include the first exposure time. That is, the first exposure time may overlap with a portion of the second exposure time.

[0037] Furthermore, the controller 120 may obtain a plurality of scale images obtained by correcting the plurality of images using an exposure value or an exposure time of each of the plurality of images.

[0038] In one embodiment, the controller 120 may generate a motion map based on the position of a first block indicating an object in a first block group corresponding to a first scale image and the position of a second block indicating an image in a second block group corresponding to a second scale image. Here, each block in the first block group may correspond to an area of ​​the first scale image, and each block in the second block group may correspond to an area of ​​the second scale image. The motion map may represent the movement of an object or the change in position of an object. The motion map may include a plurality of motion values. The motion value may correspond to a pixel value. For example, the motion value and the pixel value may have a mutually corresponding relationship according to the position.

[0039] To this end, the controller 120 may select a scale image with a minimum exposure value (minimum exposure value) among the first scale image and the second scale image as a reference scale image, and may select a scale image with a different exposure value as a target scale image. Here, the target scale image may be referred to as a selected scale image.

[0040] In addition, the above embodiment is only one embodiment, and the number of scale images may be three or more. In this case, the controller 120 may generate a motion map indicating the motion of the object based on a reference scale image having the smallest exposure value (i.e., the minimum exposure value) among the multiple scale images and a scale image having a selected exposure value.

[0041] In one embodiment, the controller 120 may output an output image using a motion map. In one embodiment, the controller 120 may output an output image generated according to a weight operation by using a motion map as a weight for the first scale image and the second scale image. In addition, this is only one embodiment, and the number of scale images may be three or more. In this case, the controller 120 may generate an output image according to a weight operation by using a motion map as a weight for a reference scale image and a scale image with a selected exposure value. The controller 120 may output the generated output image.

[0042] According to the present disclosure, an electronic device and an operating method thereof for outputting an image with improved image quality while preventing a ghost phenomenon according to a motion of an object from occurring may be provided. Hereinafter, the present disclosure will be described in more detail with reference to the accompanying drawings.

[0043] Figure 2A is a diagram showing an image sensor according to an embodiment of the present disclosure.

[0044] See also Figure 2A The image sensor 110 may include: an optical lens LS, a pixel array 111, a row decoder 112, a timing generator 113, a signal converter (or signal transformer) 114 and an output buffer 115.

[0045] The optical lens LS may refract light reflected from the object. The light refracted by the optical lens LS may travel to the pixel array 111. That is, the optical lens LS may refract the incident light to each pixel of the pixel array 111. The optical lens LS may be one lens, or may be an assembly of a plurality of lenses arranged in the path of travel of the light. In addition, the optical lens LS may include a microlens assembly. The object may include at least one of various elements existing outside the image sensor 110, such as an object, an animal, a person, and a background.

[0046] Pixel array 111 may generate an electrical signal indicative of the intensity or amount of light exposed.

[0047] The pixel array 111 may include a plurality of pixels. The plurality of pixels may be arranged in a row direction and a column direction. The plurality of pixels included in the pixel array 111 may correspond to the plurality of pixels included in the original image raw_IMG. The pixels of the pixel array 111 may be arranged in a physical area, and the pixels of the original image raw_IMG may be arranged in a digital area. The pixels of the pixel array 111 and the pixels of the original image raw_IMG may have a corresponding relationship between the same arrangement positions.

[0048] For example, a pixel (x, y) of the raw image raw_IMG may correspond to a pixel (x, y) of the pixel array 111. Here, x and y are natural numbers. In addition, a pixel value of a pixel (x, y) of the raw image raw_IMG may be proportional to the amount of light to which the pixel (x, y) of the pixel array 111 is exposed. The color of the pixel (x, y) of the raw image raw_IMG may be the same color as the color of the color filter included in the pixel (x, y) of the pixel array 111.

[0049] The pixels of the pixel array 111 may include a color filter and a sensing circuit. The color filter may be disposed on the sensing circuit. During the exposure time, light may pass through the color filter and reach the sensing circuit. Depending on the color of the color filter, the pixel may be referred to as a red pixel, a blue pixel, a green pixel, etc.

[0050] The sensing circuit may include a light sensing element. The light sensing element may utilize a photoelectric effect to generate an electrical signal when light is incident. For example, the light sensing element may be implemented using various semiconductor elements, such as a pn junction photodiode, a positive-intrinsic-negative (PIN) photodiode, an avalanche photodiode (APD), a phototransistor, and the like.

[0051] The color filter may be one of a red filter, a green filter, and a blue filter. The red filter may transmit light indicating red by filtering incident light. The green filter may transmit light indicating green by filtering incident light. The blue filter may transmit light indicating blue by filtering incident light. In addition, this is only an example, and at least one of the red filter, the green filter, and the blue filter may be converted into a filter of a different color, such as a white filter, a cyan filter, or a yellow filter, or a filter of a different color may be added.

[0052] Multiple color filters can be arranged according to the Bayer pattern. For example, red filters, green filters, and blue filters can be arranged for each unit area. In this case, a unit area can be divided into four sub-areas, such as an upper left sub-area, an upper right sub-area, a lower left sub-area, and a lower right sub-area. In each of the four sub-areas, color filters of the same color can be arranged in m×n. Here, m and n are natural numbers. For example, in the case of a four-Bayer pattern, color filters of the same color can be arranged in 2×2 in each of the four sub-areas. For example, a green color filter can be arranged in each of the upper left sub-area and the lower right sub-area located in a diagonal direction, and a red color filter and a blue color filter can be arranged in the upper right sub-area and the lower left sub-area located in another diagonal direction, respectively.

[0053] The row decoder 112 may select a pixel located in a row corresponding to the address among a plurality of pixels included in the pixel array 111 in response to the address and the control signal output from the timing generator 113. In addition, the timing generator 113 may set an exposure value for each pixel. For example, the timing generator 113 may set an exposure value for each pixel based on the controller 120.

[0054] The signal converter 114 may obtain a pixel value of each of the plurality of pixels based on each of the signals output from the pixel array 111. The signal converter 114 may transmit the pixel values ​​of the pixels corresponding to the selected row to the output buffer 115 based on the timing generator 113.

[0055] The output buffer 115 may store pixel values ​​of pixels sequentially transferred from the signal converter 114 and output a raw image raw_IMG having the pixel values ​​of the pixels.

[0056] In addition, the pixel value can be obtained based on the amount of incident light on the pixel during the exposure time. For example, the pixel value can be proportional to the amount of charge accumulated by the light exposed during the exposure time.

[0057] As a specific example, a pixel may generate a current having a level corresponding to the intensity of incident light through a sensing circuit. A pixel may accumulate charge through the current generated during the exposure time. The charge may be accumulated in the sensing circuit. A pixel may generate a voltage having a level proportional to the amount of accumulated charge through a sensing circuit. At this time, the level of the generated voltage may indicate a pixel value.

[0058] In one embodiment, the pixel array 111 may include a plurality of pixels. The pixel array 111 may obtain a pixel value during an exposure time set for each of the plurality of pixels. In this case, the pixel array 111 may obtain a raw image raw_IMG including pixels having pixel values. At this time, the raw image raw_IMG may include pixel values ​​obtained during two or more different exposure times.

[0059] In one embodiment, the pixel array 111 may include a plurality of pixel groups. Here, different exposure times may be set in each of the plurality of pixel groups. Each pixel group may include a plurality of pixels arranged at regular distances. Each pixel may be any one of a red pixel, a blue pixel, and a green pixel.

[0060] In a specific embodiment, the pixel array 111 may include a first pixel group and a second pixel group. The first pixel group may include a plurality of pixels set with a first exposure value. The second pixel group may include a plurality of pixels set with a second exposure value. The pixels included in each pixel group may sense light during an exposure time corresponding to the exposure value set in each pixel group. The pixel array 111 may obtain a pixel value corresponding to the amount of light sensed by each pixel included in each pixel group.

[0061] In another embodiment, the pixel array 111 may further include at least one pixel group different from the first pixel group and the second pixel group. In this case, the other pixel group may include a plurality of pixels having exposure values ​​different from the first exposure value and the second exposure value. That is, the number of pixel groups may be determined according to the number of different exposure values.

[0062] In addition, in the above embodiment, the image sensor 110 can be implemented as a single type of obtaining one raw image raw_IMG through one sensor in a unit time. However, this is only one embodiment. The image sensor 110 can also be implemented as a multiple type of obtaining multiple raw images raw_IMG through multiple sensors in a unit time.

[0063] To this end, the image sensor 110 may include a plurality of pixel arrays 111. The above-mentioned optical lens LS may be disposed on each pixel array 111. The row decoder 112 may select a cell located in a row corresponding to the above-mentioned address in response to an address and a control signal output from the timing generator 113. The signal converter 114 may obtain a pixel value of each of a plurality of pixels based on each of the signals output from the pixel array 111. The signal converter 114 may send the pixel value of the pixel corresponding to the selected row to the output buffer 115 based on the timing generator 113. The output buffer 115 may store the pixel values ​​of the pixels sequentially transmitted from the signal converter 114, and may output a raw image raw_IMG including the pixel values ​​of the pixels. That is, the image sensor 110 may output the same number of raw images raw_IMG as the number of the plurality of pixel arrays 111.

[0064] Figure 2B is a diagram showing a controller according to an embodiment of the present disclosure.

[0065] See also Figure 2B , the controller 120 may include: an image extractor 121, a scaler 123, a motion map generator 127, and an image synthesizer 129. In addition, the controller 120 may further include at least one of an image selector 125 and a preprocessor 126. Here, the image extractor 121, the scaler 123, the motion map generator 127, and the image synthesizer 129 may be implemented as software modules that allow the controller 120 to perform corresponding operations. However, they are not limited thereto, but may be implemented as hardware modules (e.g., circuits, etc.) that allow the controller 120 to perform corresponding operations.

[0066] The image extractor 121 may divide the raw image raw_IMG received from the image sensor 110 into a plurality of images IMG. Specifically, the image extractor 121 may generate a plurality of images IMG from the raw image raw_IMG received from the image sensor 110 according to the exposure value ET. The plurality of images IMG generated by the image extractor 121 may be transmitted to the scaler 123.

[0067] In one embodiment, when the original image raw_IMG includes pixel values ​​obtained according to a first exposure value and pixel values ​​obtained according to a second exposure value, the image extractor 121 can generate a first image and a second image as follows, the first image having pixel values ​​obtained according to the first exposure value by the first pixel group among the pixel values ​​included in the original image raw_IMG obtained by the pixel array 111, and the second image having pixel values ​​obtained according to the second exposure value by the second pixel group.

[0068] In one embodiment, when the original image raw_IMG includes pixel values ​​obtained according to a first exposure value, pixel values ​​obtained according to a second exposure value, and pixel values ​​obtained according to a third exposure value, the image extractor 121 can generate the following first image, second image, and third image, the first image having pixel values ​​obtained according to the first exposure value among the pixel values ​​included in the original image raw_IMG obtained by the pixel array 111, the second image having pixel values ​​obtained according to the second exposure value, and the third image having pixel values ​​obtained according to the third exposure value.

[0069] The scaler 123 may receive a plurality of images IMG. The scaler 123 may receive an exposure value ET of each image IMG.

[0070] Specifically, each of the plurality of images may include a plurality of pixel values. Each of the plurality of images may include pixel values ​​obtained according to different exposure values. For example, the plurality of images may include: a first image having a first exposure value, a second image having a second exposure value, a third image having a third exposure value, and a fourth image having a fourth exposure value. Specifically, the first image may include pixel values ​​obtained by a first pixel group during a first exposure time indicated by the first exposure value. The second image may include pixel values ​​obtained by a second pixel group during a second exposure time indicated by the second exposure value. The third image may include pixel values ​​obtained by a third pixel group during a third exposure time indicated by the third exposure value. The fourth image may include pixel values ​​obtained by a fourth pixel group during a fourth exposure time indicated by a fourth exposure value. Here, it is assumed that the exposure value and the exposure time are proportional to each other. In the present disclosure, unless otherwise stated, it is assumed that the values ​​become larger in the order of the first exposure value, the second exposure value, the third exposure value, and the fourth exposure value.

[0071] The calibrator 123 can generate a plurality of scaled images SIMG corresponding to different exposure values ​​by correcting the plurality of images IMG using different exposure values. This is because, when comparing the same area of ​​the plurality of images IMG, the smaller the exposure value, the smaller the pixel value tends to be, and the larger the exposure value, the larger the pixel value tends to be. That is, this is to adjust images with different exposure values ​​to the same condition.

[0072] In one embodiment, the scaler 123 may correct the pixel values ​​of the first image to generate a first scaled image having the corrected pixel values. The scaler 123 may correct the pixel values ​​of the second image to generate a second scaled image having the corrected pixel values. In addition, the scaler 123 may correct the pixel values ​​of the third image to generate a third scaled image having the corrected pixel values. The scaler 123 may correct the pixel values ​​of the fourth image to generate a fourth scaled image having the corrected pixel values.

[0073] In one embodiment, the scaler 123 may calculate an exposure ratio obtained by dividing the maximum exposure value among the exposure values ​​of each of the plurality of images IMG by the exposure value of each image. The scaler 123 may obtain a pixel value that corrects the pixel value of each image by multiplying the exposure ratio by each pixel value included in each image. The scaler 123 may obtain a scaled image SIMG having a corrected pixel value. More specific details will be referred to in detail. Figure 5A and Figure 5B to describe.

[0074] The scaler 123 can correct the pixel value of each pixel of the plurality of images IMG by using the exposure value of each of the plurality of images IMG, and can obtain a plurality of scaled images SIMG obtained by converting the color of each pixel of the plurality of images IMG into grayscale. The color of the pixel can be three channels such as red, green, and blue, and the grayscale can be a single channel of black and white. More specific details will be referred to Fig. 6A Give a description.

[0075] The image selector 125 may determine a scale image having a minimum exposure value among the plurality of scale images SIMG as a reference scale image SIMG_R. The image selector 125 may determine one or more scale images different from the reference scale image SIMG_R among the plurality of scale images SIMG as target scale images SIMG_T. A corresponding motion map may be generated for each target scale image SIMG_T.

[0076] The preprocessor 126 may perform preprocessing on the plurality of scale images SIMG. The plurality of scale images SIMG may include a reference scale image SIMG_R and one or more target scale images SIMG_T. This may be used to generate a motion map that accurately indicates the motion of an object or to reduce the amount of computation required to generate a motion map.

[0077] In one embodiment, the preprocessor 126 can obtain a grayscale image obtained by converting the colors of the multiple scale images SIMG into grayscale. Here, the colors of the multiple scale images SIMG may include three channels of red, green and blue. The grayscale image may include a reference grayscale image GSIMG_R and one or more target grayscale images GSIMG_T. For example, the preprocessor 126 can obtain a reference grayscale image GSIMG_R obtained by converting the colors of the reference scale image SIMG_R into grayscale. The preprocessor 126 can obtain a target grayscale image GSIMG_T obtained by converting the colors of the target scale image SIMG_T into grayscale.

[0078] In addition, according to an embodiment of the present disclosure, the preprocessor 126 may be omitted. In this case, the image selector 125 may send the reference scale image SIMG_R and one or more target scale images SIMG_T to the motion map generator 127 .

[0079] The motion map generator 127 may generate a motion map for each target scale image SIMG_T. Alternatively, the motion map generator 127 may generate a motion map for each target grayscale image GSIMG_T. Hereinafter, for ease of explanation, the content of generating a motion map for each target scale image SIMG_T is described.

[0080] In one embodiment, the motion map generator 127 may generate a motion map indicating motion of an object based on the position of a first block indicating the object in the first block group and the position of a second block indicating the object in the second block group.

[0081] Here, the first block group may correspond to a first scale image. Any of the multiple blocks included in the first block group may correspond to an area of ​​the first scale image. For example, the first block group may include multiple first blocks. The first scale image may include multiple first areas. Here, the first block may correspond to the first area. The first block may have a block value calculated based on the pixel values ​​included in the first area. That is, a block value may be a value calculated by using multiple pixel values. In addition, the second block group may correspond to a second scale image. The second block group may include multiple second blocks. The second scale image may include multiple second areas. The second block may correspond to the second area. The second block may have a block value calculated based on the pixel values ​​included in the second area.

[0082] A motion map may be generated for each of the remaining target scale images SIMG_T except the reference scale image SIMG_R among the plurality of scale images SIMG. That is, the number of motion maps may be a number obtained by subtracting one from the number of the plurality of scale images SIMG. In addition, the scale image SIMG used to generate the motion map may be a grayscale image converted into grayscale.

[0083] The motion map may correspond to the first block group (or the second block group). The motion map may include a plurality of motion values. Each motion value may correspond to each block included in the first block group (or the second block group). Here, the motion value may represent the degree to which the motion of the object occurs at the position indicated by the corresponding block. The motion value may be a value calculated based on the block value included in the corresponding block.

[0084] The motion value may be used as a weight for the weight operation. Specifically, the motion value may have a positional relationship corresponding to an area of ​​the image corresponding to the block. In this case, the motion value may be used as a weight for the pixel values ​​included in an area of ​​the corresponding image. As the motion value increases, the degree to which the motion of the object occurs may increase. That is, as the motion value increases, the motion value may indicate that the object moves farther or faster.

[0085] In addition, among the first scale image and the second scale image, the scale image having a small exposure value may be predetermined as a reference scale image SIMG_R, and the other scale image may be predetermined as a target scale image SIMG_T.

[0086] In one embodiment, the motion map generator 127 may generate a motion map based on a distance between a position of an object included in the reference scale image SIMG_R and a position of an object included in the scale image SIMG having a selected exposure value.

[0087] In a specific embodiment, when the above-mentioned distance is greater than the minimum reference value and equal to or less than the maximum reference value, the motion map generator 127 may increase the motion value corresponding to the position of the object among the multiple motion values ​​included in the motion map as the distance increases. That is, the motion value corresponding to the position of the object may be a value proportional to the distance. In one embodiment, when the above-mentioned distance is equal to or less than the minimum reference value, the motion map generator 127 may adjust the motion value corresponding to the position of the object among the multiple motion values ​​included in the motion map to a value of 0. In one embodiment, when the above-mentioned distance is greater than the maximum reference value, the motion map generator 127 may adjust the motion value corresponding to the position of the object among the multiple motion values ​​included in the motion map to a value of 1.

[0088] The image synthesizer 129 may generate an output image Out_IMG according to a weight operation by using the motion map as a reference scale image and a scale image having a selected exposure value among the plurality of scale images SIMG. The image synthesizer 129 may output the output image Out_IMG.

[0089] In one embodiment, the image synthesizer 129 may increase the weight of the pixel values ​​of the reference scale image as the motion value increases. As the motion value increases, the image synthesizer 129 may decrease the weight of the pixel values ​​of the target scale image. As the motion value increases, the image synthesizer 129 may generate an output image according to a weight operation that increases the weight of the pixel values ​​of the reference scale image and decreases the weight of the pixel values ​​of the target scale image.

[0090] In this regard, when the output image Out_IMG is generated based on the synthesis of the images, the possibility of a ghosting phenomenon occurring at a position where the motion value is large increases. According to the present disclosure, at a position where the motion value is large, by applying the weight of the reference scale image with a small exposure value to a large value and applying the weight of the scale image with a large exposure value to a small value, the ghosting phenomenon can be prevented from occurring.

[0091] Figure 2C is a diagram showing a detailed configuration of a scaler, a preprocessor, and a motion map generator according to an embodiment of the present disclosure.

[0092] See also Figure 2C According to (1), the scaler 123 may include a maximum exposure value determiner 123-1, an exposure value ratio calculator 123-3, and a pixel value corrector 123-5.

[0093] The maximum exposure value determiner 123-1 may determine the maximum value among the plurality of exposure values ​​as the maximum exposure value. The maximum exposure value determiner 123-1 may output information on the maximum exposure value.

[0094] In one embodiment, it is assumed that the plurality of exposure values ​​include a first exposure value and a second exposure value. In this case, the maximum exposure value determiner 123-1 may determine the maximum value of the first exposure value and the second exposure value as the maximum exposure value.

[0095] The exposure value ratio calculator 123 - 3 may output information on a ratio value obtained by dividing the maximum exposure value by each of the plurality of exposure values.

[0096] In one embodiment, it is assumed that the plurality of exposure values ​​include a first exposure value and a second exposure value. In this case, the exposure value ratio calculator 123-3 may output information about a first ratio value obtained by dividing the maximum exposure value by the first exposure value and a second ratio value obtained by dividing the maximum exposure value by the second exposure value.

[0097] The pixel value corrector 123-5 can generate a plurality of scale images by correcting each of the plurality of images. Taking the first image as an example, the pixel value corrector 123-5 can generate a first scale image by multiplying the pixel values ​​included in the first image by a first ratio value corresponding to the first image. In this case, the pixel values ​​included in the first scale image can be values ​​obtained by multiplying the pixel values ​​of the first image by the first ratio value, respectively.

[0098] In one embodiment, it is assumed that the plurality of exposure values ​​include a first exposure value and a second exposure value. In this case, the pixel value corrector 123-5 may generate a first scaled image in which the pixel values ​​of the first image are corrected using a value obtained by multiplying the pixel values ​​of the first image by a first ratio value. The pixel value corrector 123-5 may generate a second scaled image in which the pixel values ​​of the second image are corrected using a value obtained by multiplying the pixel values ​​of the second image by a second ratio value.

[0099] See also Figure 2C (2), in one embodiment, the preprocessor 126 may include a grayscale converter 126-1.

[0100] The grayscale converter 126-1 may convert the color of each of the plurality of scale images into a grayscale color. Specifically, the grayscale converter 126-1 may generate a grayscale image using a pixel value obtained by converting the color of the scale image into a grayscale color.

[0101] In one embodiment, the grayscale converter 126-1 may generate a first grayscale image, which includes pixel values ​​of grayscale colors obtained by using pixel values ​​of red, green, and blue included in one region of the first scale image, respectively. The grayscale converter 126-1 may generate a second grayscale image, which includes pixel values ​​of grayscale colors obtained by using pixel values ​​of red, green, and blue included in one region of the second scale image, respectively. In this case, Figure 2C The block value calculator 127-1 of (3) can respectively calculate the average value of pixel values ​​included in an area of ​​the first grayscale image as the block value of a block included in the first block group, and can respectively calculate the average value of pixel values ​​included in an area of ​​the second grayscale image as the block value of a block included in the second block group.

[0102] In one embodiment, the pre-processor 126 may include a pyramid image generator 126 - 3 .

[0103] The pyramid image generator 126 - 3 may generate a plurality of pyramid images in which the resolutions of the plurality of scale images are reduced. Alternatively, the pyramid image generator 126 - 3 may generate a plurality of pyramid images in which the resolutions of the plurality of grayscale images are reduced.

[0104] In one embodiment, the pyramid image generator 126-3 may obtain a reference pyramid image and a selection pyramid image, the reference pyramid image being obtained by reducing the resolution of a reference scale image, and the selection pyramid image being obtained by reducing the resolution of a target scale image. Here, the selection pyramid image may be referred to as a target pyramid image. In other words, the pyramid image generator 126-3 may obtain a first pyramid image in which the resolution of a first scale image is reduced and a second pyramid image in which the resolution of a second scale image is reduced. The first pyramid image may include a plurality of first pyramid regions. The first scale image may include a plurality of first regions. The first region may correspond to the first pyramid region. The number of pixel values ​​included in the first region may be greater than the number of pixel values ​​included in the first pyramid region. The second pyramid image may include a plurality of second pyramid regions. The second region may correspond to the second pyramid region. The number of pixel values ​​included in the second region may be greater than the number of pixel values ​​included in the second pyramid region. In this case, Figure 2C The block value calculator 127-1 of (3) may calculate the block value of each block included in the first pyramid block group by using the pixel value included in one area of ​​the first pyramid image. The block value calculator 127-1 may calculate the block value of each block included in the second pyramid block group by using the pixel value included in the second pyramid image.

[0105] refer to Figure 2C (3), the motion map generator 127 may include: a block value calculator 127-1, a block position detector 127-3, a block distance calculator 127-5 and a motion value calculator 127-7.

[0106] The block value calculator 127-1 may calculate the block value of the block by using the pixel value included in the scale image. One block may correspond to one of the plurality of regions included in the scale image. The block value of one block may be calculated by using the pixel value included in one of the plurality of regions included in the scale image.

[0107] In one embodiment, the block value calculator 127-1 may calculate the block value of each block included in the first block group by using the pixel value included in the first scale image. The block value calculator 127-1 may calculate the block value of each block included in the second block group by using the pixel value included in the second scale image.

[0108] In one embodiment, the block value calculator 127-1 may calculate the average value of the pixel values ​​included in an area of ​​the first scale image as the block value of a block included in the first block group. The block value calculator 127-1 may calculate the average value of the pixel values ​​included in an area of ​​the second scale image as the block value of a block included in the second block group. Here, each block may correspond to an area.

[0109] The block location detector 127 - 3 may determine a block indicating an object in each block group.

[0110] In one embodiment, the block position detector 127-3 may select a block value of any one of the plurality of blocks included in each block group, and when a value between the block value of the selected block and the block value of the adjacent block exceeds a reference value, determine the selected block as a block indicating an object.

[0111] In one embodiment, the block position detector 127-3 may compare a first block group corresponding to a reference scale image and a second block group corresponding to a target scale image. The block position detector 127-3 may distinguish between block values ​​of blocks included in the first block group and block values ​​of blocks included in the second block group between blocks at the same position. A value close to 0 among the distinguishing result values ​​may indicate an area where movement of an object does not occur or a background area where an object does not exist. Among the distinguishing result values, a value greater than a reference value may indicate an area where movement of an object occurs. The block position detector 127-3 may determine a block at a position having a value greater than the reference value in the distinguishing result value as a block indicating an object.

[0112] In one embodiment, the block position detector 127-3 may determine a first pyramid block having a block value indicating an object in the first pyramid block group, and may determine a second pyramid block having a block value indicating an object in the second pyramid block group. The block position detector 127-3 may determine a block corresponding to a position of the first pyramid block among blocks included in the first block group as a first block, and may determine a block corresponding to a position of the second pyramid block among blocks included in the second block group as a second block.

[0113] The block distance calculator 127-5 may calculate the distance between the position of a first block having a block value indicating an object among the blocks included in the first block group and the position of a second block having a block value indicating an object among the blocks included in the second block group. Here, the position of the first block may indicate the relative position of the first block relative to the plurality of blocks included in the first block group. The position of the second block may indicate the relative position of the second block relative to the plurality of blocks included in the second block group.

[0114] The motion value calculator 127 - 7 may calculate a motion value to be included in the motion map based on a distance between a position of a first block indicating an object among blocks included in the first block group and a position of a second block indicating the object among blocks included in the second block group.

[0115] In one embodiment, when the distance between the first block and the second block is greater than the minimum reference value and equal to or less than the maximum reference value, the motion value calculator 127-7 may calculate a value proportional to the distance as the motion value. When the distance is equal to or less than the minimum reference value, the motion value calculator 127-7 may calculate a value of 0 as the motion value. When the distance is greater than the maximum reference value, the motion value calculator 127-7 may calculate a value of 1 as the motion value.

[0116] Figure 3 is a diagram illustrating a method of operating an electronic device according to an embodiment of the present disclosure.

[0117] See also Figure 3 , the method of operating the electronic device 100 may include: obtaining a plurality of images set with different exposure times (S310); obtaining a plurality of scale images based on the exposure times of the plurality of images (S320); generating a motion map based on the position of an object included in each of the plurality of scale images (S330); and generating an output image by weighting the plurality of scale images and the motion map (S340).

[0118] Specifically, a plurality of images with different exposure times may be obtained (S310). That is, a plurality of images may be obtained from a plurality of pixel groups. Each of the plurality of pixel groups may sense pixel values ​​during different exposure times.

[0119] In one embodiment, obtaining multiple images may include: obtaining pixel values ​​through a pixel array 111 having multiple pixels during an exposure time individually set in each of the multiple pixels; obtaining multiple images, each image including pixels having pixel values ​​obtained during the same exposure time.

[0120] In addition, a plurality of scale images may be obtained based on the exposure time of the plurality of images (S320). That is, a plurality of scale images obtained by correcting the plurality of images may be obtained by using the exposure time of each of the plurality of images.

[0121] In one embodiment, obtaining multiple scale images may include: obtaining a scale image obtained by correcting pixels of a selected image by the following operation, wherein the operation is an operation of multiplying each of the pixel values ​​of the pixels included in the selected image by a ratio obtained by dividing the maximum exposure time among the exposure times of each image in the multiple images by the exposure time of the selected image among the multiple images.

[0122] In one embodiment, obtaining multiple scale images may include: correcting pixel values ​​of pixels of each of the multiple images by utilizing the exposure time of each of the multiple images, and obtaining multiple scale images by converting the color of each pixel in the multiple images into grayscale.

[0123] In addition, a motion map may be generated based on the position of the object included in each of the plurality of scale images (S330). That is, a motion map indicating a change in the position of the object may be generated based on a reference scale image having a minimum exposure time among the plurality of scale images and a selected scale image. The selected scale image may be a scale image having an exposure time different from the minimum exposure time among the plurality of scale images.

[0124] In one embodiment, generating the motion map may include generating the motion map based on a difference between a position of an object included in the reference scale image and a position of an object included in the scale image having a selected exposure time. Here, the position difference may represent a distance.

[0125] Generating the motion map may include increasing a motion value corresponding to the position of the object among a plurality of motion values ​​included in the motion map when the position difference increases, wherein the position value increases when the position difference is greater than a minimum reference value and equal to or less than a maximum reference value.

[0126] In one embodiment, generating the motion map may include adjusting a motion value corresponding to the position of the object among a plurality of motion values ​​included in the motion map to a value of 0 when the position difference is equal to or less than a minimum reference value.

[0127] In one embodiment, generating the motion map may include adjusting a motion value corresponding to the position of the object among a plurality of motion values ​​included in the motion map to a value of 1 when the position difference is greater than a maximum reference value.

[0128] In one embodiment, generating the motion map may include: generating a reference pyramid image in which a reference scale image is reduced to a preset resolution; generating a pyramid image in which a scale image having a selected exposure time is reduced to a preset resolution; and generating the motion map based on a position difference between an object included in the reference pyramid image and an object included in the pyramid image.

[0129] In addition, an output image may be generated by weighting the plurality of scale images and the motion map (S340). That is, an output image synthesized by using the reference scale image and the selected scale image as weights may be output.

[0130] FIG. 4A to FIG. 4D is a diagram showing an image according to an embodiment of the present disclosure. Here, FIG. 4A to FIG. 4D An image 400 and a portion of first to fourth images 410 to 440 are shown.

[0131] See also FIG. 4A to FIG. 4D , the controller 120 according to an embodiment of the present disclosure may obtain the image 400 through the image sensor 110 having color filters arranged according to the Bayer pattern.

[0132] In this case, the image 400 may include a plurality of pixels arranged according to the Bayer pattern. The Bayer pattern may refer to a method in which red pixels R and blue pixels B are positioned diagonally from each other in a unit area and green pixels Gr and Gb are arranged in the remaining positions.

[0133] In one embodiment, image 400 may include multiple unit areas. Each unit area may include four sub-areas arranged in 2×2. For example, a unit area may include an upper left sub-area, an upper right sub-area, a lower left sub-area, and a lower right sub-area. Each sub-area may include pixels of the same color arranged in m×n. Here, m and n are natural numbers. As a specific example, the case of a four-Bayer pattern in which m and n are 2 is taken as an example for explanation. In this case, the first green pixel Gr may be arranged in 2×2 in the upper left sub-area. In the upper right sub-area, the red pixel R may be arranged in 2×2. In the lower left sub-area, the blue pixel B may be arranged in 2×2. In the lower right sub-area, the second green pixel Gb may be arranged in 2×2. However, this is only an example, and each of m and n can be transformed into various natural numbers.

[0134] The same exposure value may be set for pixels located at a relatively same position in each sub-region. For example, in each sub-region, a first exposure value ET_1 may be set for a pixel located at (1, 1), a second exposure value ET_2 may be set for a pixel located at (2, 1), a third exposure value ET_3 may be set for a pixel located at (1, 2), and a fourth exposure value ET_4 may be set for a pixel located at (2, 2). In one example, the exposure value increases in the order of the first exposure value ET_1, the second exposure value ET_2, the third exposure value ET_3, and the fourth exposure value ET_4. Each pixel may include a pixel value obtained during an exposure time indicated by the exposure value set for each pixel.

[0135] The controller 120 may obtain the plurality of images 410 to 440 based on pixels set with the same exposure value in the image 400. In addition, such an operation may be an operation performed by a pre-processor included in the controller 120.

[0136] See also Figure 4A, the controller 120 may extract pixels 41Gr, 41R, 41B, and 41Gb in which the first exposure value ET_1 is set among a plurality of pixels included in the image 400. The controller 120 may obtain a first image 410 in which the extracted pixels 41Gr, 41R, 41B, and 41Gb are repeatedly set in an area corresponding to the extracted pixels 41Gr, 41R, 41B, and 41Gb. The area corresponding to the extracted pixels 41Gr, 41R, 41B, and 41Gb may be sub-areas 41-1, 41-2, 41-3, and 41-4 in which one of the extracted pixels 41Gr, 41R, 41B, and 41Gb is located.

[0137] See also Figure 4B , the controller 120 may extract pixels 42Gr, 42R, 42B, and 42Gb in which the second exposure value ET_2 is set among the plurality of pixels included in the image 400. The controller 120 may obtain a second image 420 in which the extracted pixels 42Gr, 42R, 42B, and 42Gb are repeatedly set in an area corresponding to the extracted pixels 42Gr, 42R, 42B, and 42Gb. The area corresponding to the extracted pixels 42Gr, 42R, 42B, and 42Gb may be sub-areas 42-1, 42-2, 42-3, and 42-4 in which one of the extracted pixels 42Gr, 42R, 42B, and 42Gb is located.

[0138] See also Figure 4C , the controller 120 may extract pixels 43Gr, 43R, 43B, and 43Gb set with the third exposure value ET_3 from among the plurality of pixels included in the image 400. The controller 120 may obtain a third image 430 in which the extracted pixels 43Gr, 43R, 43B, and 43Gb are repeatedly arranged in an area corresponding to the extracted pixels 43Gr, 43R, 43B, and 43Gb. The area corresponding to the extracted pixels 43Gr, 43R, 43B, and 43Gb may be a sub-area 43-1, 43-2, 43-3, and 43-4 in which one of the extracted pixels 43Gr, 43R, 43B, and 43Gb is located.

[0139] See also Figure 4D, the controller 120 may extract pixels 44Gr, 44R, 44B, and 44Gb in which the fourth exposure value ET_4 is set among the plurality of pixels included in the image 400. The controller 120 may obtain a fourth image 440 in which the extracted pixels 44Gr, 44R, 44B, and 44Gb are repeatedly set in an area corresponding to the extracted pixels 44Gr, 44R, 44B, and 44Gb. The area corresponding to the extracted pixels 44Gr, 44R, 44B, and 44Gb may be a sub-area 44-1, 44-2, 44-3, and 44-4 in which one of the extracted pixels 44Gr, 44R, 44B, and 44Gb is located.

[0140] Figure 5A and Figure 5B is a diagram showing a scale image according to an embodiment of the present disclosure.

[0141] See also Figure 5A , the controller 120 may obtain a plurality of images IMG_1 to IMG_n set with different exposure values ​​ET_1 to ET_n based on the pixel values ​​of the pixels obtained from the image sensor 110. For example, the controller 120 may obtain a first image IMG_1 including pixel values ​​obtained during a first exposure time indicated by the first exposure value ET_1 among a plurality of pixel values ​​obtained from the image sensor 110. The controller 120 may obtain a second image IMG_2 having pixel values ​​obtained during a second exposure time indicated by the second exposure value ET_2 among a plurality of pixel values ​​obtained from the image sensor 110.

[0142] The controller 120 may generate a plurality of scaled images in which pixel values ​​of a plurality of images having different exposure values ​​are corrected. In one embodiment, the controller 120 may obtain a scaled image in which pixels of a plurality of selected images are corrected by an operation in which a ratio obtained by dividing the maximum exposure value among the exposure values ​​of each of the plurality of images by the exposure value of the selected image among the plurality of images is multiplied by each of the pixel values ​​of the pixels included in the selected image. In addition, such an operation may be an operation performed by the scaler 123 included in the controller 120.

[0143] For example, refer to Figure 5A and Figure 5B, assuming that the exposure value increases from the first exposure value ET_1 to the nth exposure value ET_n. The controller 120 can obtain the first scale image SIMG_1 to the nth scale image SIMG_n, wherein the first image IMG_1 to the nth image IMG_n are corrected based on the exposure value and the pixel value of each of the first image IMG_1 to the nth image IMG_n. Here, a case of obtaining the kth scale image SIMG_k or 520 from the kth image IMG_k or 510 is described. The controller 120 can obtain the kth scale image SIMG_k or 520 of the pixel including the corrected pixel value 521 via the following operation, which is an operation of multiplying the pixel value PV_K or 511 of each pixel in the kth scale image IMG_k or 510 by the ratio of the exposure value. Here, the ratio of the exposure values ​​of the kth image IMG_k or 510 may be a value obtained by dividing the nth exposure value ET_n as the maximum exposure value by the kth exposure value ET_k of the kth image IMG_k or 510. By repeating such a method, the controller 120 may obtain the first to nth scale images SIMG_1 to SIMG_n.

[0144] According to one embodiment, the controller 120 may obtain a plurality of scaled images SIMG_1 to SIMG_n, which are obtained by correcting the pixel value of each pixel of the plurality of images IMG_1 to IMG_n using the exposure value ET_1 to ET_n of each of the plurality of images IMG_1 to IMG_n, and converting the color of each pixel of the plurality of images IMG_1 to IMG_n into grayscale. Here, the scaled images SIMG_1 to SIMG_n may be images having a grayscale color space. Grayscale may represent the color of a single channel, such as black and white (or gray shades). In this case, the pixel value of a pixel may represent brightness or amount of light. This will refer to Fig. 6A Specific description.

[0145] Fig. 6A is a diagram showing a scaled image converted into grayscale according to an embodiment of the present disclosure.

[0146] See also Fig. 6A , the controller 120 may obtain a plurality of scaled images 620 obtained by converting the color of a pixel of each of the plurality of images 610 into grayscale. Here, the plurality of images 610 may include red pixels R, green pixels Gr and Gb, and blue pixels B. In addition, such an operation may be an operation performed by the scaler 123 included in the controller 120.

[0147] In addition, the plurality of images 610 may be obtained by using a reference Figure 5A and Figure 5BThe exposure values ​​ET_1 to ET_n of each of the plurality of images IMG_1 to IMG_n described above are used to correct the pixel values. Figure 5A and Figure 5B As described, the controller 120 may obtain a plurality of scale images by first correcting pixel values ​​using an exposure value of each of the plurality of images and then converting the color of the pixels included in each of the plurality of images into grayscale, as shown in FIG. Fig. 6A However, this is only one embodiment, and the controller 120 may obtain a plurality of scale images by first converting the color of the pixels included in each of the plurality of images into grayscale and then correcting the pixel values ​​by using the exposure values ​​ET_1 to ET_n of each of the plurality of images converted into grayscale, as shown in FIG. Figure 5A and Figure 5B shown.

[0148] For example, the controller 120 may obtain the pixel value converted to the grayscale of the area 621 corresponding to the unit sub-area 611 by the following weight operation, which is an operation of multiplying the pixel values ​​of the red pixel R, the green pixels Gr and Gb, and the blue pixel B included in the unit sub-area 611 of the image 610 by a separate weight. In addition, the controller 120 may obtain a scaled image 620 including a brightness pixel Y having a converted pixel value. Here, the unit sub-area 611 may be an area where there are one red pixel R, one green pixel Gr and Gb, and one blue pixel B. At this time, the scaled image 620 may be referred to as a grayscale image to distinguish the scaled image 620 from the above-mentioned scaled image 520.

[0149] When the controller 120 converts the red pixel R, the green pixels Gr and Gb, and the blue pixel B included in the unit subregion 611 into grayscale, the controller 120 may calculate the luminance pixel Y of the region 621 corresponding to the unit subregion 611 through the following weight operation.

[0150] For example, the operation result of (0.257×pixel value of red pixel R)+(0.504×sum (or average) of pixel values ​​of green pixels Gr and Gb)+(0.098×pixel value of blue pixel B)+16 may be the pixel value of brightness pixel Y. For another example, the operation result of (0.299×pixel value of red pixel R)+(0.587×sum (or average) of green pixels Gr and Gb)+(0.114×pixel value of blue pixel B) may be the pixel value of brightness pixel Y. Meanwhile, the above examples are merely examples, and the pixel value of brightness pixel Y may be calculated by weight operations to which various weights are applied.

[0151] Figure 6B is a diagram showing blocks according to an embodiment of the present disclosure.

[0152] See also Figure 6B , the controller 120 according to the embodiment of the present disclosure can obtain the block value of each of the multiple regions included in the grayscale image 630. Specifically, each of the multiple regions included in the grayscale image 630 may include M×N pixel values ​​of brightness pixels Y. Here, any one region 631 of the multiple regions may correspond to any one block 641 (Block). That is, the controller 120 can calculate the block value of the block 641 corresponding to any one region 631 by using the pixel values ​​included in any one region 631. Here, M and N are natural numbers. For example, a block may include brightness pixels Y of 8×8, 16×16, 32×32, 64×64, etc. The grayscale image 630 may correspond to the block image 640. The M×N brightness pixels Y included in the grayscale image 630 may correspond to the block 641 included in the block image 640.

[0153] Here, the block may include a block value. The block value may be any one of an average pixel value of the luminance pixels Y included in the area corresponding to the block or a central value of the pixel values ​​of the luminance pixels Y.

[0154] The controller 120 can identify objects based on block values ​​of blocks. For example, in one embodiment, the controller can identify blocks with adjacent positions having block values ​​within a preset value based on the block value of one block as objects. The controller 120 can determine the position of the identified object as the position of the corresponding block. Alternatively, the controller 120 can determine the position of the identified object as the position of the pixel corresponding to the corresponding block. In addition, the controller 120 can identify objects by utilizing various algorithms, such as programming libraries for analyzing real-time computer vision (e.g., OpenCV, Python, etc.), various filters such as Sobel filters or blur filters, Canny edge detection, color-based, template-based or background differentiation methods. In addition, the above embodiment is only one embodiment, and multiple pixels of the image 610 can be grouped into blocks without the need to group them into blocks. Fig. 6A The image 610 is converted to grayscale.

[0155] FIG. 7A to FIG. 7C is a diagram illustrating a motion graph according to an embodiment of the present disclosure.

[0156] See also Fig. 7A , the controller 120 may select the scale image GSIMG_1 having the minimum exposure value among the plurality of scale images 710 and 720 as the reference scale image 710. Here, the reference scale image 710 may be the scale image GSIMG_1 having the minimum exposure value, and the reference scale image 710 may be a reference image that becomes a reference for comparison with the position of an object included in another scale image GSIMG_k.

[0157] The controller 120 may obtain a motion map 730 of each of the remaining scale images 720 by comparing the remaining scale images 720 excluding the reference scale image 710 among the plurality of scale images 710 and 720 with the reference scale image 710 one by one.

[0158] For example, a first scale image having the smallest first exposure value among a plurality of scale images may be selected as a reference scale image. A second motion map corresponding to the second scale image may be generated by utilizing a second scale image having a second exposure value greater than the first exposure value and a reference scale image. A third motion map corresponding to the third scale image may be generated by utilizing a third scale image having a third exposure value greater than the first exposure value and a reference scale image. A fourth motion map corresponding to the fourth scale image may be generated by utilizing a fourth scale image having a fourth exposure value greater than the first exposure value and a reference scale image. That is, the first motion map corresponding to the first scale image may be omitted.

[0159] Hereinafter, for convenience of description, the remaining scale image 720 is described based on one scale image.

[0160] Here, the controller 120 may obtain the motion map 730 of the scale image 720 by comparing the positions of the objects included in the reference scale image 710 with the positions of the objects included in the scale image 720. In addition, such an operation may be an operation performed by the motion map generator 127 included in the controller 120.

[0161] In one embodiment, the controller 120 may generate a reference pyramid image 715 obtained by reducing the reference scale image 710 to a preset resolution, and may generate a pyramid image 725 obtained by reducing the scale image 720 having a selected exposure value to a preset resolution. Here, the reference pyramid image 715 and the pyramid image 725 may be images in which the resolutions of the reference scale image 710 and the scale image 720 are gradually reduced. Here, the resolution may refer to the number of pixels arranged in a row direction and the number of pixels arranged in a column direction.

[0162] For example, the controller 120 may obtain a reference pyramid image 715 and a pyramid image 725 obtained by applying a Gaussian filter to the reference scale image 710 and the scale image 720 and reducing the resolution to 1 / 2. The resolution of the image may be reduced by removing even or odd pixels in the row direction and the column direction. Here, the reference scale image 710 and the scale image 720 may be a first level, the reference pyramid image 715 and the pyramid image 725 whose resolution is reduced to 1 / 2 may be a second level, the reference pyramid image 715 and the pyramid image 725 whose resolution is reduced to 1 / 4 may be a third level, and the reference pyramid image 715 and the pyramid image 725 whose resolution is reduced to 1 / 8 may be a fourth level. In this method, the resolution of the image may be repeatedly reduced, and the level of the image may indicate the degree of resolution reduction of the image.

[0163] The controller 120 may determine a change in the position of the object by comparing a block of the reference pyramid image 715 and a block of the pyramid image 725 of the same level. The controller 120 may determine the position of the object in the reference pyramid image 715 and the pyramid image 725 of the upper level, and may determine the position of the object in the reference pyramid image 715 and the pyramid image 725 of the lower level immediately before the upper level based on the position of the object determined in the upper level. By repeating such a process, the controller 120 may determine the position of the object in the reference scale image 710 and the scale image 720 of the lowest level.

[0164] For example, the controller 120 may determine the position of the recognized object based on the block value of the block included in each of the reference pyramid image 715 and the pyramid image 725 of the third level. In addition, the controller 120 may determine the position of the recognized object, that is, the object is recognized based on the block value of the block corresponding to the position of the object determined at the third level among the blocks included in each of the reference pyramid image 715 and the pyramid image 725 of the second level. In addition, the controller 120 may determine the position of the recognized object, that is, the object is recognized based on the block value of the block corresponding to the position of the object determined at the second level among the blocks included in each of the reference pyramid image 715 and the pyramid image 725 of the first level. As described above, after determining the position of the object of the higher level with a small resolution, the position of the object of the lower level with a large resolution may be utilized. Therefore, the amount of calculation required to determine the position of the object may be reduced, and fast image analysis may be performed.

[0165] See also Figure 7B, the controller 120 may determine the position difference of the object by comparing the reference scale image 710 and the selected scale image 720. Here, the reference scale image 710 may be a first scale image. The selected scale image 720 may be one of a second scale image to an nth scale image. The first scale image may be an image in which the pixel values ​​of pixels having a first exposure value (which is the minimum exposure value) are corrected, the second scale image may be an image in which the pixel values ​​of pixels having a second exposure value (which is greater than the first exposure value) are corrected, and the nth scale image may be an image in which the pixel values ​​of pixels having an nth exposure value (which is greater than the first exposure value) are corrected.

[0166] For example, it is assumed that the position of the block 711 indicating the object A among the blocks included in the reference scale image 710 is (5, 4), and the position of the block 723 indicating the same object A among the blocks included in the scale image 720 is (2, 2). In this case, when the position of the object changes from (5, 4) to (2, 2) relative to the block 723 at the position (2, 2), the controller 120 may calculate the position difference of the object as (5-2)^2+(4-2)^2=13. The controller 120 may apply a value of 0 as the position difference relative to the block (e.g., block (7, 2)) where the object does not exist. In addition, the controller 120 may apply a value of 13 equal to the value of the block 723 at the position (2, 2) as the position difference relative to the block 721 at the position (5, 4) where the object A existed but does not exist at present. However, this is merely an example, and the value applied to the block where the object does not exist and the block 721 where the object A existed but does not exist at present may be modified to another arbitrary value.

[0167] Also, see Figure 7C , the controller 120 can be controlled by Figure 7B Substitute the values ​​calculated in the described method into Figure 7C Here, the motion value MV_XY of the position (X, Y) of the block included in the motion map may be used as a weight. For example, the motion value MV_XY may have a value in the range of 0 or more and 1 or less.

[0168] In one embodiment, when the position difference D_XY of the object at the block position (X, Y) is greater than the minimum reference value Min and equal to or less than the maximum reference value Max, the controller 120 may increase the motion value MV_XY corresponding to the position of the object among the plurality of motion values ​​included in the motion map as the position difference D_XY increases.

[0169] In one embodiment, when the position difference D_XY is equal to or less than the minimum reference value Min, the controller 120 may adjust the motion value MV_XY corresponding to the position of the object among the plurality of motion values ​​included in the motion map to a value of 0. That is, when the position difference D_XY is equal to or less than the minimum reference value Min, the motion value MV_XY may have a value of 0 as a minimum value.

[0170] In one embodiment, when the position difference D_XY is greater than the maximum reference value Max, the controller 120 may adjust the motion value MV_XY corresponding to the position of the object among the plurality of motion values ​​included in the motion map to a value of 1. That is, when the position difference D_XY exceeds the maximum reference value Max, the motion value MV_XY may have a value of 1 as a maximum value.

[0171] In addition, the controller 120 may perform a noise removal process on the calculated motion value MV_XY. For example, the controller 120 may perform a closing process after performing an opening process on the motion value MV_XY. Here, the opening process may be performing a dilation process after an erosion process, and the closing process may be performing an erosion process after an dilation process.

[0172] FIG. 8A to FIG. 8C is a diagram showing an output image according to an embodiment of the present disclosure.

[0173] See also Fig. 8A , the controller 120 may select a first scale image having a minimum exposure value among the plurality of scale images SIMG_1 to SIMG_4 as a reference scale image 810. The controller 120 may generate an output image 850 by performing an operation of weighting using motion maps 825, 835, and 845 as weights of each of the reference scale image 810 and the remaining scale images 820, 830, and 840. In addition, such an operation may be an operation performed by the image synthesizer 129 included in the controller 120.

[0174] The motion maps 825, 835, and 845 may be generated as a result of comparing one of the remaining scale images 820, 830, and 840 to the reference scale image 810, as described above with reference to FIG. FIG. 7A to FIG. 7C The reference scale image 810 and the residual scale images 820, 830, and 840 used to generate the motion maps 825, 835, and 845 may be images converted into grayscale.

[0175] As a specific example, the controller 120 may perform a first weight operation by utilizing a second motion map 825 corresponding to the second scale image 820 as a weight of the reference scale image 810 and the second scale image 820. The controller 120 may perform a second weight operation by utilizing a third motion map 835 corresponding to the third scale image 830 as a weight of the reference scale image 810 and the third scale image 830. The controller 120 may perform a third weight operation by utilizing a fourth motion map 845 corresponding to the fourth scale image 840 as a weight of the reference scale image 810 and the fourth scale image 840. The controller 120 may generate an output image 850 by an average value of the first weight operation to the third weight operation.

[0176] In addition, the above embodiment assumes that the number of the plurality of scale images SIMG_1 to SIMG_4 is four. In other words, the number of exposure values ​​having different values ​​is four. However, the present disclosure is not limited thereto, but may be variously modified and implemented according to the number of exposure values, etc.

[0177] In one embodiment, the controller 120 may Figure 8B , the output image Out_Img is generated by performing a weight operation using the motion map Motion_map_i as a weight for each of the reference scale image SIMG_1 and the remaining scale image SIMG_i. Here, n may be the number of a plurality of scale images or the number of exposure values ​​having different values.

[0178] The output image Out_Img may be an image synthesized through a weight operation, a weight of the reference scale image SIMG_1 having the minimum exposure value is applied as a large value and a weight of the scale image SIMG_i is applied as a small value.

[0179] See also Figure 8C , weight calculation can be performed on the scale image 860 and the motion map 870 by using the values ​​in the positional relationship corresponding to each other. Here, the scale image 860 may be a reference scale image or a scale image corresponding to the motion map 870. The motion map 870 may include motion values ​​calculated based on blocks corresponding to pixels, as described above with reference to 6A to 8C The pixel values ​​of the pixels included in the scale image 860 and the motion values ​​included in the motion map 870 may be operated on values ​​in a positional relationship corresponding to each other. For example, when the first area 861 included in the scale image 860 and the first motion value 871 of the motion map 870 are in a positional relationship corresponding to each other, an operation of multiplying the motion value 871 by the pixel values ​​of the pixels included in the first area 861 may be performed.

[0180] According to the present disclosure as described above, an electronic device 100 and an operating method thereof can be provided for outputting an image with improved image quality while preventing a ghost phenomenon from occurring. In addition, the occurrence of a ghost phenomenon in an image obtained by synthesizing images with different exposure times can be minimized.

[0181] Fig. 9 is a diagram showing an implementation example of an electronic device according to an embodiment of the present disclosure.

[0182] See also Fig. 9 , the electronic device 100 may be implemented as a computing system 2000. The computing system 2000 may include an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input / output device 2050, and a display device 2060. Fig. 9 Although not shown in the figure, the computing system 2000 may further include a port capable of communicating with the storage device 2030, the memory device 2040, the input / output device 2050, and the display device 2060 or capable of communicating with an external device.

[0183] The image sensor 2010 may obtain an image having a plurality of pixels to which exposure values ​​are individually applied. The image sensor 2010 may be connected to and communicate with the processor 2020 via an address bus, a control bus, a data bus, or other communication link.

[0184] The image sensor 2010 may be implemented as various types of packages. For example, at least some configurations of the image sensor 2010 may be implemented using packages such as package on package (PoP), ball grid array (BGA), chip scale package (CSP), plastic lead chip carrier (PLCC), plastic dual in-line package (PDIP), waffle package die, wafer form die, chip on board (COB), ceramic dual in-line package (CERDIP), plastic metric quad flat package (MQFP), thin quad flat package (TQFP), small outline (SOIC), shrink small outline package (SSOP), thin small outline (TSOP), system in package (SIP), multi-chip package (MCP), wafer level manufacturing package (WFP), wafer level processing stacked package (WSP), etc. According to one embodiment, the image sensor 2010 may be integrated with the processor 2020 in one chip, or the image sensor 2010 and the processor 2020 may be integrated in different chips, respectively.

[0185] The processor 2020 may control the overall operation of the computing system 2000. The processor 2020 may control the display device 2060 to display the output image. The processor 2020 may store the output image in the storage device 2030.

[0186] The processor 2020 may perform specific calculations or tasks. According to an embodiment of the present disclosure, the processor 2020 may include at least one of a central processing unit (CPU), an application processing unit (APU), a graphics processing unit (GPU), and the like.

[0187] The processor 2020 may be connected to the storage device 2030, the memory device 2040, and the input / output device 2050 through an address bus, a control bus, and a data bus to perform communication. According to an embodiment of the present disclosure, the processor 2020 may also be connected to an expansion bus such as a peripheral component interconnect (PCI) bus.

[0188] The processor 2020 may obtain a scaled image according to the exposure value of the image. The processor 2020 may select a scaled image with a minimum exposure value as a reference scaled image, and compare the reference scaled image with a scaled image with another exposure value to generate a motion map. The processor 2020 may generate an output image by weighting operation using the motion map as a weight of the reference scaled image and the scaled image with the other exposure values.

[0189] The storage device 2030 may store data such as output images. Here, the data stored in the storage device 2030 may be retained not only when the computing system 2000 is driven, but also when the computing system 2000 is not driven. For example, the storage device 2030 may be configured as one of all types of nonvolatile memories such as a flash memory device, a solid state drive (SSD), a hard disk drive (HDD), and an optical disk.

[0190] The storage device 2040 may store data such as an output image. The storage device 2040 may temporarily store data to be processed by the processor 2020 or temporarily store data processed by the processor 2020. Here, the data stored in the storage device 2040 may be retained only when the computing system 2000 is driven. Alternatively, the data stored in the storage device 2040 may be retained even when the computing system 2000 is driven or not driven. For example, the storage device 2040 may include: a volatile storage device such as a dynamic random access memory (DRAM) and a static random access memory (SRAM), and a non-volatile storage device such as an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory device.

[0191] The input / output device 2050 may include an input device and an output device. An input device may be a device capable of inputting a user command through interaction, for example, the input device may be implemented as a keyboard, a keypad, a mouse, a microphone, etc. An output device may be a device capable of outputting data, for example, the output device may be implemented as a printer, a speaker, etc.

[0192] The display device 2060 is a device that visually outputs data such as an output image. To this end, the display device 2060 can be implemented as various types of displays, such as a liquid crystal display, which uses a separate backlight unit (e.g., a light emitting diode (LED), etc.) as a light source, and controls the degree of light transmitted through the liquid crystal (brightness of light or intensity of light) emitted from the backlight unit by controlling the molecular arrangement of the liquid crystal; and a display that does not require a separate backlight unit or liquid crystal as a light source and uses a self-luminous element (e.g., a mini LED with a size of 100μm to 200μm, a micro LED with a size of 100μm or less, an organic LED (OLED), a quantum dot LED (QLED), etc.). In this case, the display device 2060 can emit red, green, and blue light corresponding to the output image to the outside.

Claims

1. An electronic device, comprising: An image sensor including a plurality of pixel groups corresponding to a plurality of exposure values, respectively; as well as A controller that: selecting a reference scale image having a minimum exposure value and a target scale image having an exposure value different from the minimum exposure value among a plurality of scale images obtained based on the plurality of pixel groups, and outputting an output image obtained using a motion map indicating a position change of an object commonly included in the reference scale image and the target scale image, wherein the controller includes a motion value calculator that obtains a motion map including motion values ​​calculated based on a distance between a first area included in a reference scale image and a second area included in the target scale image, each of the first area and the second area indicating an object, When the distance exceeds a minimum reference value and is equal to or less than a maximum reference value, the motion value calculator calculates a value proportional to the distance as the motion value.

2. The electronic device according to claim 1, wherein: The controller outputs the output image using the motion map as a weight for the reference scale image and the target scale image.

3. The electronic device according to claim 1, wherein: The controller comprises: an image extractor that obtains a plurality of images based on an exposure value of each of the plurality of pixel groups; and A scaler obtains the plurality of scaled images obtained by correcting the plurality of images using a ratio of a maximum exposure value among the plurality of exposure values ​​to an exposure value of each of the plurality of images.

4. The electronic device according to claim 1, wherein: The controller comprises: A block distance calculator obtains the distance between the first region among a plurality of first regions included in the reference scale image and the second region among a plurality of second regions included in the target scale image.

5. The electronic device according to claim 4, wherein: When the distance is equal to or less than the minimum reference value, the motion value calculator calculates a value of 0 as the motion value, and When the distance exceeds the maximum reference value, the motion value calculator calculates a value of 1 as the motion value.

6. The electronic device according to claim 4, wherein: The controller includes an image synthesizer, which generates the output image based on a weight operation, wherein the weight operation increases the weight of the pixel value corresponding to the motion value among the pixel values ​​included in the reference scale image, and decreases the weight of the pixel value corresponding to the motion value among the pixel values ​​included in the target scale image as the motion value increases.

7. The electronic device according to claim 4, wherein: The controller includes a block value calculator, the block value calculator obtaining an average value or a median value of pixel values ​​included in each of the plurality of first regions included in the reference scale image as a block value of each of the plurality of first regions, and An average value or a median value of pixel values ​​included in each of the plurality of second regions included in the target scale image is obtained as a block value of each of the plurality of second regions.

8. The electronic device according to claim 7, wherein: The controller includes a block position detector, the block position detector determining the first selected area as the first area indicating the object when a difference between a block value of a first selected area among the plurality of first areas and a block value of a first peripheral area located within a preset distance from the first selected area exceeds a reference value, and When a difference between a block value of a second selected area among the plurality of second areas and a block value of a second peripheral area located within a preset distance from the second selected area exceeds the reference value, the second selected area is determined as the second area indicating the object.

9. The electronic device according to claim 4, wherein: The controller comprises: a pyramid image generator that obtains a reference pyramid image obtained by reducing a resolution of the reference scale image and a selection pyramid image obtained by reducing a resolution of the target scale image; and Block position detector, which determining a first pyramid region indicating the object among a plurality of first pyramid regions included in the reference pyramid image, and a second pyramid region indicating the object among a plurality of second pyramid regions included in the selection pyramid image, and A region corresponding to the first pyramid region among the plurality of first regions is determined as the first region, and a region corresponding to the second pyramid region among the plurality of second regions is determined as the second region.

10. The electronic device according to claim 1, wherein: The controller includes a grayscale converter that obtains a plurality of grayscale images by converting pixel values ​​for red, pixel values ​​for green, and pixel values ​​for blue included in each of the plurality of scale images into pixel values ​​for grayscale.

11. The electronic device according to claim 10, wherein: The controller comprises: a block distance calculator that obtains a distance between a first area that is a first area indicating the object among a plurality of first areas included in a reference grayscale image having a minimum exposure value among a plurality of grayscale images and a second area that is a second area indicating the object among a plurality of second areas included in a selected grayscale image; and A motion value calculator obtains the motion map including motion values ​​calculated based on the distance.

12. A method for operating an electronic device, the method comprising: obtaining a plurality of images from a plurality of pixel groups sensing pixel values ​​during different exposure times; obtaining a plurality of scale images obtained by correcting the plurality of images using an exposure time of each of the plurality of images; generating a motion map indicating positional changes of an object based on a reference scale image having a minimum exposure time among the plurality of scale images and a selected scale image having an exposure time different from the minimum exposure time; and outputting an output image synthesized by using the motion map as a weight for the reference scale image and the selected scale image, The object is usually contained in the reference scale image and the selected scale image. The generating of the motion map includes calculating a value proportional to the position change as the motion value when the position change exceeds a minimum reference value and is equal to or less than a maximum reference value, The motion map includes motion values.

13. The method according to claim 12, wherein: Obtaining the plurality of scale images includes obtaining the plurality of scale images obtained by correcting the plurality of images using a ratio of a maximum exposure time among exposure times of each of the plurality of images to an exposure time of each of the plurality of images.

14. The method according to claim 12, wherein: Obtaining the plurality of scale images comprises: obtaining a plurality of corrected images obtained by correcting the plurality of images using an exposure time of each of the plurality of images; and The plurality of scale images obtained by converting the corrected image into grayscale are obtained.

15. The method according to claim 12, wherein: Generating the motion map includes calculating a value of 0 as the motion value when the position change is equal to or less than a minimum reference value.

16. The method according to claim 12, wherein: Generating the motion map includes calculating a value of 1 as the motion value when the position change exceeds a maximum reference value.

17. The method according to claim 12, wherein: Generating the motion map comprises: obtaining a reference pyramid image obtained by reducing a resolution of the reference scale image and a selection pyramid image obtained by reducing a resolution of the selection scale image; determining a first pyramid region indicating the object among a plurality of first pyramid regions included in the reference pyramid image and a second pyramid region indicating the object among a plurality of second pyramid regions included in the selection pyramid image; and A position difference between a region corresponding to the first pyramid region among a plurality of first regions included in the reference scale image and a region corresponding to the second pyramid region among a plurality of second regions included in the selection scale image is determined as the position change.

18. The method according to claim 12, wherein: Obtaining the plurality of images comprises: obtaining a first image having pixel values ​​sensed during a first exposure time from among a first pixel group included in the plurality of pixel groups; and A second image having pixel values ​​sensed during a second exposure time is obtained from a second pixel group included in the plurality of pixel groups, the second exposure time including the first exposure time.

Images