Image processing apparatus and image processing method
By utilizing the motion manager, similarity map generator, and noise remover in the image processing device, and taking advantage of information from the inertial sensor and image sensor, a similarity map is generated and a bilateral filter is applied. This solves the problem of noise removal when the image sensor is in motion, and achieves rapid and efficient image quality improvement.
Patent Information
- Application Number
- CN202310106238.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-14
- Filing Date
- 2023-01-31
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-01-31
AI Technical Summary
Existing technologies struggle to quickly and effectively remove noise from images when the image sensor is in motion, especially while maintaining image quality and reducing motion detection time.
By using the motion manager, similarity map generator, and noise remover in the image processing device, the motion information of the image sensor is generated using the velocity information of the inertial sensor and the pixel values of the image sensor. A similarity map is generated and a bilateral filter is applied to remove temporal and spatial noise.
It enables rapid and accurate removal of temporal and spatial noise during image sensor motion, improving image quality and reducing motion detection time.
Smart Images

Figure CN117412191B_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application claims priority to Korean Patent Application No. 10-2022-0086997, filed on July 14, 2022, the entire disclosure of which is incorporated herein by reference. Technical Field
[0003] The various embodiments disclosed herein generally relate to image processing apparatus, and more specifically to image processing apparatus and image processing method. Background Technology
[0004] Image sensors are typically categorized into charge-coupled device (CCD) image sensors and complementary metal-oxide-semiconductor (CMOS) image sensors. Recently, CMOS image sensors, with their low manufacturing cost, low power consumption, and ease of integration with peripheral circuitry, have attracted considerable attention.
[0005] Image sensors included in smartphones, tablet PCs, or digital cameras can acquire image information of external objects by converting light reflected from them into electrical signals. Image signal processing devices can perform operations to convert the electrical signals obtained from the image sensor or improve image quality.
[0006] Image processing devices can remove noise from images. They can detect motion by comparing consecutive images and remove noise. However, the time required to detect motion through image comparison can increase when the image sensor is moved. Therefore, new methods are needed to remove noise from images while reducing motion detection time. Summary of the Invention
[0007] Various embodiments of this disclosure relate to an image processing apparatus and an image processing method that rapidly acquire motion information from an image sensor, determine a comparison region in a corresponding image based on the acquired information, and remove noise contained in the corresponding image based on the result of an operation performed on the comparison region.
[0008] One embodiment of this disclosure may provide an image processing apparatus. The image processing apparatus may include: a motion manager configured to: generate motion information of an image sensor based on pixel values of a first frame image received from an image sensor, pixel values of a second frame image received from the image sensor prior to the first frame image, and velocity information of an inertial sensor configured to sense the velocity of the image sensor; a similarity map generator configured to generate a similarity map indicating the weights of a noise removal filter based on the pixel values of the first frame image; and a noise remover configured to: remove temporal and spatial noise contained in the first frame image based on pixel values of the first frame image to which the noise removal filter is applied according to the similarity map and pixel values of the second frame image reflecting the motion information.
[0009] One embodiment of this disclosure provides an image processing method. The image processing method may include: generating motion information of an image sensor based on pixel values of a first frame image received from an image sensor, pixel values of a second frame image received from the image sensor prior to the first frame image, and velocity information from an inertial sensor included in the image sensor; generating a similarity map indicating the weights of a noise removal filter based on the pixel values of the first frame image; and removing temporal and spatial noise contained in the first frame image based on the pixel values of the first frame image to which the noise removal filter is applied according to the similarity map and the pixel values of the second frame image reflecting the motion information.
[0010] One embodiment of this disclosure provides an image processing method. The image processing method may include: generating motion information based on the sum of absolute differences between corresponding pixel values of a first region of interest (ROI) and a second image provided from a moving image sensor; removing spatial noise from a target image based on a weighted value representing the difference between a center pixel value and adjacent pixel values within the first ROI; and removing temporal noise from the target image by combining the second ROI and the first ROI in the target image from which spatial noise has been removed. The second image may be earlier than the first image, and both temporal and spatial noise can be removed from the second image. The amount of movement of the image sensor between the first and second images can be separated from the region corresponding to the first ROI by the second ROI. Attached Figure Description
[0011] Figure 1 This is a diagram illustrating an image processing apparatus according to an embodiment of the present disclosure.
[0012] Figure 2This is a diagram illustrating a temporal and spatial noise removal method according to an embodiment of the present disclosure.
[0013] Figure 3 This is a diagram illustrating motion information of an image sensor according to an embodiment of the present disclosure.
[0014] Figure 4 This is a diagram illustrating a temporal noise removal method according to an embodiment of the present disclosure.
[0015] Figure 5 This is a flowchart illustrating a temporal and spatial noise removal method according to an embodiment of the present disclosure.
[0016] Figure 6 This is a flowchart illustrating a method for generating motion information of an image sensor according to an embodiment of the present disclosure.
[0017] Figure 7 This is a flowchart illustrating a method for generating a modified image according to an embodiment of the present disclosure.
[0018] Figure 8 This is a block diagram illustrating an electronic device including an image processing apparatus according to an embodiment of the present disclosure. Detailed Implementation
[0019] Specific structural or functional descriptions of embodiments of the present disclosure incorporated herein are provided as examples to describe embodiments based on the concept of the present disclosure. Embodiments based on the concept of the present disclosure may be practiced in a variety of forms and should not be construed as being limited to the embodiments described herein.
[0020] Various embodiments of the present disclosure will now be described more fully below with reference to the accompanying drawings, which illustrate preferred embodiments of the present disclosure, so that those skilled in the art can readily practice the technical spirit of the present disclosure.
[0021] Figure 1 This is a diagram illustrating an image processing apparatus according to an embodiment of the present disclosure.
[0022] refer to Figure 1 The image processing apparatus 100 can perform processing that improves the image quality of pixel data received from the image sensor and can output the processed image data. This processing may include electronic image stabilization (EIS), interpolation, tone correction, image quality correction, resizing, etc.
[0023] The image processing apparatus 100 can receive pixel values from the image sensor. The image processing apparatus can also receive velocity information (or velocity data) related to the movement of the image sensor from an inertial sensor. The inertial sensor can be located within the image sensor. The velocity information from the inertial sensor can be information related to the angular velocity or rotational velocity of the inertial sensor in the direction of motion (movement) of the image sensor.
[0024] In one embodiment of this disclosure, the image processing apparatus 100 may include a motion manager 110, a similarity map generator 120, and a noise remover 130. The image processing apparatus 100 can remove temporal and spatial noise contained in the image, taking into account the movement of the image sensor.
[0025] The motion manager 110 can generate motion information of the image sensor based on the pixel values of a first frame image received from the image sensor, the pixel values of a second frame image received from the image sensor prior to the first frame image, and the velocity information of the inertial sensor. The velocity information of the inertial sensor may include information related to the angular velocity generated by the rotation of the inertial sensor relative to the x-axis, y-axis, and z-axis.
[0026] The motion manager 110 can calculate the first amount of movement obtained by integrating velocity information in the time domain. The motion manager 110 can calculate the Euler's change from the angular velocity. The motion manager 110 can calculate the amount of movement of the image sensor by integrating the Euler's change.
[0027] The motion manager 110 can generate motion information of the image sensor based on the difference between the pixel values of a first region of interest (ROI) set in the first frame image and the pixel values of a second region of interest (ROI) set in the second frame image. The motion manager 110 can move the second region of interest from its position corresponding to the first region of interest by a first movement amount in the second frame image.
[0028] In one embodiment of this disclosure, when the sum of the absolute values of the differences between the pixel values of the first region of interest and the second region of interest (i.e., the sum of absolute differences) is less than a preset reference value, the motion manager 110 can generate motion information of the image sensor based solely on the first amount of movement. Here, the motion manager 110 can accurately calculate the amount of movement of the image sensor based on the velocity information of the inertial sensor.
[0029] When the sum of the absolute differences between pixel values is equal to or greater than the reference value, the motion manager 110 can detect a second movement amount in which the sum of the absolute differences between pixel values is minimized. The motion manager 110 can generate motion information of the image sensor based on the second movement amount. The motion manager 110 can correct the first movement amount based on the second movement amount.
[0030] When moving a second region of interest within a first range, the motion manager 110 can detect a second amount of movement. The motion manager 110 can determine the first range based on the first amount of movement. In one embodiment of this disclosure, the motion manager 110 can determine the first range as a wide range as the first amount of movement increases.
[0031] The similarity map generator 120 can generate a similarity map indicating the weights of a noise removal filter based on the pixel values of the first frame image. In one embodiment of this disclosure, the noise removal filter may be a bilateral filter. The similarity map generator 120 can determine the weights of the bilateral filter based on at least one of the differences between pixel values in the first frame image and the differences between the positions of pixels in the first frame image.
[0032] The similarity map generator 120 can generate a similarity map based on the weights of a bilateral filter, wherein the weights increase as the difference between the pixel values of the center pixel and its neighboring pixels in the first region of interest decreases. The similarity map generator 120 can also generate a similarity map based on the weights of a bilateral filter, wherein the weights increase as the positions of the center pixel and its neighboring pixels in the first region of interest become closer to each other.
[0033] The noise remover 130 can remove temporal and spatial noise contained in the first frame image based on the pixel values of the first frame image to which a noise removal filter is applied according to a similarity map and the pixel values of the second frame image reflecting the motion information of the image sensor. The noise remover 130 can remove spatial noise contained in the first frame image based on the similarity map. The noise remover 130 can remove temporal noise contained in the first frame image based on the motion information of the image sensor.
[0034] The noise remover 130 can calculate the average of the pixel values of a first frame image for which a bilateral filter is applied based on a similarity map, and the average of the pixel values of a second frame image for which the image sensor's motion information is shifted in parallel. The noise remover 130 can then generate a corrected image of the first frame image based on the calculated average.
[0035] In one embodiment of this disclosure, the noise remover 130 can perform a noise removal operation by combining a first region of interest in a first frame image (for which spatial noise has been removed by applying a bilateral filter) with a second region of interest in a second frame image to remove temporal noise. The noise remover 130 can move the second region of interest from a position corresponding to the first region of interest by a first or second amount based on motion information from the image sensor.
[0036] In one embodiment of this disclosure, the image processing apparatus 100 may be included in an image processing system comprising both an image sensor and an inertial sensor. The inertial sensor may output velocity information related to the motion (movement) of the image sensor. The inertial sensor may be located inside or outside the image sensor.
[0037] An image processing system according to one embodiment of this disclosure can store or display processed images or output processed images to an external device. In one embodiment, the image processing system can be implemented as a packaged module, component, etc. In this case, the image processing system can be installed in a host computer. The host computer can be implemented as any type of electronic device from a variety of types. For example, the host computer can be implemented as a digital camera, mobile device, smartphone, personal computer (PC), tablet PC, laptop computer, personal digital assistant (PDA), enterprise digital assistant (EDA), portable multimedia player (PMP), wearable device, black box, robot, autonomous vehicle, etc.
[0038] An image sensor according to one embodiment of this disclosure can be implemented as a charge-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor. The image sensor can generate image data relating to an object input (or captured) through a lens.
[0039] An image sensor may include multiple pixels. The image sensor can generate multiple pixel values corresponding to an image captured from the multiple pixels. The multiple pixel values generated by the image sensor can be sent to the image processing apparatus 100. That is, the image sensor can generate multiple pixel values corresponding to a single frame.
[0040] In one embodiment of this disclosure, the image processing system may further include a memory. The memory may store multiple pixel values sensed in each frame. The memory may store filtered images and may provide images required for noise removal operations performed by the image processing apparatus 100.
[0041] Figure 2 This is a diagram illustrating a temporal and spatial noise removal method according to an embodiment of the present disclosure.
[0042] refer to Figure 2 The image processing device can remove temporal and spatial noise contained in the first frame image based on the pixel values of the first frame image, the pixel values of the second frame image, and the velocity information of the inertial sensor. The image processing device can detect the movement (motion) of the image sensor and remove the temporal noise contained in the corresponding image by taking the detected movement into account.
[0043] exist Figure 2In this process, velocity information can be received from an inertial sensor. Pixel data can indicate the pixel values of the first frame image. Previous pixel data can indicate the pixel values of the second frame image. The second frame image can be an image received from an image sensor prior to the first frame image. In one embodiment of this disclosure, the first frame image and the second frame image can be consecutive frame images.
[0044] The motion manager can generate motion information of the image sensor based on the pixel values of the first frame image received from the image sensor, the pixel values of the second frame image received from the image sensor before the first frame image, and the velocity information of the inertial sensor.
[0045] The motion manager can calculate a first amount of movement indicative of the image sensor's movement using only the velocity information from the inertial sensor. The velocity information from the inertial sensor can include angular velocity information about each axis of the coordinate plane. The motion manager can calculate the Euler change based on the angular velocity information, and can calculate the first amount of movement of the image sensor by integrating the Euler change.
[0046] The motion manager can correct for a first movement amount. Specifically, the motion manager can calculate the sum of the absolute differences between the pixel values of the first region of interest and the pixel values of the second region of interest, whereby the second region of interest moves by the first movement amount from its position corresponding to the first region of interest in the second frame image. When the sum of the absolute differences between the pixel values is less than a preset reference value, the motion manager can generate motion information of the image sensor using only the first movement amount.
[0047] When the sum of the absolute differences between pixel values is equal to or greater than a preset reference value, the motion manager can additionally use the pixel values of the first frame image and the second frame image to generate motion information of the image sensor.
[0048] The motion manager can move a second region of interest within a first range. While moving the second region of interest within the first range, the motion manager can calculate the sum of the absolute differences between pixel values. The motion manager can detect the position where the sum of the absolute differences between the pixel values of the first and second regions of interest within the first range is minimized. The motion manager can detect a second movement amount indicating the position where the sum of the absolute differences between pixel values is minimized. Based on the second movement amount, the motion manager can correct the motion information of the image sensor generated based on the first movement amount. In one embodiment of this disclosure, the motion information of the image sensor may be generated based solely on the first movement amount or may be corrected based on the second movement amount.
[0049] A similarity map generator can determine the weights of a bilateral filter for spatial noise removal using only the pixel values of a first frame of the image. The similarity map generator can determine similarity based on the differences between pixel values in regions of interest within the image or between the locations of those pixels. In one embodiment of this disclosure, a higher similarity can be defined as a smaller difference between pixel values or a smaller distance between pixels.
[0050] A similarity mapping generator can assign higher weights to pixels with higher similarity to each other. In one embodiment of this disclosure, a Gaussian filter can be strongly applied to pixels with higher similarity to each other. Because noise has a low impact on the similarity within each pixel group, a Gaussian filter can be strongly applied to the pixel group, thereby removing noise from the corresponding pixel group. Because the similarity between pixels at the boundary is low, which is not noise, a Gaussian filter can be weakly applied to the boundary, so that the boundary can be preserved in the corrected image.
[0051] In one embodiment of this disclosure, the similarity map generator can generate a similarity map using the following formula.
[0052]
[0053] Here, SIM map (i, j) can be a similarity map of size N*N at position (i, j), and ROI can be a similarity map for (c i c j The kernel of the region of interest with size N*N is located at the center of the region of interest. in (t, i, j) can be the pixel value at position (i, j) in a frame captured at time t, w n It can be a normalization factor, and σ r and σ d It can indicate the constant of the similarity mapping.
[0054] According to the above formula, the similarity mapping generator can assign greater weights to pixel values within the corresponding region of interest that have smaller differences from the center pixel value. The similarity mapping generator can also assign greater weights to pixels closer to the center within the corresponding region of interest.
[0055] The noise remover can use the average of the pixel values of a first frame image with a noise removal filter applied based on a similarity map and the pixel values of a second frame image that reflects the motion information of the image sensor to remove temporal and spatial noise contained in the first frame image.
[0056] The noise remover can apply a bilateral filter to the pixel values of the first frame image based on a similarity map. With the application of the bilateral filter, spatial noise included in the pixel values of the first frame image can be removed.
[0057] The noise remover can move a second region of interest (ROI) from a position corresponding to the first ROI by a first or second amount based on motion information from the image sensor. The noise remover can remove temporal noise by combining the first ROI in a first frame image (with spatial noise removed) with the second ROI in a moved second frame image. The noise remover can generate a corrected image that includes the average pixel values of the first and second ROIs.
[0058] In one embodiment of this disclosure, the noise-removed image can be stored in a memory. The image stored in the memory can be loaded to remove the noise contained in the next frame image.
[0059] Figure 3 This is a diagram illustrating motion information of an image sensor according to an embodiment of the present disclosure.
[0060] refer to Figure 3 The motion manager can generate motion information for the image sensor. During image sensing, the image sensor can be moved. When the image sensor is moved, the similarity between pixels can decrease. Image quality can only be improved by taking the amount of image sensor movement into account during image processing operations.
[0061] exist Figure 3 The image may show a first frame 310 and a second frame 320. In the first frame 310, a first region of interest 311 may be defined. The position of the center pixel of the first region of interest 311 can be (c... i c j ).
[0062] The motion manager can calculate a first displacement of the image sensor based on the velocity information from the inertial sensor. The first displacement can be a value obtained by integrating the Euler change in the angular velocity of the inertial sensor in the time domain. In one embodiment of this disclosure, the first displacement can be represented by (Δi, Δj).
[0063] From the position corresponding to the first region of interest 311, at position (c i c j A second region of interest 321, which is moved by a first movement amount (Δi, Δj), can be set in the second frame image. The position of the center pixel in the second region of interest 321 can be determined by (c i +Δi, c j It can be represented by +Δj).
[0064] The motion manager can calculate the first sum of the absolute values of the differences between pixel values in the first region of interest 311 and the second region of interest 321 that has been moved by a first amount of movement (Δi, Δj). The first sum of the absolute differences between pixel values can be expressed by the following formula:
[0065]
[0066] Here, SAD1 can be the first sum of the absolute differences between pixel values, I in (t) can be the first frame image 310, and I out (t-1) can be the second frame image 320.
[0067] When the first sum of the absolute differences between pixel values (SAD1) is less than a preset reference value, the motion manager can generate motion information (Δi, Δj) of the image sensor using only the first amount of movement.
[0068] When the first sum of the absolute differences between pixel values is equal to or greater than a preset reference value, the motion manager can calculate the second sum of the absolute differences between pixel values when moving the second region of interest within the first range, and can detect the corrected second region of interest 322 where the second sum of the absolute differences between pixel values is minimized. In one embodiment of this disclosure, the additional movement amount can be achieved through (δ... i δ j The position of the center pixel in the corrected second region of interest 322 can be represented by (c). i +Δi+δ i c j +Δj+δ j ) is used to represent this.
[0069] The second sum of the absolute differences between pixel values can be calculated using the following formula.
[0070]
[0071] exist Figure 3 In the process, the second movement amount detected by the motion manager can be obtained through (Δi+δ). i ,Δj+δ j The motion manager can correct the motion information of the image sensor generated based on the first motion amount, using the second motion amount as an example. In one embodiment of this disclosure, the motion information of the image sensor can be generated based on either the first or the second motion amount. The motion manager can calculate the accurate motion amount of the image sensor.
[0072] In one embodiment of this disclosure, the motion manager may set a first range. The motion manager may set the first range based on a first movement amount (Δi, Δj). The motion manager may set the first range to be wider as the first movement amount (Δi, Δj) increases. In one embodiment of this disclosure, the first range may be a preset range. As the first range widens, additional movement amount (δ) of the second region of interest 321 is detected. i δ j The computational load required can be increased. To improve image processing speed, the first range can be a fixed range.
[0073] Figure 4 This is a diagram illustrating a temporal noise removal method according to an embodiment of the present disclosure.
[0074] refer to Figure 4 The noise remover can remove temporal and spatial noise contained in the first frame image 410 based on the first frame image 410 and the second frame image 420. The noise remover can take into account the movement of the image sensor to remove temporal and spatial noise.
[0075] The first frame image 410 may include a first region of interest 411. A noise remover can remove spatial noise contained in the first frame image 410. The noise remover can apply a bilateral filter to the first region of interest 411. The coefficients for the bilateral filter can be determined based on the similarity between pixels. In one embodiment of this disclosure, a Gaussian filter can be strongly applied to the first region of interest as the similarity between pixels increases. The similarity between pixels can change based on the difference between pixel values or the difference between pixel positions.
[0076] exist Figure 4 In the first region of interest 411, the center pixel can be (c i ,c j With the center pixel (c) of the first region of interest 411 i ,c j As the difference between the pixel value of the first region of interest (411) and the pixel values of the other pixels decreases, the weight to be applied to the other pixels can be increased. With the center pixel (c) of the first region of interest 411... i ,c j As the distance between the pixel and the rest of the pixel decreases, the weight to be applied to the rest of the pixel can be increased.
[0077] In one embodiment of this disclosure, the weight values to be applied to the first region of interest 411 can reflect both the difference between pixel values and the distance between pixels. For example, a weight that is larger than the weights of other pixels can be applied to pixels with smaller differences between pixel values and closer to the center pixel (c). i ,c jThe Gaussian filter can be applied very strongly as the weights increase. As the strength of the Gaussian filter increases, the pixel to which the Gaussian filter is applied can become similar to its neighboring pixels.
[0078] In one embodiment of this disclosure, the first frame image may include multiple regions of interest (ROIs). A noise remover can remove spatial noise by applying a bilateral filter to the multiple ROIs. Spatial noise may occur at the same location in different frames. The noise remover can remove spatial noise by applying a bilateral filter.
[0079] exist Figure 4 In this image processing device, the first frame image 410 can be an image that has undergone a similarity mapping process with the application of a bilateral filter to remove spatial noise. The second frame image 420 can be an image with temporal and spatial noise removed. The second frame image 420 can be an image received from the image sensor prior to the first frame image 410. The second frame image 420 can be stored in a memory inside or outside the image processing device.
[0080] The noise remover removes temporal noise by comparing the first frame image 410 with the second frame image 420. Temporal noise can be noise that appears at different locations in each frame. The noise remover can also remove temporal noise by combining consecutive frame images.
[0081] In one embodiment of this disclosure, the noise remover can generate a corrected image that includes the average pixel values of a first region of interest 411 and a moved second region of interest 422. The second frame image 420 may include a second region of interest 421 corresponding to the position of the first region of interest 411. When the image sensor is stationary, the corrected image can be generated by combining the first region of interest 411 with the second region of interest 421.
[0082] Based on the movement of the image sensor, the noise remover can move the second region of interest (ROI) according to the movement information of the image sensor. The position of the center pixel of the moved ROI can be determined by (c i +Δi′,c j It can be represented by +Δj′).
[0083] The motion manager can calculate the movement (Δi′, Δj′) of the second region of interest. The movement (Δi′, Δj′) can be either the movement (Δi, Δj) calculated via the inertial sensor, or an additional movement (δ) where the sum of the absolute differences between the movement (Δi, Δj) calculated via the inertial sensor and the pixel values of the region of interest is minimized. i δ j The sum of (Δi+δ) i ,Δj+δ jThe movement (Δi′, Δj′) of the second region of interest calculated by the motion manager can be compared with... Figure 3 The description corresponds to that.
[0084] The noise remover can remove noise by using the center pixel (c i c j The first region of interest 411 and the region with the center pixel (c) i +Δi′,c j The corrected image is generated by combining the second region of interest 422 of +Δj′. The noise remover can generate the corrected image using the following formula.
[0085]
[0086] The noise remover can calculate the average of the pixel values of the first region of interest 411 to which the bilateral filter is applied and the pixel values of the second region of interest 422 to which the image sensor's motion information is shifted, and can generate a corrected image including the average value.
[0087] Figure 5 This is a flowchart illustrating a temporal and spatial noise removal method according to an embodiment of the present disclosure.
[0088] refer to Figure 5 The image processing device can remove temporal and spatial noise contained in the sensed image by taking into account the movement of the image sensor. Instead of comparing all the frame images to detect the movement of the image sensor, the image processing device can calculate motion information related to the movement of the image sensor.
[0089] In operation S510, the motion manager can generate motion information of the image sensor based on the pixel values of a first frame image received from the image sensor, the pixel values of a second frame image received from the image sensor before the first frame image, and the velocity information of the inertial sensor. The motion manager can calculate the amount of motion of the image sensor based on the velocity information of the inertial sensor. The motion manager can correct the amount of motion of the image sensor calculated based on the received pixel values.
[0090] In operation S520, the similarity map generator can generate a similarity map indicating the weights of the noise removal filter based on the pixel values of the first frame image. The similarity map generator can determine the weights based on at least one of the differences between pixel values in the first frame image and the differences between pixel positions in the first frame image. In one embodiment of this disclosure, the noise removal filter may be a bilateral filter.
[0091] The similarity map generator can increase its weight as the difference between the pixel values of the center pixel and its neighboring pixels in the first region of interest decreases. The similarity map generator can also increase its weight as the center pixel and its neighboring pixels in the first region of interest become closer to each other.
[0092] In operation S530, the noise remover can remove temporal and spatial noise contained in the first frame image based on the pixel values of the first frame image to which a noise removal filter is applied according to a similarity map and the pixel values of the second frame image reflecting the motion information of the image sensor. The noise remover can remove spatial noise by applying a bilateral filter to the first frame image based on the similarity map. The noise remover can remove temporal noise by combining the first frame image with the second frame image, which is moved in parallel according to the motion information of the image sensor.
[0093] Figure 5 The corresponding operations in can be compared with Figures 1 to 4 The description corresponds to that.
[0094] Figure 6 This is a flowchart illustrating a method for generating motion information of an image sensor according to an embodiment of the present disclosure.
[0095] refer to Figure 6 The motion manager can calculate the amount of movement of the image sensor. The motion manager can correct the image sensor movement calculated based on velocity information from the inertial sensor using pixel values.
[0096] In operation S610, the motion manager can receive velocity information from the inertial sensor and calculate a first movement of the image sensor based on the received velocity information. The velocity information from the inertial sensor can be information related to the angular velocity of the inertial sensor. The motion manager can calculate the Euler change based on the information related to the angular velocity and can calculate the first movement by integrating the calculated Euler change in the time domain.
[0097] The motion manager can calculate the initial movement without comparing pixel values from consecutive frames. The motion manager can reduce the computational load required to detect image sensor movement by calculating the initial movement based on velocity information from the inertial sensor.
[0098] In operation S620, the motion manager can calculate the sum of the absolute differences between the pixel values of a first region of interest in the first frame and a second region of interest in the second frame. The motion manager can move the second region of interest from the position corresponding to the first region of interest by a first movement amount. The motion manager can compare the sum of the absolute differences (SAD) between the pixel values of the first and second regions of interest with a preset reference value.
[0099] In operation S630, when the sum of the absolute differences between pixel values is equal to or greater than a preset reference value, the motion manager can detect a second movement amount that minimizes the sum of the absolute differences between pixel values. The motion manager can calculate the sum of the absolute differences between pixel values when moving a second region of interest within a first range. The first range can be a range that changes according to the first movement amount or a preset range.
[0100] In operation S640, the motion manager can generate motion information of the image sensor based on a first motion amount or a second motion amount. When the sum of the absolute differences between the pixel values of the first region of interest and the second region of interest is less than a preset reference value, the motion manager can generate motion information of the image sensor including the first motion amount. In one embodiment of this disclosure, when the sum of the absolute differences between the pixel values of the first region of interest and the second region of interest is less than the preset reference value, the motion manager can correct the motion information of the image sensor generated based on the first motion amount based on the second motion amount. The motion manager can accurately detect the motion amount of the image sensor.
[0101] Figure 6 The corresponding operations in can be compared with Figure 3 The description corresponds to that.
[0102] Figure 7 This is a flowchart illustrating a method for generating a modified image according to an embodiment of the present disclosure.
[0103] refer to Figure 7 The noise remover can remove temporal and spatial noise contained in the first frame of the image and generate a corrected image. The noise remover can take into account the movement of the image sensor to remove temporal and spatial noise.
[0104] In operation of S710, the noise remover can remove spatial noise contained in the first frame image based on the pixel values of the first frame image. The noise remover can apply a bilateral filter to the first frame image. The noise remover can change the strength of the Gaussian filter to be applied based on the similarity between the images to which the bilateral filter is applied. The noise remover can generate a first frame image with spatial noise removed.
[0105] In operation S720, the noise remover can determine the location of a second region of interest in a second frame image to be combined with a first region of interest in the first frame image. The noise remover can set the location of the second region of interest based on motion information from the image sensor. The noise remover can move the second region of interest by a first movement amount or a second movement amount based on the motion information from the image sensor.
[0106] In operation S730, the noise remover can combine a first region of interest (ROI) in a first frame image (with spatial noise removed) with a second ROI in a moved second frame image. The noise remover can generate a corrected image combining the first and second ROIs. The noise remover can also generate a corrected image that includes the average of the pixel values of the first and second ROIs.
[0107] In one embodiment of this disclosure, the noise remover can calculate a weighted sum of pixel values in a first region of interest and a second region of interest. The noise remover can also calculate a weighted sum of pixel values from a first frame and multiple images. The noise remover can assign higher weights to more recently generated images.
[0108] Figure 8 This is a block diagram illustrating an electronic device including an image processing apparatus according to an embodiment of the present disclosure.
[0109] refer to Figure 8 The electronic device 2000 may include an image sensor 2010, a processor 2020, a storage device 2030, a storage device 2040, an input device 2050, and an output device 2060. Although Figure 8 Not shown, but the electronic device 2000 may also include a port capable of communicating with a video card, sound card, memory card or USB device or with other electronic devices.
[0110] Image sensor 2010 can generate image data corresponding to incident light. Output device 2060 can display the image data. Storage device 2030 can store the image data. Processor 2020 can control the operation of image sensor 2010, storage device 2030 and output device 2060.
[0111] In one embodiment of this disclosure, the image sensor 2010 may include an inertial sensor for sensing the motion (movement) of the image sensor 2010. The inertial sensor may output velocity information relating to the angular velocity or rotational velocity in the direction of movement.
[0112] Processor 2020 can perform specific calculations or tasks. Processor 2020 can perform and control the operation of electronic device 2000. According to one embodiment of this disclosure, processor 2020 may be a microprocessor, a central processing unit (CPU), or an application processor (AP). Processor 2020 may be coupled to storage device 2030, storage device 2040, input device 2050, and output device 2060 via address bus, control bus, and data bus, and may then communicate with said devices. According to one embodiment of this disclosure, processor 2020 may also be coupled to an expansion bus such as a peripheral component interconnect (PCI) bus.
[0113] In one embodiment of this disclosure, the processor 2020 can generate motion information of the image sensor 2010 based on the velocity information of the inertial sensor. The processor 2020 can correct the motion information of the image sensor based on pixel values. The corrected motion information can accurately represent the movement of the image sensor.
[0114] The processor 2020 can determine the weights of a bilateral filter used for removing spatial noise based on pixel values. The processor 2020 can generate a corrected image that removes temporal and spatial noise by combining the pixel values of the current frame image to which the bilateral filter is applied with the pixel values of the previous frame image that reflects motion information from the image sensor.
[0115] Storage device 2030 may include all types of non-volatile storage devices, including flash memory devices, solid-state drives (SSDs), hard disk drives (HDDs), and CD-ROMs.
[0116] Storage device 2040 can store data required for the operation of electronic device 2000. For example, storage device 2040 may include volatile storage devices such as dynamic random access memory (DRAM) or static random access memory (SRAM), and non-volatile storage devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Processor 2020 can control image sensor 2010 and output device 2060 by executing a set of instructions stored in storage device 2040.
[0117] Input device 2050 may include input tools such as a keyboard, keypad, or mouse. Output device 2060 may include output tools such as a printer or monitor.
[0118] The image sensor 2010 can be coupled to the processor 2020 via a bus or other communication link for performing communication.
[0119] The image sensor 2010 can be implemented in a variety of packages. For example, at least some components of the image sensor 2010 can be implemented using any of the following packages: package-on-package (PoP), ball grid array (BGA), chip-scale package (CSP), plastic leaded chip carrier (PLCC), plastic dual in-line package (PDIP), die-in-waffle, die-on-chip, chip-on-board (COB), ceramic dual in-line package (CERDIP), plastic metric quad flat package (MQFP), thin quad flat package (TQFP), small outline integrated circuit (SOIC), shrink small outline package (SSOP), thin small outline package (TSOP), system package (SIP), multi-chip package (MCP), wafer-level fabricated package (WFP), and wafer-level processed stacked package (WSP).
[0120] According to one embodiment, the image sensor 2010 and the processor 2020 can be integrated into a single chip or can be integrated into different chips.
[0121] Furthermore, electronic device 2000 should be interpreted as any computing system among all computing systems using image sensor 2010. For example, electronic device 2000 may include digital cameras, mobile phones, personal digital assistants (PDAs), portable multimedia players (PMPs), smartphones, etc.
[0122] According to this disclosure, an image processing apparatus can be provided that can quickly acquire motion information of an image sensor and remove noise while taking into account the motion of the image sensor.
[0123] It should be noted that the scope of this disclosure is defined by the appended claims rather than by the detailed description above, and all changes or modifications derived from the claims and their equivalents are included within the scope of this disclosure. Furthermore, embodiments may be combined to form other embodiments.
Claims
1. An image processing apparatus, comprising: A motion manager that generates motion information of the image sensor based on pixel values of a first frame image received from the image sensor, pixel values of a second frame image received from the image sensor before the first frame image, and velocity information from an inertial sensor that senses the velocity of the image sensor. A similarity map generator that: generates a similarity map indicating the weights of a noise removal filter based on the pixel values of the first frame image; and A noise remover that removes temporal and spatial noise contained in the first frame image based on pixel values of the first frame image to which the noise removal filter is applied according to the similarity map and pixel values of the second frame image reflecting the motion information. The motion manager generates the motion information by calculating a first motion amount through integration of the velocity information in the time domain. The motion manager generates the motion information based on whether the sum of the absolute differences between the pixel values of a first region of interest (ROI) in the first frame image and the pixel values of a second ROI in the second frame image is less than a preset reference value. The position of the second ROI is moved by the first movement amount from the position corresponding to the first ROI.
2. The image processing apparatus according to claim 1, wherein, The motion manager generates the motion information based on the first movement amount when the sum is less than the preset reference value.
3. The image processing apparatus according to claim 2, in, The motion manager: When the sum is equal to or greater than the preset reference value, a second shift amount is determined by moving the second ROI within the first range to minimize the sum. The movement information is generated based on the second movement amount.
4. The image processing apparatus according to claim 3, wherein, The first range depends on the first amount of movement.
5. The image processing apparatus according to claim 4, wherein, The first range widens as the first movement amount increases.
6. The image processing apparatus according to claim 3, in, The noise removal filter is a bilateral filter, and The similarity mapping generator determines the weight based on at least one of the differences between the position of a pixel in the first frame image and the pixel value of the first frame image.
7. The image processing apparatus according to claim 6, wherein, The similarity mapping generator generates the similarity mapping based on a weight that increases as the difference between the pixel values of the center pixel and its neighboring pixels in the first ROI decreases.
8. The image processing apparatus according to claim 7, wherein, The similarity mapping generator generates the similarity mapping based on a weight that increases as the positions of the center pixel and the adjacent pixels in the first ROI become closer to each other.
9. The image processing apparatus according to claim 6, wherein, The noise remover removes spatial noise contained in the first frame image based on the similarity mapping, and removes temporal noise contained in the first frame image based on the motion information.
10. The image processing apparatus according to claim 9, wherein, The noise remover calculates the average of the pixel values of the first frame image, which is applied to the bilateral filter based on the similarity map, and the average of the pixel values of the second frame image, which is moved in parallel based on the motion information, and generates a corrected image of the first frame image based on the average value.
11. The image processing apparatus according to claim 10, in, The noise remover removes the temporal noise by combining the second ROI with the first ROI in the first frame image after removing the spatial noise. The second ROI moves from the position corresponding to the first ROI by the first movement amount or the second movement amount.
12. An image processing method, comprising: Motion information of the image sensor is generated based on the pixel values of a first frame image received from the image sensor, the pixel values of a second frame image received from the image sensor before the first frame image, and the velocity information of the inertial sensor included in the image sensor. A similarity map indicating the weights of the noise removal filter is generated based on the pixel values of the first frame image; as well as Temporal and spatial noise contained in the first frame image is removed based on the pixel values of the first frame image to which the noise removal filter is applied according to the similarity mapping and the pixel values of the second frame image reflecting the motion information. The generation of the movement information includes: The first movement is calculated by integrating the velocity information in the time domain. Move the second ROI in the second frame image from the position corresponding to the first ROI in the first frame image by the first movement amount; and Calculate the sum of the absolute differences between the pixel values of the first ROI and the pixel values of the second ROI.
13. The image processing method according to claim 12, wherein, Generating the movement information further includes: generating the movement information based on the first movement amount when the sum is less than a preset reference value.
14. The image processing method according to claim 12, wherein, Generating the movement information also includes: When the sum is equal to or greater than a preset reference value, a second movement amount is detected that minimizes the sum; and The movement information is generated based on the second movement amount.
15. The image processing method according to claim 14, wherein, The noise removal filter is a bilateral filter, and Generating the similarity mapping includes determining the weight based on at least one of the differences between the position of a pixel in the first frame image and the pixel value of the first frame image.
16. The image processing method according to claim 15, wherein, Determining the weights includes: The weight is increased as the difference between the pixel values of the center pixel and its adjacent pixels in the first ROI decreases; and The weight increases as the center pixel and the adjacent pixels in the first ROI become closer to each other.
17. The image processing method according to claim 15, wherein, Removing the temporal and spatial noise includes: Based on the similarity mapping, the spatial noise contained in the first frame image is removed by applying the bilateral filter to the first frame image; and Temporal noise in the first frame image is removed by combining a first ROI in the first frame image where spatial noise has been removed and a second ROI in the second frame image where the motion information has been moved in parallel.
18. The image processing method according to claim 17, wherein, Removing the time noise includes: Move the second ROI by the second movement amount; and Generate a corrected image of the first frame image, the corrected image comprising the average of the pixel values of the first ROI with spatial noise removed and the pixel values of the moved second ROI.
Citation Information
Patent Citations
System and method for providing optimal model of porous structure for implant
KR1020220086997A
Image processing device, image processing method, image processing program, and recording medium
CN104995908A
