Image processing method and device, storage medium, communication equipment and chip
By performing demosaic interpolation and warping on the Bayer format images, an efficient intermediate image is generated, which solves the problem of difficulty in signal interpolation during image warping of Bayer format images, and achieves efficient and high-quality image processing effects.
Patent Information
- Application Number
- CN202311413744.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-27
- Publication Date
- 2025-04-29
AI Technical Summary
The signal interpolation of Bayer format images is difficult during image warping, resulting in serious loss of clarity and poor processing effect.
By performing demosaic interpolation processing on the Bayer format image, an intermediate image in RGB format is generated and interpolated into an image in the second format, reducing the calculation amount and improving efficiency. At the same time, the input image is combined with the intermediate image to determine the second output image, and the ability to remove pseudo-color is improved.
The calculation amount of image processing is reduced, the efficiency and quality of image processing are improved, and the ability to remove image pseudo-color is enhanced.
Smart Images
Figure CN119906902A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, apparatus, storage medium, communication equipment and chip. Background Art
[0002] With the popularization of smart terminal devices and the development of image processing, the formats of input images in image processing have become more diverse. Bayer format is a common image format and is usually used as an imaging image format for mobile phones, cameras and other devices.
[0003] The color channels of Bayer format images are arranged in a certain pattern, that is, each pixel position only samples one color channel among R, G and B. Compared with RGB format images (each pixel position samples three color channels among R, G and B), Bayer format images lack 2 / 3 of the information. When warping Bayer format images, that is, when performing translation, rotation, perspective, non-rigid deformation, etc., it is necessary to interpolate the image signal. Due to the low sampling rate of each color channel, signal interpolation is difficult, the interpolation result has serious loss of clarity, and the image processing effect is poor. Summary of the invention
[0004] The present disclosure provides an image processing method, apparatus, storage medium, communication equipment and chip to solve the problems in the related art, improve the signal sampling rate of each color channel in a Bayer format image, reduce the amount of calculation for image processing, and improve the image processing efficiency and image processing quality.
[0005] A first aspect embodiment of the present disclosure proposes an image processing method, which includes: determining a second area of an input image based on a first area of a first output image; performing a first processing on an image of the second area to determine a first intermediate image, wherein the image format of the first intermediate image is a first format; performing a second processing on the first intermediate image to determine a second intermediate image, wherein the image format of the second intermediate image is a second format, and the second format is different from the first format; and determining a second output image of the input image through a third processing based on the image of the second area and / or the second intermediate image.
[0006] In some embodiments, determining a second area of an input image based on a first area of a first output image includes: determining a first model, the first model being used to determine the first output image based on the input image; determining first pixel coordinates of the first area; determining second pixel coordinates of the second area based on the first model and the first pixel coordinates; and determining the second area based on the second pixel coordinates.
[0007] In some embodiments, performing a second processing on the first intermediate image to determine the second intermediate image includes: determining a first color channel of a first pixel point in the first area and a first pixel value of the first intermediate image; based on the first color channel and the first pixel value, performing a second processing on the first intermediate image to determine a second color channel and a second pixel value of a third pixel point of the second intermediate image, the first pixel point corresponding to the third pixel point.
[0008] In some embodiments, based on the image of the second area and / or the second intermediate image, determining the second output image of the input image through a third processing includes: performing a fifth processing on the image of the second area to obtain a third intermediate image; and determining the second output image based on the second intermediate image and / or the third intermediate image.
[0009] In some embodiments, based on the second intermediate image and / or the third intermediate image, determining the second output image includes: determining the third pixel coordinates of the second intermediate image, the fourth pixel coordinates of the third intermediate image and the third pixel value, the third pixel and the fourth pixel corresponding to each other; determining the second output image based on the third pixel coordinates, the second pixel value, the fourth pixel coordinates and the third pixel value.
[0010] In some embodiments, determining the second output image based on the third pixel coordinates, the second pixel value, the fourth pixel coordinates and the third pixel value includes: determining a first interpolation based on the third pixel coordinates and the fourth pixel coordinates; determining a first weight and a second weight based on the first interpolation and a preset threshold, wherein the first weight corresponds to the third pixel coordinates and the second pixel value, and the second weight corresponds to the fourth pixel coordinates and the third pixel value; determining the coordinates and pixel value of the fifth pixel based on the third pixel coordinates, the second pixel value, the fourth pixel coordinates, the third pixel value, the first weight and the second weight; determining the second output image based on the coordinates and pixel value of the fifth pixel.
[0011] A second aspect embodiment of the present disclosure proposes an image processing device, which includes: a first determination unit, used to determine a second area of an input image based on a first area of a first output image; a first processing unit, used to perform a first processing on an image of the second area to determine a first intermediate image, the image format of the first intermediate image being in a first format; a second processing unit, used to perform a second processing on the first intermediate image to determine a second intermediate image, the image format of the second intermediate image being in a second format, and the second format being different from the first format; and a third processing unit, used to determine a second output image of the input image through a third processing based on the image of the second area and / or the second intermediate image.
[0012] The third method embodiment of the present disclosure proposes a computer-readable storage medium, wherein when the computer program is executed by a processor, the method described in the first aspect embodiment of the present disclosure is performed.
[0013] The fourth aspect embodiment of the present disclosure proposes a communication device, including: a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the method described in the first aspect embodiment of the present disclosure when executing the computer program.
[0014] The fifth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of a communication device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the communication device executes the method described in the first aspect embodiment of the present disclosure.
[0015] In summary, according to the image processing method proposed in the present disclosure, the method includes: determining the second area of the input image based on the first area of the first output image; performing a first processing on the image of the second area to determine the first intermediate image, the image format of the first intermediate image is the first format; performing a second processing on the first intermediate image to determine the second intermediate image, the image format of the second intermediate image is the second format, and the second format is different from the first format; based on the image of the second area and / or the second intermediate image, through a third processing, determining the second output image of the input image. The scheme of the present disclosure realizes the direct interpolation processing of the first intermediate image in the first format into the second intermediate image in the second format through the second processing, which reduces the amount of image processing calculation and improves the image processing efficiency; at the same time, through the third processing, the input image is combined with the second intermediate image to determine the second output image, which improves the ability to remove image pseudo-color, thereby improving the image processing quality.
[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0018] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;
[0019] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;
[0020] Figure 3 An example diagram of an image processing method provided by an embodiment of the present disclosure;
[0021] Figure 4 A flowchart of an image processing method provided by an embodiment of the present disclosure;
[0022] Figure 5 A flowchart of an image processing method provided by an embodiment of the present disclosure;
[0023] Figure 6 An example diagram of an image processing method provided by an embodiment of the present disclosure;
[0024] Figure 7 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;
[0025] Figure 8 A schematic diagram of the structure of a communication device provided in an embodiment of the present disclosure;
[0026] Fig. 9 A schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0027] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0028] With the popularization of smart terminal devices and the development of the image processing field, the formats of input images during image processing have become more diverse. Among them, the Bayer format is a common image format and is usually used as an imaging image format for mobile phones, cameras and other devices.
[0029] The color channels of Bayer format images are arranged in a certain pattern, that is, each pixel position only samples one color channel among R, G and B. Compared with RGB format images (each pixel position samples three color channels among R, G and B), Bayer format images lack 2 / 3 of the information. When warping Bayer format images, that is, when performing translation, rotation, perspective, non-rigid deformation, etc., it is necessary to interpolate the image signal. Due to the low sampling rate of each color channel, signal interpolation is difficult, the interpolation result has serious loss of clarity, and the image processing effect is poor.
[0030] In the related art, a warping solution for Bayer format images is to obtain an RGB image by demosaicing the Bayer format image, and then warp the RGB format image. Since the demosaicing algorithm is easy to interpolate pseudo-color, and from the perspective of hardware architecture, the warping module needs to perform non-raster scanning data acquisition, and the demosaicing algorithm module is generally a raster scanning pixel processing, which is difficult to be placed before the warping module. Therefore, the scope of application and image processing effect of this method have certain limitations.
[0031] Another solution for warping Bayer format images is to process the Bayer format images by color channel. For example, to interpolate the pixel at the R color channel position, only the R pixel value of the input image (RGB format) is used for bilinear, bicubic, and other interpolations. This type of channel-by-channel processing algorithm interpolates each color channel independently, but a single color channel has only 1 / 4 of the resolution of the input image, resulting in severe signal loss. Details such as lines with a single pixel width will be directly lost, resulting in severe loss of clarity in the processed image.
[0032] Therefore, in order to solve the problems existing in the related art, the present disclosure proposes an image processing solution, which, through the second processing, directly interpolates the first intermediate image in the first format into the second intermediate image in the second format, thereby reducing the computational complexity of image processing and improving the image processing efficiency; at the same time, through the third processing, the input image is combined with the second intermediate image to determine the second output image, thereby improving the ability to remove image pseudo-color and thus improving the image processing quality.
[0033] Before introducing the detailed solution of the present disclosure, the scenario to which the solution of the present disclosure is applied is described first. Figure 1 FIG. 1 is an application scenario diagram of an image processing method in an embodiment. Figure 1 As shown, the application scenario includes an electronic device 104, in which a camera module and several applications can be installed. The application can initiate an image acquisition instruction to acquire an image, and the camera module acquires the image 102. Among them, the camera module can include a front camera module and / or a rear camera module. Finally, the target image is sent to the target application. Among them, the electronic device 104 can be a smart phone, a tablet computer, a personal digital assistant, a wearable device, etc.
[0034] In some optional embodiments, the electronic device may also be a vehicle-mounted device or a vehicle networking device, such as a smart car, etc. In the present disclosure, only a smart phone is taken as an example, but it does not limit the scope of the present disclosure.
[0035] A camera can be installed on an electronic device, and images can be acquired through the installed camera. Cameras can be divided into laser cameras, visible light cameras and other types according to the different images acquired. Laser cameras can acquire images formed by irradiating lasers onto objects, and visible light cameras can acquire images formed by irradiating visible light onto objects. Several cameras can be installed on an electronic device, and the installation locations are not limited. For example, a camera can be installed on the front panel of an electronic device, and two cameras can be installed on the back panel. The camera can also be installed inside the electronic device in an embedded manner, and then the camera can be opened by rotating or sliding. Specifically, a front camera and a rear camera can be installed on an electronic device. The front camera and the rear camera can acquire images from different perspectives. Generally, the front camera can acquire images from the front perspective of the electronic device, and the rear camera can acquire images from the back perspective of the electronic device.
[0036] It should be understood that in the present disclosure, the front camera or the rear camera is only used as an example to distinguish the shooting angles of different cameras, rather than to limit the functions of multiple cameras. The multiple cameras in the present disclosure can be rear cameras or front cameras at the same time, which is not limited in the present disclosure.
[0037] An electronic device can have several applications installed. An application is software written for a certain purpose in an electronic device. An electronic device can use an application to provide services to users. When an application needs to capture an image, it will initiate an image capture instruction. The electronic device will call the camera module to capture the image according to the image capture instruction. An image capture instruction is an instruction used to trigger an image capture operation.
[0038] The electronic device is also provided with a processor, and the image processing module in the processor can perform image processing on the image captured by the camera module, for example, execute the image processing method provided in the present disclosure.
[0039] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure. The method can be applied to Figure 1 The application scenario shown is, for example, executed by a terminal with an integrated image processing function or an image processor in the terminal, and the present disclosure is not limited thereto. Figure 2 As shown, Figure 2 The following steps may be included.
[0040] Step 201: determine a second area of an input image based on a first area of a first output image.
[0041] In some embodiments, the first output image may be a processed image generated from the input image before the second region image is subjected to the first processing.
[0042] In some embodiments, the first area can be determined based on the capability of the device executing the present method, that is, the larger the capability of the device, the larger the first area; the first area can also be determined based on the area where image processing is required, for example, if only the portrait area is processed, the portrait area can be determined as the first area, but the present disclosure does not limit the method for determining the first area.
[0043] In some embodiments, a second area of the input image is determined based on a first area of the first output image, so as to perform a first process based on an image of the second area.
[0044] In some embodiments, the image format of the first output image is a second format, wherein the second format may be a Bayer format.
[0045] In some embodiments, the format of the input image may be in the second format.
[0046] In some embodiments, the second area may be larger than or equal to the first area. In other words, the second area at least includes a corresponding area of the first area in the input image.
[0047] In some embodiments, the coordinate system of the first region and the second region is the same, and the pixel coordinates of the second region may be determined by warping the pixel coordinates of the first region.
[0048] Step 202: Perform a first process on the image of the second area to determine a first intermediate image.
[0049] In some embodiments, a first process is performed on the image of the second area to perform demosaic interpolation on the second area, thereby generating a first intermediate image, wherein the image format of the first intermediate image is a first format.
[0050] In some embodiments, the first format is an RGB format.
[0051] In some embodiments, the first processing is a de-mosaicing process, which can be implemented by an algorithm such as a Harris-Stephens corner detection algorithm and a Gradient Based Threshold Free (GBTF) algorithm.
[0052] Optionally, in some embodiments, before performing the first processing on the image of the second area, preprocessing may also be performed, wherein the preprocessing may include black level correction (BLC), white balance correction (WBC), denoising and other processing to enhance the image quality of the second area and reduce the possibility of false color when interpolating the graphics of the second area.
[0053] Step 203: Perform a second process on the first intermediate image to determine a second intermediate image.
[0054] In some embodiments, a second processing is performed on the first intermediate image to perform warping interpolation on the first intermediate image, thereby directly interpolating the first intermediate image in the first format into a second intermediate image in the second format, thereby reducing the computational complexity of image processing and improving image processing efficiency.
[0055] In some embodiments, the third pixel point coordinates and the second pixel value of the second intermediate image can be determined by a second process to determine the second intermediate image based on the third pixel point coordinates and the second pixel value, wherein the second pixel value is the pixel value of the third pixel point.
[0056] In some embodiments, the image format of the second intermediate image is in the second format.
[0057] In some embodiments, performing a second process on the first intermediate image to determine the second intermediate image includes: determining a first color channel of a first pixel point in the first region and a first pixel value of the first intermediate image; performing a fourth process on the first intermediate image based on the first color channel and the first pixel value to determine a second color channel and a second pixel value of a third pixel point in the second intermediate image, the first pixel point corresponding to the third pixel point, and the first pixel value corresponding to the first color channel. In other words, the second process includes determining the second color channel and the fourth process.
[0058] That is, the second color channel of the third pixel of the second intermediate image can be determined based on the first color channel of the first pixel, wherein the second color channel is the same as the first color channel. For example, if the determined first color channel is the R channel, then the second color channel is the R channel.
[0059] That is, the second pixel value of the second pixel point of the second intermediate image can be determined by the fourth algorithm through the determined second color channel and the first pixel value of the first intermediate image. For example, if the second color channel of the determined second pixel point is the R channel, the second pixel value can be determined based on the pixel value of the R channel pixel component in the first intermediate image through the warping interpolation algorithm.
[0060] In some embodiments, based on the second processing, all pixels of the first intermediate image are traversed to generate a second intermediate image.
[0061] In some embodiments, the fourth processing is pixel interpolation processing, such as bilinear pixel interpolation, trilinear pixel interpolation, and other processing methods. The present disclosure does not limit the processing method of the fourth processing.
[0062] Step 204: Determine a second output image of the input image through a third process based on the image of the second area and / or the second intermediate image.
[0063] In some embodiments, based on the image of the second area and / or the second intermediate image, a second output image of the input image is determined through a third process, thereby improving the ability to remove image pseudo-color by combining the input image with the second intermediate image to determine the second output image.
[0064] In some embodiments, based on the image of the second area and / or the second intermediate image, determining the second output image of the input image through a third processing includes: performing a fifth processing on the image of the second area to obtain a third intermediate image; and determining the second output image based on the second intermediate image and / or the third intermediate image.
[0065] In some embodiments, the third intermediate image is the same size as the first intermediate image.
[0066] In some embodiments, the fifth processing is pixel interpolation processing, such as bilinear pixel interpolation, trilinear pixel interpolation, and other processing methods. The present disclosure does not limit the processing method of the fifth processing.
[0067] It should be understood that the specific processing methods of the fourth processing and the fifth processing may be the same or different, and the present disclosure does not limit this.
[0068] In some embodiments, the fifth processing is performed on the image of the second area to determine the fourth pixel coordinates and the third pixel value of the third intermediate image, and then the third intermediate image is determined based on the fourth pixel coordinates and the third pixel value, the third pixel value is the pixel value of the fourth pixel, and the fourth pixel coordinates are the coordinates of the fourth pixel.
[0069] In some embodiments, the second output image is determined by assigning weights to the second intermediate image and / or the third intermediate image and by weighted summation, wherein the weight corresponding to the first intermediate image or the weight corresponding to the third intermediate image may be 0.
[0070] Optionally, in some embodiments, the device executing the present solution has a specific preprocessing module, or in subsequent image processing, when it is desired to obtain a third intermediate image generated without the above-mentioned preprocessing, the image of the second area can be post-processed before the fifth processing is performed on the image of the second area, wherein the post-processing corresponds to the preprocessing in step 202, that is, the post-processing is the inverse of the preprocessing.
[0071] In summary, according to the image processing method proposed in the present disclosure, the method includes: determining the second area of the input image based on the first area of the first output image; performing a first processing on the image of the second area to determine the first intermediate image, the image format of the first intermediate image is the first format; performing a second processing on the first intermediate image to determine the second intermediate image, the image format of the second intermediate image is the second format, and the second format is different from the first format; based on the image of the second area and / or the second intermediate image, through a third processing, determining the second output image of the input image. The scheme of the present disclosure realizes the direct interpolation processing of the first intermediate image in the first format into the second intermediate image in the second format through the second processing, which reduces the computational amount of image processing, improves the image processing efficiency, and makes the scheme easy to implement in hardware; at the same time, through the third processing, the input image is combined with the second intermediate image to determine the second output image, which improves the ability to remove image pseudo-color.
[0072] Figure 3 The following further shows a flowchart of an image processing method proposed in the present disclosure. Figure 2 The embodiment shown further explains step 201. Figure 3 The following steps may be included.
[0073] Step 301, determine the first model.
[0074] In some embodiments, the first model is used to determine the first output image based on the input image.
[0075] In some embodiments, the first model may be at least one of an affine transformation model, a transmission transformation model, an optical flow model, and the like, and the present disclosure does not limit the first model.
[0076] Step 302: determine the coordinates of a first pixel point in a first area.
[0077] In some embodiments, the coordinates of a first pixel point in the first area are determined to determine the coordinates of a second pixel point.
[0078] In some embodiments, the coordinates of the first pixel point in the first area may be determined by a coordinate system of the first area.
[0079] Step 303: Determine the second pixel coordinates of the second area based on the first model and the first pixel coordinates.
[0080] For example, the first model is a transmission transformation model. Let the 3x3 homography matrix be H and the coordinates of the first pixel be P. out =[x o ,y o ,1] T , then the coordinates of the second pixel point are P in =[xi ,y i ,1] T =HP out .
[0081] Step 304: determine a second area based on the second pixel coordinates.
[0082] In some embodiments, the area formed by the second pixel point coordinates may be determined as the second area, and the area beyond the second pixel point coordinates may be determined as the second area.
[0083] In other words, the second area at least includes the second pixel coordinates. For example, if the area formed by the second pixel coordinates is a circular area with the coordinates (0, 0) as the center and 5 pixels as the radius, then the second area can be a circular area with the coordinates (0, 0) as the center and 5 pixels as the radius, or a circular area with the coordinates (0, 0) as the center and 6 pixels as the radius.
[0084] In summary, according to the image processing method proposed in the present disclosure, the method includes: determining a first model, the first model is used to determine a first output image based on an input image; determining the coordinates of a first pixel point in a first area; determining the coordinates of a second pixel point in a second area based on the first model and the coordinates of the first pixel point; and determining the second area based on the second pixel point coordinates. The method of the present disclosure determines the second area by determining the first model and the coordinates of the first pixel point, laying a foundation for implementing the solution of the present disclosure.
[0085] Figure 4 The following further shows a flowchart of an image processing method proposed in the present disclosure. Figure 2 The embodiment shown further explains the determination of the second output image based on the second intermediate image and / or the third intermediate image. Figure 4 The following steps may be included.
[0086] Step 401, determining the coordinates of a third pixel point of the second intermediate image, the coordinates of a fourth pixel point of the third intermediate image, and a third pixel value.
[0087] In some embodiments, the coordinates of the third pixel point of the second intermediate image can be determined by the coordinate system of the second intermediate image, or the coordinates of the third pixel point can be calculated and determined when the second processing is performed on the first intermediate image. The present disclosure does not limit the method for determining the coordinates of the third pixel point of the second intermediate image.
[0088] In some embodiments, the coordinates of the fourth pixel point of the third intermediate image can be determined by the coordinate system of the third intermediate image, or the coordinates of the fourth pixel point can be calculated and determined when the fifth processing is performed on the second intermediate image. The present disclosure does not limit the method for determining the coordinates of the fourth pixel point of the third intermediate image.
[0089] In some embodiments, the third pixel value may be calculated and determined when the fifth processing is performed on the image of the second area.
[0090] Step 402: determine a second output image based on the third pixel point coordinates, the second pixel value, the fourth pixel point coordinates and the third pixel value.
[0091] In some embodiments, the first weight of the third pixel point coordinates and the second pixel value can be determined, the second weight of the fourth pixel point coordinates and the third pixel value can be determined, the coordinates and pixel value of the fifth pixel point can be determined by weighted summation, and then based on the coordinates and pixel value of the fifth pixel point, the second output image can be determined, wherein the second output image is an image composed of the coordinates and pixel value of the fifth pixel point.
[0092] In summary, the image processing method proposed in the present disclosure includes: determining the third pixel coordinates, the second pixel value, the fourth pixel coordinates and the third pixel value of the second intermediate image, and the third pixel corresponds to the fourth pixel; and determining the second output image based on the third pixel coordinates, the second pixel value, the fourth pixel coordinates and the third pixel value. The solution of the present disclosure improves the ability to remove pseudo-color in an image by combining the input image with the second intermediate image to determine the second output image.
[0093] Figure 5 The following further shows a flowchart of an image processing method proposed in the present disclosure. Figure 4 The embodiment shown further explains step 402. Figure 5 The following steps may be included.
[0094] Step 501: determine a first interpolation value based on the third pixel point coordinates and the fourth pixel point coordinates.
[0095] In some embodiments, the first interpolation value is an absolute value of a coordinate difference between the third pixel point coordinate and the fourth pixel point coordinate.
[0096] The first interpolation value can be determined by the following formula:
[0097] d(i,j)=|p c (i,j)-p d (i,j)| (Formula 1)
[0098] Among them, d(i,j) represents the first interpolation, p c (i,j) represents the coordinates of the fourth pixel, p d (i, j) represents the coordinates of the third pixel point.
[0099] Step 502: Determine a first weight and a second weight based on a first interpolation value and a preset threshold.
[0100] In some embodiments, the first weight and the second weight may be determined by presetting a threshold of the first interpolation, that is, a maximum value and a minimum value of the first interpolation.
[0101] In some embodiments, the first weight corresponds to the third pixel point coordinates and the second pixel value, and the second weight corresponds to the fourth pixel point coordinates and the third pixel value.
[0102] In some embodiments, when the first interpolation is greater than the maximum value of a preset threshold, it can be determined that the second intermediate image has image pseudo color, and the first weight is 0 and the second weight is 1; when the first interpolation is less than the minimum value of the preset threshold, it can be determined that the second intermediate image does not have image pseudo color, and the first weight is 1 and the second weight is 0.
[0103] In some embodiments, when the first interpolation value is within a preset threshold, the first weight and the second weight may be determined by the following formula:
[0104] w(i,j)=(d(i,j)-th0) / (th1-th0) (Formula 2)
[0105] Wherein, w(i,j) represents the second weight, 1-w(i,j) represents the first weight, th1 represents the maximum value of the preset threshold, and th0 represents the minimum value of the preset threshold. However, it is not limited thereto, and the first weight and the second weight may also be determined by other methods.
[0106] Optionally, in some embodiments, when the first weight and the second weight are in coordinate form, the real number form weight corresponding to the coordinate number weight can be determined through a preset lookup table to determine the pixel value of the fifth pixel point based on the first weight and the second weight in real number form.
[0107] Step 503, determining the coordinates and pixel value of the fifth pixel point based on the third pixel point coordinates, the second pixel value, the fourth pixel point coordinates, the third pixel value, the first weight and the second weight.
[0108] In some embodiments, the coordinates of the fifth pixel point can be determined by weighted summation based on the third pixel point coordinates, the fourth pixel point coordinates, the first weight and the second weight. In other words, the coordinates obtained by weighted summation are the coordinates of the fifth pixel point.
[0109] In some embodiments, the pixel value of the fifth pixel point can be determined by weighted summation based on the second pixel value, the third pixel value, the first weight and the second weight. In other words, the pixel value obtained by weighted summation is the pixel value of the fifth pixel point.
[0110] Step 504: determine a second output image based on the coordinates and pixel value of the fifth pixel point.
[0111] In some embodiments, the image formed by the coordinates and pixel values of the fifth pixel point is the second output image.
[0112] In summary, the image processing method proposed in the present disclosure includes: determining a first interpolation based on the third pixel coordinates and the fourth pixel coordinates; determining a first weight and a second weight based on the first interpolation and a preset threshold, wherein the first weight corresponds to the third pixel coordinates and the second pixel value, and the second weight corresponds to the fourth pixel coordinates and the third pixel value; determining the coordinates and pixel value of the fifth pixel based on the third pixel coordinates, the second pixel value, the fourth pixel coordinates, the third pixel value, the first weight and the second weight; determining the second output image based on the coordinates and pixel value of the fifth pixel. The scheme of the present disclosure determines the first interpolation to determine the first weight and the second weight, and then combines the input image with the second intermediate image by weighted summation to determine the second output image, thereby improving the ability to remove image pseudo-color and improving the image processing quality.
[0113] Figure 6 An example diagram of an image processing process provided by an embodiment of the present disclosure is shown in FIG. Figure 6 The following steps may be included.
[0114] 1. Coordinate mapping and input window determination.
[0115] In some embodiments, an M*M output block is taken and mapped to an input image coordinate system according to a transformation model, wherein the image format of the input image is a Bayer format.
[0116] In some embodiments, a local N*N input window may be taken based on the mapping position of the output block.
[0117] In some embodiments, M may be any positive integer, and N is determined according to the windowing requirements of the interpolation process.
[0118] In some embodiments, the deformation model can be affine transformation, transmission transformation, optical flow, etc., which is not limited in this example. Taking transmission transformation as an example, let the 3x3 homography matrix be H and the output pixel coordinates be P out =[x o ,y o ,1] T , then the coordinates mapped to the input graph are P in =[x i ,y i ,1] T =HP out .
[0119] 2. Perform demosaic interpolation on the N*N input window to obtain an RGB image.
[0120] Optionally, in some embodiments, the input window is preprocessed. Before de-mosaicing and interpolation, the input window may be preprocessed by black level correction, white balance correction, denoising, etc. to enhance the image quality of the input window and reduce the possibility of interpolation of false colors.
[0121] In some embodiments, a demosaicing algorithm is used to interpolate an RGB image from the preprocessed input window, wherein the demosaicing algorithm may be any common scheme, such as the classic HA algorithm, GBTF algorithm, etc., without limitation.
[0122] 3. Perform warping interpolation.
[0123] In some embodiments, the RGB image result of the previous step is subjected to warping interpolation, that is, warping interpolation is performed based on the coordinate mapping result of step 1 and the RGB image of step 2 to obtain an interpolated image block of the Bayer format of size M*M.
[0124] Specifically, traverse each pixel of the image block of size M*M, determine the color channel (for example, the R channel) of the current output pixel, take the corresponding channel (for example, the R channel) in the RGB image of step 2, and perform pixel interpolation, wherein the pixel interpolation can be a common algorithm such as bilinear or bicubic, and this example does not limit this. This method can directly interpolate the RGB image of step 2 to obtain a Bayer format image block, compared to first performing pixel interpolation of all channels of the RGB image of step 2 (the interpolation result is an image in RGB format), and then downsampling it into a Bayer format image, saving 2 / 3 of the computational effort.
[0125] 4. Channel interpolation.
[0126] In some embodiments, pixel interpolation is performed on the N*N input window obtained in step 1 to obtain an M*M Bayer format image, wherein the pixel interpolation can be a common interpolation algorithm such as bilinear and bicubic, which is not limited in this example.
[0127] 5. False color determination and fusion.
[0128] Optionally, the M*M Bayer format image obtained in step 4 is post-processed, which is matched with step 2 to perform inverse WBC, inverse BLC and other operations.
[0129] In some embodiments, false color determination and pixel-by-pixel fusion are performed on the M*M Bayer format images in step 3 and step 4.
[0130] Specifically, the channel interpolation result of step 4 is recorded as B c ={p c (i,j)|i,j∈[0,M-1]}, let step 3 combined with demosaic interpolation obtain the result as Bd ={p d (i,j)|i,j∈[0,M-1]}. c and B d Fusion is performed pixel by pixel to obtain the final output image block B f ={p f (i,j)|i,j∈[0,M-1]}, where p f (i,j) can be determined by the following formula group,
[0131] d(i,j)=|p c (i,j)-p d (i,j)| (Formula 3)
[0132] d(i,j)=min (max(d(i,j),th0),th1) (Formula 4)
[0133] w(i,j)=(d(i,j)-th0) / (th1-th0) (Formula 5)
[0134] p f (i,j)=p c (i,j)*w(i,j)+p d (i,j)*(1-w(i,j)) (Formula 6)
[0135] In the above formula, th0 and th1 are the set thresholds. First, calculate B at (i, j) c and B d The absolute value of the pixel difference d(i,j) is calculated and set to the range [th0,th1], and then a weight value in the range [0,1] is calculated. c and B d Pixel weighting.
[0136] In some embodiments, when B c With B d When the difference is less than th0, the result based on demosaic interpolation is considered reliable. At this time, the fusion result is completely from B. d ; When B c With B d When the difference between th1 and th2 is greater than th1, it is considered that the result obtained based on demosaic interpolation has pseudo-color, and the channel interpolation result should be used. At this time, the fusion result can be completely derived from B c When the difference is between th0 and th1, the fusion result can be determined as the weighted result of the two, thereby achieving a smooth transition of pixels.
[0137] Therefore, this solution has the following beneficial effects:
[0138] 1. The solution disclosed herein performs a first process on the image of the second area to provide a de-mosaic interpolation processing capability, thereby improving the signal sampling rate of each color channel of the image of the second area and improving the clarity of the image processing result.
[0139] 2. The scheme disclosed in the present invention realizes direct interpolation processing of the first intermediate image in the first format into the second intermediate image in the second format through the second processing, which reduces the computational complexity of image processing, improves image processing efficiency, and is easy to implement in hardware.
[0140] 3. The solution disclosed in the present invention combines the input image with the second intermediate image through the third processing to determine the second output image, thereby improving the ability to remove false colors in the image.
[0141] Figure 7 The structure diagram of an image processing device 700 provided in an embodiment of the present disclosure includes:
[0142] A first determining unit 710, configured to determine a second area of the input image based on the first area of the first output image;
[0143] A first processing unit 720 is configured to perform a first processing on the image of the second area to determine a first intermediate image, where the image format of the first intermediate image is a first format;
[0144] A second processing unit 730 is configured to perform a second processing on the first intermediate image to determine a second intermediate image, where the image format of the second intermediate image is a second format, which is different from the first format;
[0145] The third processing unit 740 is configured to determine a second output image of the input image through a third processing based on the image of the second area and / or the second intermediate image.
[0146] In some embodiments, the first determination unit 710 is also used to determine a first model, and the first model is used to determine a first output image based on an input image; determine first pixel coordinates of a first area; determine second pixel coordinates of a second area based on the first model and the first pixel coordinates; and determine a second area based on the second pixel coordinates.
[0147] In some embodiments, the second processing unit 730 is also used to determine the first color channel of the first pixel point of the first area and the first pixel value of the first intermediate image; based on the first color channel and the first pixel value, perform second processing on the first intermediate image to determine the second color channel and the second pixel value of the third pixel point of the second intermediate image, and the first pixel point corresponds to the third pixel point.
[0148] In some embodiments, the third processing unit 740 is also used to perform a fifth processing on the image of the second area to obtain a third intermediate image, which is the same size as the first intermediate image; based on the second intermediate image and / or the third intermediate image, determine the second output image.
[0149] In some embodiments, the third processing unit 740 is also used to determine the third pixel coordinates, the second pixel value, the fourth pixel coordinates and the third pixel value of the second intermediate image, and the third pixel and the fourth pixel correspond to each other; and determine the second output image based on the third pixel coordinates, the second pixel value, the fourth pixel coordinates and the third pixel value.
[0150] In some embodiments, the third processing unit 740 also determines a first interpolation based on the third pixel coordinates and the fourth pixel coordinates; determines a first weight and a second weight based on the first interpolation and a preset threshold, wherein the first weight corresponds to the third pixel coordinates and the second pixel value, and the second weight corresponds to the fourth pixel coordinates and the third pixel value; determines the coordinates and pixel value of the fifth pixel based on the third pixel coordinates, the second pixel value, the fourth pixel coordinates, the third pixel value, the first weight and the second weight; determines the second output image based on the coordinates and pixel value of the fifth pixel.
[0151] According to the image processing device proposed in the present disclosure, based on the first area of the first output image, the second area of the input image is determined; the image of the second area is subjected to a first processing to determine a first intermediate image, the image format of the first intermediate image is the first format; the first intermediate image is subjected to a second processing to determine a second intermediate image, the image format of the second intermediate image is the second format, and the second format is different from the first format; based on the image of the second area and / or the second intermediate image, the second output image of the input image is determined through a third processing. The scheme of the present disclosure realizes direct interpolation processing of the first intermediate image in the first format into the second intermediate image in the second format through the second processing, thereby reducing the amount of image processing calculation and improving the image processing efficiency; at the same time, through the third processing, the input image is combined with the second intermediate image to determine the second output image, thereby improving the ability to remove image pseudo-color.
[0152] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.
[0153] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the communication device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.
[0154] Figure 8 800 is a schematic diagram of the structure of a communication device 800 provided in an embodiment of the present application. The communication device 800 may be a network device, or a terminal device, or a chip, a chip system, or a processor that supports the network device to implement the above method, or a chip, a chip system, or a processor that supports the terminal device to implement the above method. The device may be used to implement the method described in the above method embodiment, and the details may refer to the description in the above method embodiment.
[0155] The communication device 800 may include one or more processors 801. The processor 801 may be a general-purpose processor or a dedicated processor, etc. For example, it may be a baseband processor or a central processing unit. The baseband processor may be used to process the communication protocol and communication data, and the central processing unit may be used to control the communication device (such as a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute a computer program, and process the data of the computer program.
[0156] Optionally, the communication device 800 may further include one or more memories 802, on which a computer program 804 may be stored, and the processor 801 executes the computer program 804 so that the communication device 800 performs the method described in the above method embodiment. Optionally, data may also be stored in the memory 802. The communication device 800 and the memory 802 may be provided separately or integrated together.
[0157] Optionally, the communication device 800 may further include a transceiver 805 and an antenna 806. The transceiver 805 may be referred to as a transceiver unit, a transceiver, or a transceiver circuit, etc., and is used to implement a transceiver function. The transceiver 805 may include a receiver and a transmitter, the receiver may be referred to as a receiver or a receiving circuit, etc., and is used to implement a receiving function; the transmitter may be referred to as a transmitter or a transmitting circuit, etc., and is used to implement a transmitting function.
[0158] Optionally, the communication device 800 may further include one or more interface circuits 807. The interface circuit 807 is used to receive code instructions and transmit them to the processor 801. The processor 801 executes the code instructions to enable the communication device 800 to execute the method described in the above method embodiment.
[0159] In one implementation, the processor 801 may include a transceiver for implementing receiving and sending functions. For example, the transceiver may be a transceiver circuit, an interface, or an interface circuit. The transceiver circuit, interface, or interface circuit for implementing the receiving and sending functions may be separate or integrated. The above-mentioned transceiver circuit, interface, or interface circuit may be used for reading and writing code / data, or the above-mentioned transceiver circuit, interface, or interface circuit may be used for transmitting or delivering signals.
[0160] In one implementation, the processor 801 may store a computer program 803, which runs on the processor 801 and enables the communication device 800 to perform the method described in the above method embodiment. The computer program 803 may be fixed in the processor 801, in which case the processor 801 may be implemented by hardware.
[0161] In one implementation, the communication device 800 may include a circuit that can implement the functions of sending or receiving or communicating in the aforementioned method embodiments. The processor and transceiver described in the present application can be implemented in an integrated circuit (IC), an analog IC, a radio frequency integrated circuit RFIC, a mixed signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductor (CMOS), N-type metal oxide semiconductor (NMOS), P-type metal oxide semiconductor (positive channel metal oxide semiconductor, PMOS), bipolar junction transistor (BJT), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.
[0162] The communication device described in the above embodiments may be a network device or a terminal device, but the scope of the communication device described in this application is not limited thereto, and the structure of the communication device may not be limited thereto. Figure 8 The communication device may be a stand-alone device or may be part of a larger device. For example, the communication device may be:
[0163] (1) Independent integrated circuit IC, or chip, or chip system or subsystem;
[0164] (2) having a set of one or more ICs, and optionally, the IC set may also include a storage component for storing data and computer programs;
[0165] (3) ASIC, such as modem;
[0166] (4) Modules that can be embedded in other devices;
[0167] (5) Receivers, terminal devices, intelligent terminal devices, cellular phones, wireless devices, handheld devices, mobile units, vehicle-mounted devices, network devices, cloud devices, artificial intelligence devices, etc.;
[0168] (6)Others
[0169] For the case where the communication device may be a chip or a chip system, see Fig. 9 Schematic diagram of the chip structure shown.
[0170] The embodiment of the present disclosure also provides a chip, such as Fig. 9 The chip shown includes a processor 901 and an interface 902. The number of the processor 901 may be one or more, and the number of the interface 902 may be multiple.
[0171] Optionally, the chip further includes a memory 903, and the memory 903 is used to store necessary computer programs and data.
[0172] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the above embodiments of the present disclosure.
[0173] Those skilled in the art may also appreciate that the various illustrative logic blocks and steps listed in the embodiments of the present application may be implemented by electronic hardware, computer software, or a combination of the two. Whether such functions are implemented by hardware or software depends on the design requirements of the specific application and the entire system. Those skilled in the art may use various methods to implement the functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.
[0174] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0176] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0177] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0178] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0179] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0181] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. An image processing method, characterized in that: The method comprises: determining a second region of the input image based on the first region of the first output image; Performing a first process on the image of the second area to determine a first intermediate image, wherein the image format of the first intermediate image is a first format; performing a second processing on the first intermediate image to determine a second intermediate image, wherein the image format of the second intermediate image is a second format, and the second format is different from the first format; Based on the image of the second area and / or the second intermediate image, a second output image of the input image is determined through a third process.
2. The method according to claim 1, characterized in that The determining the second area of the input image based on the first area of the first output image comprises: determining a first model, the first model being used to determine a first output image based on the input image; Determining coordinates of a first pixel point in the first area; Determine second pixel coordinates of the second area based on the first model and the first pixel coordinates; The second area is determined based on the second pixel coordinates.
3. The method according to claim 1, characterized in that: The performing a second processing on the first intermediate image to determine a second intermediate image comprises: Determine a first color channel of a first pixel point in the first area and a first pixel value of the first intermediate image; The fourth processing is performed on the first intermediate image based on the first color channel and the first pixel value to determine a second color channel and a second pixel value of a third pixel point of the second intermediate image, and the first pixel point corresponds to the second pixel point.
4. The method according to claim 1, characterized in that: The determining, by a third process based on the image of the second area and / or the second intermediate image, a second output image of the input image comprises: performing fifth processing on the image of the second area to obtain a third intermediate image; Based on the second intermediate image and / or the third intermediate image, the second output image is determined.
5. The method according to claim 4, characterized in that The determining the second output image based on the second intermediate image and / or the third intermediate image comprises: Determine coordinates of a third pixel point of the second intermediate image, coordinates of a fourth pixel point of the third intermediate image, and a third pixel value, the third pixel point corresponding to the fourth pixel point; A second output image is determined based on the third pixel point coordinates, the second pixel value, the fourth pixel point coordinates and the third pixel value.
6. The method according to claim 5, characterized in that The determining of the second output image based on the third pixel point coordinates, the second pixel value, the fourth pixel point coordinates and the third pixel value comprises: Determine a first interpolation value based on the third pixel point coordinates and the fourth pixel point coordinates; Determine a first weight and a second weight based on the first interpolation and a preset threshold, wherein the first weight corresponds to the third pixel point coordinates and the second pixel value, and the second weight corresponds to the fourth pixel point coordinates and the third pixel value; Determine the coordinates and pixel value of a fifth pixel point based on the third pixel point coordinates, the second pixel value, the fourth pixel point coordinates, the third pixel value, the first weight, and the second weight; The second output image is determined based on the coordinates and the pixel value of the fifth pixel point.
7. An image processing device, characterized in that: The device comprises: A first determining unit, configured to determine a second area of the input image based on the first area of the first output image; A first processing unit is used to perform a first processing on the image of the second area to determine a first intermediate image, where the image format of the first intermediate image is a first format; a second processing unit, configured to perform a second processing on the first intermediate image to determine a second intermediate image, wherein the image format of the second intermediate image is a second format, and the second format is different from the first format; The third processing unit is used to determine a second output image of the input image through a third processing based on the image of the second area and / or the second intermediate image.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. A communication device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
10. A chip, characterized in that: The invention comprises one or more interfaces and one or more processors; the interface is used to receive a signal from a memory of a communication device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the communication device executes the method described in any one of claims 1 to 6.