Image processing method, apparatus and program product

Abstract

The application discloses an image processing method, device and program product method, and belongs to the computer field. The method comprises the following steps: acquiring a first image, the first image having a plurality of color channels; creating a second image according to the first image; and setting a pixel value of each pixel in the second image according to the first image and a target mapping relationship of each color channel in the plurality of color channels, the target mapping relationship being used for indicating corresponding source coordinates of a same pixel coordinate in different color channels of the first image. According to the target mapping relationship of each color channel, the refraction deviation caused by the wavelength difference can be eliminated while the first image is corrected, so that the image processing effect can be improved, and better image quality can be obtained. Moreover, the image correction can be quickly and accurately realized only through the target mapping relationship, so that the operation amount can be greatly reduced, and the image correction efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, and program product. Background Technology

[0002] With the continuous development of technology, near-eye display devices such as augmented reality (AR) and virtual reality (VR) have become widespread. To achieve thinness and a wide field of view, near-eye display devices typically employ complex imaging systems (such as freeform surface systems and waveguide systems). These imaging systems inevitably introduce optical distortion and dispersion, resulting in distorted images and rainbow effects in the image transmitted to the human eye. To address this, related technologies often perform monochrome geometric distortion correction on the image. However, this correction method often leaves residual lateral chromatic aberration in the corrected image, severely affecting image quality. Summary of the Invention

[0003] This application provides an image processing method, apparatus, and program product, which can improve image processing results and obtain better image quality. The technical solution is as follows: In a first aspect, an image processing method is provided, the method comprising: Acquire a first image, which has multiple color channels; A second image is created based on the first image, and both the second image and the first image are of a preset size; Based on the target mapping relationship between the first image and each color channel in the plurality of color channels, the pixel value of each pixel in the second image is set; wherein, the target mapping relationship of each color channel in the plurality of color channels is generated based on the distortion model corresponding to each color channel respectively, and the target mapping relationship is used to indicate the source coordinates corresponding to the same pixel coordinate in the second image in different color channels of the first image.

[0004] In this application, the target mapping relationship of each color channel can accurately reflect the source coordinates of each pixel coordinate in the second image corresponding to the first image. Furthermore, the target mapping relationships of different color channels can precisely match the actual color difference characteristics of the optical system. Therefore, by setting the pixel value of each pixel in the second image according to the first image and the multiple target mapping relationships of these multiple color channels, distortion correction of the first image can be achieved simultaneously with color difference correction, thus improving the image correction effect and obtaining better image quality. Moreover, the embodiments of this application can achieve image correction quickly and accurately using only the target mapping relationship, thereby greatly reducing the computational load and improving image correction efficiency. In scenarios requiring real-time image processing, computational overhead and latency can be reduced, thus better meeting the requirements of real-time rendering.

[0005] Optionally, the pixel value includes multiple color components that correspond one-to-one with the plurality of color channels, and the step of setting the pixel value of each pixel in the second image according to the target mapping relationship between the first image and each color channel in the plurality of color channels includes: The first source coordinates of the second pixel in the first image are obtained from the target mapping relationship of the target color channel, wherein the target color channel is any one of the plurality of color channels, and the second pixel is any pixel in the second image; n first pixels are determined from the first image based on the first source coordinates, where n is a positive integer; Based on the target color component in the pixel value of each of the n first pixels, the target color component in the pixel value of the second pixel is set, wherein the target color component is the color component corresponding to the target color channel.

[0006] Optionally, determining n first pixels from the first image based on the first source coordinates includes: If the first source coordinates belong to the image coordinate range of the first image, the n first pixels are determined from the first image based on the first source coordinates; The method further includes: If the first source coordinates do not belong to the image coordinate range of the first image, the target color component in the pixel value of the second pixel is set to a preset value.

[0007] Optionally, determining n first pixels from the first image based on the first source coordinates includes: If the first source coordinates are the same as the pixel coordinates of a pixel in the first image, the pixel in the first image whose pixel coordinates are the first source coordinates is determined as the first pixel; When the first source coordinates are different from the pixel coordinates of all pixels in the first image, multiple neighboring pixels in the first image that are adjacent to the first source coordinates are all determined as the first pixel.

[0008] Optionally, setting the target color component in the pixel value of the second pixel based on the target color component in the pixel value of each of the n first pixels includes: When n is 1, the target color component in the pixel values ​​of the n first pixels is determined as the target color component in the pixel values ​​of the second pixel; When n is greater than or equal to 2, linear interpolation is performed on the target color component in the pixel value of the n first pixels based on the first source coordinates and the pixel coordinates of each of the n first pixels to obtain the target color component in the pixel value of the second pixel.

[0009] Optionally, before setting the pixel value of each pixel in the second image according to the target mapping relationship between the first image and each color channel in the plurality of color channels, the method further includes: Normalize multiple pixel coordinates to obtain multiple normalized coordinates that correspond one-to-one with the multiple pixel coordinates, where the multiple pixel coordinates are all pixel coordinates in the image of the preset size; For any one of the multiple normalized coordinates, a distortion factor is determined based on the normalized coordinate and the distortion model of the target color channel, wherein the target color channel is any one of the multiple color channels. The source coordinates corresponding to the normalized coordinates are obtained based on the distortion factor and the normalized coordinates; The pixel coordinates and source coordinates corresponding to the normalized coordinates are added to the target mapping relationship of the target color channel.

[0010] Optionally, obtaining the source coordinates corresponding to the normalized coordinates based on the distortion factor and the normalized coordinates includes: Multiplying the distortion factor by the normalized coordinates yields inverse normalized coordinates, resulting in the source coordinates corresponding to the normalized coordinates; or... The coordinates obtained by multiplying the distortion factor by the normalized coordinates are inversely normalized to obtain the mapped coordinates; the scaling factor is multiplied by the mapped coordinates to obtain the source coordinates corresponding to the normalized coordinates. The scaling factor is determined based on the distortion coefficients and image center coordinates in the distortion model of the target color channel.

[0011] Optionally, the method further includes: The scaling factor is determined using the following formula based on the distortion coefficients in the distortion model of the target color channel and the coordinates of the image center;

[0012]

[0013]

[0014]

[0015] The The distortion coefficient is the value of the distortion coefficient. The x-coordinate value of the center coordinate of the image, the The ordinate value of the center coordinate of the image, The scaling factor is the coefficient component along the horizontal axis. The scaling factor is the coefficient component along the vertical axis.

[0016] Secondly, an image processing apparatus is provided, the apparatus comprising: A first acquisition module is used to acquire a first image, the first image having multiple color channels; A creation module is used to create a second image based on the first image, wherein both the second image and the first image are preset sizes; The setting module is used to set the pixel value of each pixel in the second image according to the target mapping relationship of each color channel in the first image and the plurality of color channels; wherein, the target mapping relationship of each color channel in the plurality of color channels is generated based on the distortion model corresponding to each color channel respectively, and the target mapping relationship is used to indicate the source coordinates corresponding to the same pixel coordinate in the second image in different color channels of the first image.

[0017] Thirdly, a computer device is provided, the computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image processing method described in the first aspect.

[0018] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method described in the first aspect.

[0019] Fifthly, a computer program product is provided that, when the computer program product is run on a computer device, causes the computer device to perform the image processing method described in the first aspect.

[0020] It is understood that the beneficial effects of the second, third, fourth, and fifth aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0021] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and technologies are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0023] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0024] It should be understood that "one or more" as used in this application refers to one, two, or more, and "multiple" as used in this application refers to two or more. In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0025] To facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0026] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0027] The application scenarios involved in the embodiments of this application are described below.

[0028] With the continuous development of technology, near-eye display devices such as AR and VR have become widespread. To achieve thinness and a wide field of view, near-eye display devices typically employ complex optical systems (such as freeform surface systems and waveguide systems). The freeform surface system uses freeform surface technology, employing specially designed asymmetric mirrors to directly and precisely fold and control the light path. The waveguide system uses waveguide technology, with total internal reflection as its core principle. When an image is emitted from the miniature display of the near-eye display device, it is coupled into the waveguide sheet through an input coupling grating for total internal reflection propagation, and then coupled out through an output coupling grating to enter the human eye. However, asymmetric mirrors and coupling gratings require extremely high precision in processing and assembly. Even minute errors can lead to optical distortion in the image. Due to the physical characteristics of the grating, different colors of light are diffracted at slightly different angles, causing chromatic aberration and resulting in a distorted image and rainbow effect when transmitted to the human eye. Therefore, when using near-eye display devices, image correction is necessary to ensure the quality of the image transmitted to the human eye.

[0029] In related technologies, three schemes are typically used to correct images. The first uses monochrome geometric distortion correction, primarily modeling and correcting image shape distortion at a single wavelength. The second uses an image sampling scheme, performing real-time backsampling calculations for each frame. The third uses a unified correction model, applying the same set of correction parameters to correct geometric distortion across all color channels. However, these three schemes each have certain problems to varying degrees. The first scheme fails to adequately consider the refractive differences of different wavelengths of light, resulting in residual lateral chromatic aberration that severely impacts image quality. The second scheme requires a huge amount of computation, increasing computational overhead and latency, making it difficult to meet real-time rendering requirements. The third scheme cannot accurately match the actual chromatic aberration characteristics of the optical system, limiting its correction effectiveness.

[0030] Therefore, this application provides an image processing method. This image processing method can be applied to various imaging systems, such as wide-angle lenses, fisheye lenses, microscopic imaging, freeform surface systems, and waveguide systems. In this method, the distortion model of each color channel in multiple color channels of the image is obtained through imaging system calibration. Then, the target mapping relationship is determined based on the distortion model of each color channel. Finally, image correction processing is performed based on the target mapping relationship of each color channel. In this way, image correction can be achieved quickly and accurately, improving image quality.

[0031] The image processing method provided in this application can automatically complete the image processing process and is easily integrated into the camera image signal processor (ISP) or post-processing software, enabling real-time image quality enhancement. Furthermore, the image processing method provided in this application does not depend on specific optical hardware; as long as the distortion model of each color channel in the image can be obtained through calibration, it can be applied to various imaging systems, exhibiting high versatility.

[0032] The calibration process provided in the embodiments of this application will be explained in detail below.

[0033] It should be noted that the imaging system in this application embodiment can be applied to devices requiring forward calibration, such as cameras and video cameras, or to devices requiring reverse calibration, such as AR and VR. Forward calibration refers to using known standard reference objects and measurement systems to determine the internal and external parameters of the imaging system. Reverse calibration refers to calibrating the imaging system by inferring the input conditions or system state from known output results (such as images generated by the imaging system).

[0034] For example, a computer device can obtain the distortion model of each of the multiple color channels using the following two calibration methods.

[0035] The first method (forward calibration): This calibration method may include the following steps 1 to 4.

[0036] Step 1: Obtain the distorted image.

[0037] For example, the calibration board can be placed in front of the lens of a camera, video camera, or other device, and the calibration board can be photographed to obtain a distorted image.

[0038] For example, the distorted image may include multiple color channels. A color channel is a fundamental color component that constitutes an image. These multiple color channels can constitute the color of pixels in the distorted image. For example, if the distorted image is an RGB image, then the multiple color channels of the distorted image can be red (R) channels, green (G) channels, and blue (B) channels. If the distorted image is a CMYK image, then the multiple color channels of the distorted image can be cyan (C) channels, magenta (M) channels, yellow (Y) channels, and black (K) channels. Of course, this is not a limitation; when the distorted image is in other image formats, the multiple color channels of the distorted image can also be other color channels, and this embodiment of the application does not limit this.

[0039] For example, the calibration plate can be photographed from different positions and angles to obtain multiple distorted images.

[0040] For example, the calibration board may include multiple corner points, which may be the intersections of lines of a pattern (such as a checkerboard) on the calibration board.

[0041] Step 2: Determine the image coordinates of each corner point in each of the multiple single-channel images corresponding to the distorted image.

[0042] For example, the distorted image can be separated into multiple color channels to obtain multiple single-channel images that correspond one-to-one with each color channel. Then, for any single-channel image among these multiple single-channel images, multiple corner points in the single-channel image are detected to obtain the image coordinates of each corner point in the single-channel image.

[0043] The image coordinates of these multiple corner points are coordinates in an image coordinate system. For example, this image coordinate system can be a coordinate system with the top-left corner of the image as the origin. The horizontal axis of this image coordinate system can be the length of the image, and the vertical axis can be the width of the image. For example, for an image with a resolution of 1920×1080, the horizontal axis of this image coordinate system has a range of [0, 1920], and the vertical axis has a range of [0, 1080].

[0044] Step 3: For any one of the multiple single-channel images, normalize the image coordinates of multiple corner points in the single-channel image to obtain the normalized coordinates of each corner point.

[0045] The normalized coordinates are coordinates in a normalized coordinate system. For example, the normalized coordinate system can be a standardized coordinate system with the image center as the origin. The range of both the horizontal and vertical axes of the normalized coordinate system can be [-1, 1].

[0046] For example, the image coordinates of multiple corner points in a single-channel image can be normalized using the following formula to obtain the normalized coordinates of each corner point.

[0047]

[0048]

[0049]

[0050]

[0051] in, This represents the x-coordinate of the image center of the single-channel image. The length of this single-channel image. This represents the ordinate value of the image center of the single-channel image. This represents the width of the single-channel image. The x-coordinate value of the normalized coordinates of the corner point. The x-coordinate value of the image coordinates of the corner point. The ordinate value is the normalized coordinate of the corner point. The vertical coordinate of the corner point is the image coordinate value.

[0052] Step 4: Determine the distortion model of the color channel corresponding to the single-channel image based on the normalized coordinates of multiple corner points in the single-channel image and the true coordinates of these corner points on the calibration plate.

[0053] This distortion model describes the degree of distortion in the effective image location. It is used to correct distorted images. The model takes normalized coordinates of pixel coordinates (i.e., the image coordinates of the pixels) as input and outputs a distortion factor corresponding to those pixel coordinates. This distortion factor describes the proportional relationship between the distance from the normalized coordinates of the pixel to the image center in the undistorted state and the distance from the normalized coordinates of the pixel to the image center in the distorted state, and is used to calculate the corrected pixel position.

[0054] For example, the normalized coordinates of each corner point in the single-channel image and the true coordinates of the corner points on the calibration board can be used as input to determine the distortion coefficients of the color channel corresponding to the single-channel image through a camera calibration algorithm. A distortion model for the color channel is then generated based on these distortion coefficients. For example, the camera calibration algorithm can be the Zhang Zhengyou calibration method, etc., and this application embodiment does not limit this approach.

[0055] In some implementations, if the distortion coefficient of a color channel is greater than 0, it indicates that the single-channel image corresponding to that color channel has experienced barrel distortion. If the distortion coefficient of a color channel is equal to 0, it indicates that the single-channel image corresponding to that color channel has not experienced distortion. If the distortion coefficient of a color channel is less than 0, it indicates that the single-channel image corresponding to that color channel has experienced pincushion distortion.

[0056] For example, in the case where the multiple color channels include R channel, G channel, and B channel, the distortion model of R channel, G channel, and B channel is as follows.

[0057] The distortion model of the R channel:

[0058] G-channel distortion model:

[0059] B-channel distortion model:

[0060]

[0061] in, This represents the distortion factor of the R channel. The distortion coefficients of the R channel (also known as fitting constants) are the distortion coefficients. This is the distance from the normalized pixel coordinates to the image center. The distortion factor for the G channel. The distortion coefficient of the G channel. This represents the distortion factor for channel B. This represents the distortion coefficient of channel B.

[0062] The second method (reverse calibration): This calibration method may include the following steps 1 to 4.

[0063] Step 1: Obtain the distorted image.

[0064] For example, a high-precision camera (such as an eye-tracking camera) is pointed at the lens of an AR or VR device, and a standard image (such as a grid image) is sent to the display of the AR or VR device. The image presented through the lens of the AR or VR device is then captured by the high-precision camera, resulting in a distorted image.

[0065] For example, the standard image may include multiple intersections, which are the intersections of lines of a pattern (such as a grid) in the standard image.

[0066] Step 2: Determine the image coordinates of each intersection point among the multiple intersection points in each of the multiple single-channel images corresponding to the distorted image.

[0067] For example, the distorted image can be separated into multiple color channels to obtain multiple single-channel images that correspond one-to-one with each color channel. Then, for any single-channel image among these multiple single-channel images, multiple intersection points in the single-channel image are detected to obtain the image coordinates of each intersection point in the single-channel image.

[0068] Step 3: For any one of the multiple single-channel images, normalize the image coordinates of the multiple intersection points in the single-channel image to obtain the normalized coordinates of each intersection point.

[0069] Step 4: Determine the distortion model of the color channel corresponding to the single-channel image based on the normalized coordinates of multiple intersection points in the single-channel image and the image coordinates of these multiple intersection points in the standard image.

[0070] For example, based on the normalized coordinates of each intersection point in the single-channel image and its corresponding real image coordinates in the standard image, the distortion coefficients (i.e., inverse distortion coefficients) of the color channel corresponding to the single-channel image can be determined by an optimization algorithm, and the distortion model of the color channel can be generated based on the distortion coefficients of the color channel.

[0071] This application embodiment only uses the above two calibration methods as examples to illustrate the process of obtaining the distortion model of each color channel in multiple color channels. In actual applications, the distortion model of each color channel in multiple color channels can also be obtained by other methods, and this application embodiment does not limit this.

[0072] It should be noted that the distortion model in this embodiment is processed based on normalized coordinates, which makes the distortion model independent of the specific resolution of the image, thereby enhancing the versatility of the distortion model.

[0073] In addition, by establishing a distortion model independently for each of the multiple color channels, the differences in the representation of different colors in the image caused by optical properties can be captured more precisely. As a result, when the image is subsequently corrected using multiple distortion models corresponding to multiple color channels, the entire area of ​​the image (especially the edge area) can obtain higher geometric accuracy and color fidelity, thereby effectively eliminating color fringing.

[0074] After obtaining the distortion model of each color channel in multiple color channels, the target mapping relationship of each color channel can be determined based on the distortion model of each color channel in multiple color channels. This will be explained in detail below.

[0075] For example, the target mapping relationship generation process may include steps 1 through 4 as follows.

[0076] Step 1: Normalize multiple pixel coordinates to obtain multiple normalized coordinates that correspond one-to-one with the multiple pixel coordinates. These multiple pixel coordinates are all pixel coordinates in an image of a preset size.

[0077] The preset size can be set in advance. For example, the preset size can be 720×480, 1280×720, 1920×1080, or 2560×1440, etc., but this application embodiment does not limit it.

[0078] The preset size is the size of the image to be processed by the imaging system that requires subsequent image correction. The preset size can be set according to image processing requirements. In this embodiment, different target mapping relationships can be generated for multiple color channels of images of different sizes based on the distortion model.

[0079] Step 2: For any one of the multiple normalized coordinates, determine the distortion factor based on the normalized coordinate and the distortion model of the target color channel. The target color channel is any one of the multiple color channels.

[0080] The distortion factor is used to describe the proportional relationship between the distance from the normalized coordinate to the image center in the undistorted case and the distance from the normalized coordinate to the image center in the distorted case.

[0081] For example, a computer device can determine the distortion factor based on the normalized coordinates and the distortion model of the target color channel using the following formula.

[0082]

[0083]

[0084] in, Normalized coordinates Distance to the center of the image It is a distortion factor. The distortion coefficient is the target color channel.

[0085] Step 3: Obtain the source coordinates corresponding to the normalized coordinates based on the distortion factor and the normalized coordinates.

[0086] The distortion factor can describe the proportional relationship between the distance from the normalized coordinate to the image center in the case of no distortion and the case of distortion. The normalized coordinate is the normalized coordinate in the corrected image. Therefore, based on the distortion factor and the normalized coordinate, the image coordinate corresponding to the normalized coordinate in the image before correction can be accurately determined. In this embodiment, this is referred to as the source coordinate.

[0087] For example, the operation by which the computer device obtains the source coordinates corresponding to the normalized coordinates based on the distortion factor and the normalized coordinates can include the following two methods.

[0088] The first method involves multiplying the distortion factor by the normalized coordinates and then inversely normalizing the resulting coordinates to obtain the source coordinates corresponding to the normalized coordinates.

[0089] For example, the first method can be expressed by the following formula:

[0090]

[0091] in, For the normalized coordinates The corresponding x-coordinate value of the source coordinates, This is the distortion factor. For the normalized coordinates The corresponding ordinate value of the source coordinates, The horizontal dimension of the image is the preset size. This represents the x-coordinate of the image center. The vertical dimension of the image. The ordinate value is the image center coordinate of the image.

[0092] The second method involves the computer device multiplying the distortion factor by the normalized coordinates and then inversely normalizing the resulting coordinates to obtain the mapped coordinates. The scaling factor is then multiplied by the mapped coordinates to obtain the source coordinates corresponding to the normalized coordinates. This scaling factor is determined based on the distortion coefficients in the distortion model of the target color channel and the coordinates of the image center (i.e., the image coordinates of the image center).

[0093] Because distorted images (such as barrel or pincushion distortion) may have invalid black borders after correction, resulting in a poor user experience, the mapped coordinates can be scaled using a scaling factor after obtaining them. This aims to eliminate black borders in the corrected image, fully utilize the entire image frame, and ensure that the effective image fills the entire frame, preserving as much image information as possible from the original image. This improves the visual appeal and usability of the corrected image.

[0094] For example, the computer device can determine the scaling factor using the following formula, based on the distortion coefficients in the distortion model of the target color channel and the coordinates of the image center.

[0095]

[0096]

[0097]

[0098]

[0099] in, The distortion coefficient is... The x-coordinate value of the image center coordinates. The ordinate value of the center coordinate of the image. This represents the coefficient component of the scaling factor along the horizontal axis. This is the coefficient component of the scaling factor along the vertical axis.

[0100] For example, the operation of multiplying the scaling factor by the mapped coordinates to obtain the source coordinates corresponding to the normalized coordinates can be expressed by the following formula:

[0101]

[0102] in, This is the x-coordinate value of the source coordinates corresponding to the normalized coordinates. This is the ordinate value of the source coordinates corresponding to the normalized coordinates. This is the x-coordinate value of the mapped coordinates. This is the ordinate value of the mapped coordinates. Used to limit the x-coordinate value of the source coordinate to a threshold. Inside. Used to limit the ordinate value of the source coordinate to a threshold. Within. Threshold and threshold All of these can be preset.

[0103] Step 4: Add the pixel coordinates and source coordinates corresponding to the normalized coordinates to the target mapping relationship of the target color channel.

[0104] The target mapping relationship of the target color channel is used to indicate the source coordinates of the pixel coordinates of each pixel in the single-channel image corresponding to the target color channel in the corrected image in the image before correction.

[0105] This application embodiment only uses the above process as an example to illustrate the process of generating the target mapping relationship of each color channel in multiple color channels. In actual applications, the target mapping relationship of each color channel in multiple color channels can also be generated in other ways, and this application embodiment does not limit it.

[0106] After generating the target mapping relationship for each color channel in multiple color channels, the image can be corrected based on the target mapping relationship for each color channel in multiple color channels.

[0107] The image processing procedure provided in the embodiments of this application will be explained in detail below.

[0108] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this application. See also... Figure 1 The method may include the following steps: Step 101: Obtain the first image, which has multiple color channels.

[0109] The first image is the image that needs to be corrected. For example, the first image can be an image acquired in real time by a computer device, or the first image can be an image stored by the computer device. Of course, it is not limited to these. The first image can also be an image acquired by the computer device through other means. This application embodiment does not limit this.

[0110] For example, the first image can be an image captured by a device such as a camera or camcorder, or it can be an image that needs to be displayed on a monitor in an AR or VR device.

[0111] Step 102: Create a second image based on the first image. Both the second and first images are preset sizes.

[0112] Creating a second image based on the first image ensures that the output image (i.e., the second image) is the same size as the input image (i.e., the first image) during subsequent image correction, thus avoiding processing errors caused by size mismatch.

[0113] Step 103: Set the pixel value of each pixel in the second image according to the target mapping relationship of the first image and each color channel in the plurality of color channels; wherein, the target mapping relationship of each color channel in the plurality of color channels is generated based on the distortion model corresponding to each color channel respectively, and the target mapping relationship is used to indicate the source coordinates corresponding to the same pixel coordinates in the second image in different color channels of the first image.

[0114] The distortion model of color channels and the target mapping relationship of color channels have been explained in the previous text and will not be repeated here.

[0115] It should be noted that, for any one of the multiple color channels, the operation of generating the target mapping relationship of that color channel based on the distortion model of that color channel can be performed before step 103. For example, it can be performed before step 101; or after step 101 and before step 102; or after step 102 and before step 103. This application embodiment does not specifically limit the timing of generating the target mapping relationship of the color channel.

[0116] The target mapping relationship of each color channel can accurately reflect the source coordinates of each pixel coordinate in the second image corresponding to the first image. Furthermore, the target mapping relationship of different color channels can precisely match the actual color difference characteristics of the optical system. Therefore, by setting the pixel value of each pixel in the second image according to the first image and the multiple target mapping relationships of these multiple color channels, color difference correction of the first image can be achieved simultaneously with distortion correction. This avoids image distortion, improves the image correction effect, and obtains better image quality. Moreover, the embodiments of this application can achieve image correction quickly and accurately using only the target mapping relationship, thereby greatly reducing the computational load and improving image correction efficiency. In scenarios requiring real-time image processing, it can reduce computational overhead and latency, thus better meeting the requirements of real-time rendering.

[0117] The pixel value includes multiple color components that correspond one-to-one with the multiple color channels. For example, for an RGB image, which includes R channels, G channels, and B channels, the pixel value of each pixel in the RGB image can include an R value, a G value, and a B value, where the R value is the color component corresponding to the R channel, the G value is the color component corresponding to the G channel, and the B value is the color component corresponding to the B channel.

[0118] In some implementations, step 103 may include steps A through C as follows: Step A: The computer device obtains the first source coordinates of the second pixel in the first image from the target mapping relationship of the target color channel. The target color channel is any one of the multiple color channels, and the second pixel is any one pixel in the second image.

[0119] Step B: The computer device determines n first pixels from the first image based on the first source coordinates, where n is a positive integer.

[0120] The n first pixels are the pixels in the first image that are related to the first source coordinates.

[0121] Since the pixel coordinates of pixels in the image are represented in integer form, while the first source coordinates may be floating-point numbers, that is, the first source coordinates may not be directly the pixel coordinates in the first image. Therefore, we can first determine n first pixels in the first image that are related to the first source coordinates, and then use the pixel values ​​of these n first pixels to set the pixel values ​​of the second pixel in the second image.

[0122] In some implementations, the operation of the computer device determining n first pixels from the first image based on the first source coordinates can be as follows: if the first source coordinates fall within the image coordinate range of the first image, determine n first pixels from the first image based on the first source coordinates. Subsequently, the target color component in the pixel value of the second pixel can be set according to the target color component in the pixel value of the n first pixels, where the target color component is the color component corresponding to the target color channel.

[0123] The image coordinate range of the first image can be preset. If the first source coordinates belong to the image coordinate range of the first image, the first pixel related to the first source coordinates can be determined from the first image; otherwise, the first pixel related to the first source coordinates cannot be determined from the first image.

[0124] For example, when the first source coordinates do not belong to the image coordinate range of the first image, the target color component in the pixel value of the second pixel can be set to a preset value. The preset value can be set in advance. For example, the preset value can be 0, 1, etc., and this application embodiment does not limit it.

[0125] In some implementations, the operation of the computer device to determine n first pixels from the first image based on the first source coordinates can be as follows: if the first source coordinates are the same as the pixel coordinates of a pixel in the first image, the pixel in the first image whose pixel coordinates are the first source coordinates is determined as the first pixel; if the first source coordinates are different from the pixel coordinates of all pixels in the first image, multiple neighboring pixels in the first image that are adjacent to the first source coordinates are all determined as the first pixel.

[0126] In the first image, the multiple neighboring pixels adjacent to the first source coordinates refer to the nearest ring of pixels around the first source coordinates, such as the four pixels in the upper left, upper right, lower left, and lower right directions.

[0127] Step C: The computer device sets the target color component in the pixel value of the second pixel based on the target color component in the pixel value of each of the n first pixels.

[0128] In some implementations, step C can be performed as follows: when n is 1, the target color component in the pixel values ​​of the n first pixels is determined as the target color component in the pixel values ​​of the second pixel; when n is greater than or equal to 2, linear interpolation is performed on the target color component in the pixel values ​​of the n first pixels based on the first source coordinates and the pixel coordinates of each of the n first pixels to obtain the target color component in the pixel values ​​of the second pixel.

[0129] For example, the linear interpolation operation can be a one-dimensional linear interpolation operation, a bilinear interpolation operation, or a trilinear interpolation operation, etc., and the embodiments of this application do not limit this.

[0130] In this way, when there is only one first pixel, the target color component in the pixel value of the second pixel is directly set to the target color component in the pixel value of the first pixel, thus avoiding unnecessary complex calculations. When there are multiple first pixels, the target color component in the pixel value of the second pixel is determined through linear interpolation, which makes the second image smoother and more natural, thereby improving the image correction effect.

[0131] For example, when the multiple color channels include R channel, G channel, and B channel, steps A to C above can be expressed by the following formula:

[0132]

[0133]

[0134] in, The R value is the pixel value of the second pixel, where [0] represents... It is the first color component (i.e., R value) in the RGB values. Represents the pixel coordinates of the second pixel The source coordinates corresponding to the target mapping relationship in the R channel In Within the indicated image coordinate range. Represents the pixel coordinates of the second pixel The source coordinates corresponding to the target mapping relationship in the R channel Not in Within the indicated image coordinate range. The G value in the pixel value of the second pixel, [1] represents It is the second color component (i.e., G value) in the RGB value. Represents the pixel coordinates of the second pixel The source coordinates corresponding to the target mapping relationship in the G channel In Within the indicated image coordinate range. Represents the pixel coordinates of the second pixel The source coordinates corresponding to the target mapping relationship in the G channel Not in Within the indicated image coordinate range. The B value in the pixel value of the second pixel, [1] represents It is the second color component (i.e., the B value) in the RGB value. Represents the pixel coordinates of the second pixel The source coordinates corresponding to the target mapping relationship in channel B. In Within the indicated image coordinate range. Represents the pixel coordinates of the second pixel The source coordinates corresponding to the target mapping relationship in channel B. Not in Within the indicated image coordinate range.

[0135] In this embodiment, a first image is acquired, which has multiple color channels. A second image is created based on the first image, and both the first and second images are of a preset size. Based on the target mapping relationship between the first image and each of the multiple color channels, the pixel value of each pixel in the second image is set. This target mapping relationship is generated through the distortion model of the color channel, and it indicates the source coordinates of each pixel in the second image corresponding to the pixel coordinates in the first image. The target mapping relationships of each color channel accurately reflect the source coordinates of each pixel coordinate in the second image corresponding to the pixel coordinates in the first image. Furthermore, the target mapping relationships of different color channels can precisely match the actual color difference characteristics of the optical system. Therefore, by setting the pixel value of each pixel in the second image based on the multiple target mapping relationships of the first image and the multiple color channels, color difference correction of the first image can be achieved simultaneously with distortion correction, thus improving the image correction effect and obtaining better image quality. Moreover, this embodiment can achieve image correction quickly and accurately using only the target mapping relationship, thereby greatly reducing the computational load and improving image correction efficiency. In scenarios requiring real-time image processing, it can reduce computational overhead and latency, thus better meeting the requirements of real-time rendering.

[0136] Figure 2 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The apparatus can be implemented as part or all of a computer device by software, hardware, or a combination of both, and this computer device can be as described below. Figure 3 The computer equipment shown. See also Figure 2 The device includes: a first acquisition module 201, a creation module 202, and a setting module 203.

[0137] The first acquisition module 201 is used to acquire a first image, the first image having multiple color channels; Module 202 is used to create a second image based on the first image. Both the second image and the first image are preset sizes. The setting module 203 is used to set the pixel value of each pixel in the second image according to the first image and the target mapping relationship of each color channel in the plurality of color channels; wherein, the target mapping relationship of each color channel in the plurality of color channels is generated based on the distortion model corresponding to each color channel respectively, and the target mapping relationship is used to indicate the source coordinates corresponding to the same pixel coordinates in the second image in different color channels of the first image.

[0138] Optionally, the pixel value includes multiple color components that correspond one-to-one with the plurality of color channels, and the setting module 203 is used for: The first source coordinates of the second pixel in the first image are obtained from the target mapping relationship of the target color channel. The target color channel is any one of the multiple color channels, and the second pixel is any pixel in the second image. n first pixels are determined from the first image based on the first source coordinates, where n is a positive integer; Based on the target color component in the pixel value of each of the n first pixels, the target color component in the pixel value of the second pixel is set, and the target color component is the color component corresponding to the target color channel.

[0139] Optionally, the setting module 203 is used for: If the first source coordinates belong to the image coordinate range of the first image, determine the n first pixels from the first image based on the first source coordinates; The method also includes: If the first source coordinates are not within the image coordinate range of the first image, the target color component in the pixel value of the second pixel is set to a preset value.

[0140] Optionally, the setting module 203 is used for: If the first source coordinates are the same as the pixel coordinates of a pixel in the first image, the pixel in the first image whose pixel coordinates are the first source coordinates is determined as the first pixel; If the first source coordinates are different from the pixel coordinates of all pixels in the first image, then all neighboring pixels in the first image that are adjacent to the first source coordinates are determined as the first pixel.

[0141] Optionally, the setting module 203 is used for: When n is 1, the target color component in the pixel values ​​of the n first pixels is determined as the target color component in the pixel values ​​of the second pixel; When n is greater than or equal to 2, linear interpolation is performed on the target color component in the pixel value of the n first pixels based on the first source coordinates and the pixel coordinates of each of the n first pixels to obtain the target color component in the pixel value of the second pixel.

[0142] Optionally, the device further includes: The normalization module is used to normalize multiple pixel coordinates to obtain multiple normalized coordinates that correspond one-to-one with the multiple pixel coordinates. The multiple pixel coordinates are all pixel coordinates in an image of a preset size. The first determining module is used to determine the distortion factor for any one of the multiple normalized coordinates based on the distortion model of the normalized coordinate and the target color channel, wherein the target color channel is any one of the multiple color channels. The second acquisition module is used to obtain the source coordinates corresponding to the normalized coordinates based on the distortion factor and the normalized coordinates; Add a module to add the pixel coordinates and source coordinates corresponding to the normalized coordinates to the target mapping relationship of the target color channel.

[0143] Optionally, the second acquisition module is used for: Multiplying the distortion factor by the normalized coordinates yields inverse normalized coordinates, which gives the source coordinates corresponding to the normalized coordinates; or, Multiplying the distortion factor by the normalized coordinates yields inverse normalized coordinates, resulting in mapped coordinates. Multiplying the scaling factor by the mapped coordinates yields the source coordinates corresponding to the normalized coordinates. The scaling factor is determined based on the distortion coefficients in the distortion model of the target color channel and the image center coordinates.

[0144] Optionally, the device further includes: The second determining module is used to determine the scaling factor based on the distortion coefficients in the distortion model of the target color channel and the coordinates of the image center using the following formula;

[0145]

[0146]

[0147]

[0148] The distortion coefficient is... This represents the x-coordinate of the center of the image. This represents the ordinate value of the center coordinate of the image. This represents the scaling factor's coefficient component along the horizontal axis. This represents the scaling factor in the vertical direction.

[0149] In this embodiment, a first image is acquired, which has multiple color channels. A second image is created based on the first image, and both the first and second images are of a preset size. Based on the target mapping relationship between the first image and each of the multiple color channels, the pixel value of each pixel in the second image is set. This target mapping relationship is generated through the distortion model of the color channel, and it indicates the source coordinates of each pixel in the second image corresponding to the pixel coordinates in the first image. The target mapping relationships of each color channel accurately reflect the source coordinates of each pixel coordinate in the second image corresponding to the pixel coordinates in the first image. Furthermore, the target mapping relationships of different color channels can precisely match the actual color difference characteristics of the optical system. Therefore, by setting the pixel value of each pixel in the second image based on the multiple target mapping relationships of the first image and the multiple color channels, color difference correction of the first image can be achieved simultaneously with distortion correction, thus improving the image correction effect and obtaining better image quality. Moreover, this embodiment can achieve image correction quickly and accurately using only the target mapping relationship, thereby greatly reducing the computational load and improving image correction efficiency. In scenarios requiring real-time image processing, it can reduce computational overhead and latency, thus better meeting the requirements of real-time rendering.

[0150] It should be noted that the image processing device provided in the above embodiments is only illustrated by the division of the above functional modules when processing images. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0151] The functional modules in the above embodiments can be integrated into one processing unit, or each functional module can exist as a separate physical processing unit, or two or more functional modules can be integrated into one processing unit. The processing unit can be implemented in hardware or software. Furthermore, the specific names of the functional modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0152] The image processing apparatus and image processing method embodiments provided in the above embodiments belong to the same concept. The specific working process and technical effects of the functional modules in the above embodiments can be found in the method embodiment section, and will not be repeated here.

[0153] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 3As shown, the computer device 3 includes a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, it implements the steps in the image processing method in the above embodiments.

[0154] Computer device 3 can be a general-purpose computer device or a special-purpose computer device. In specific implementations, computer device 3 can be a desktop computer, a portable computer, a network server, a handheld computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of this application do not limit the type of computer device 3. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0155] Processor 30 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor.

[0156] In some embodiments, memory 31 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, memory 31 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device 3. Furthermore, memory 31 may include both internal storage units and external storage devices of the computer device 3. Memory 31 is used to store operating systems, applications, boot loader, data, and other programs. Memory 31 may also be used to temporarily store data that has been output or will be output.

[0157] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0158] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0159] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0160] This application provides a computer program product that, when run on a computer, causes the computer to perform the steps described in the various method embodiments above.

[0161] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above method embodiments of this application can be implemented by a computer program. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable storage medium can include at least: any entity or device capable of carrying computer program code to a computer device, recording media, computer memory, read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices. The computer-readable storage medium mentioned in this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0162] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.

[0163] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this application according to actual needs.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0166] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first image, which has multiple color channels; A second image is created based on the first image, and both the second image and the first image are of a preset size; Based on the target mapping relationship between the first image and each color channel in the plurality of color channels, the pixel value of each pixel in the second image is set; wherein, the target mapping relationship of each color channel in the plurality of color channels is generated based on the distortion model corresponding to each color channel respectively, and the target mapping relationship is used to indicate the source coordinates corresponding to the same pixel coordinate in the second image in different color channels of the first image.

2. The method as described in claim 1, characterized in that, The pixel value includes multiple color components that correspond one-to-one with the plurality of color channels. The step of setting the pixel value of each pixel in the second image according to the target mapping relationship between the first image and each color channel in the plurality of color channels includes: The first source coordinates of the second pixel in the first image are obtained from the target mapping relationship of the target color channel, wherein the target color channel is any one of the plurality of color channels, and the second pixel is any pixel in the second image; n first pixels are determined from the first image based on the first source coordinates, where n is a positive integer; Based on the target color component in the pixel value of each of the n first pixels, the target color component in the pixel value of the second pixel is set, wherein the target color component is the color component corresponding to the target color channel.

3. The method as described in claim 2, characterized in that, The step of determining n first pixels from the first image based on the first source coordinates includes: If the first source coordinates belong to the image coordinate range of the first image, the n first pixels are determined from the first image based on the first source coordinates; The method further includes: If the first source coordinates are not within the image coordinate range of the first image, the target color component in the pixel value of the second pixel is set to a preset value.

4. The method as described in claim 2, characterized in that, The step of determining n first pixels from the first image based on the first source coordinates includes: If the first source coordinates are the same as the pixel coordinates of a pixel in the first image, the pixel in the first image whose pixel coordinates are the first source coordinates is determined as the first pixel; When the first source coordinates are different from the pixel coordinates of all pixels in the first image, multiple neighboring pixels in the first image that are adjacent to the first source coordinates are all determined as the first pixel.

5. The method as described in claim 2, characterized in that, The step of setting the target color component in the pixel value of the second pixel based on the target color component in the pixel value of each of the n first pixels includes: When n is 1, the target color component in the pixel values ​​of the n first pixels is determined as the target color component in the pixel values ​​of the second pixel; When n is greater than or equal to 2, linear interpolation is performed on the target color component in the pixel value of the n first pixels based on the first source coordinates and the pixel coordinates of each of the n first pixels to obtain the target color component in the pixel value of the second pixel.

6. The method according to any one of claims 1 to 5, characterized in that, Before setting the pixel value of each pixel in the second image based on the target mapping relationship between the first image and each color channel in the plurality of color channels, the method further includes: Normalize multiple pixel coordinates to obtain multiple normalized coordinates that correspond one-to-one with the multiple pixel coordinates, where the multiple pixel coordinates are all pixel coordinates in the image of the preset size; For any one of the plurality of normalized coordinates, the distortion factor is determined based on the normalized coordinate and the distortion model of the target color channel, wherein the target color channel is any one of the plurality of color channels; The source coordinates corresponding to the normalized coordinates are obtained based on the distortion factor and the normalized coordinates; The pixel coordinates and source coordinates corresponding to the normalized coordinates are added to the target mapping relationship of the target color channel.

7. The method as described in claim 6, characterized in that, The step of obtaining the source coordinates corresponding to the normalized coordinates based on the distortion factor and the normalized coordinates includes: Multiplying the distortion factor by the normalized coordinates yields inverse normalized coordinates, resulting in the source coordinates corresponding to the normalized coordinates; or... The coordinates obtained by multiplying the distortion factor by the normalized coordinates are inversely normalized to obtain the mapped coordinates; the scaling factor is multiplied by the mapped coordinates to obtain the source coordinates corresponding to the normalized coordinates. The scaling factor is determined based on the distortion coefficients and image center coordinates in the distortion model of the target color channel.

8. The method as described in claim 7, characterized in that, The method further includes: The scaling factor is determined using the following formula based on the distortion coefficients in the distortion model of the target color channel and the coordinates of the image center; The The distortion coefficient is the value of the distortion coefficient. The x-coordinate value of the center coordinate of the image, the The ordinate value of the center coordinate of the image, The scaling factor is the coefficient component along the horizontal axis. The scaling factor is the coefficient component along the vertical axis.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program, when executed by the processor, implements the method as claimed in any one of claims 1 to 8.

10. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device causes the computer device to perform the method as described in any one of claims 1 to 8.

Images