Image processing method and apparatus

Abstract

This application provides an image processing method and apparatus, comprising: acquiring a first YUV image; acquiring a YUV mapping relationship, wherein the YUV mapping relationship indicates the positional mapping relationship between pixels in an initial YUV image and each pixel in a target YUV image, wherein the target YUV image is an image obtained by performing distortion correction processing on the initial YUV image; and obtaining a second YUV image based on the YUV mapping relationship and the first YUV image. The image processing method provided by this application can reduce the time consumed in the distortion correction process.

Description

Technical Field

[0001] This application relates to the field of intelligent vehicle technology, and more specifically, to image processing methods and apparatus. Background Technology

[0002] In recent years, with the rapid development of vision technology, various types of camera sensors, such as fisheye cameras, eagle-eye cameras, or monocular and binocular cameras, have been utilized in autonomous driving equipment. These devices can use computing platforms to process images acquired by camera sensors using machine learning and other algorithms to make further decisions. However, images acquired by camera sensors often contain a certain degree of distortion. In this case, directly using a distorted image as input to a processing algorithm may affect the algorithm's results. Therefore, computing platforms typically first perform distortion correction, also known as image straightening, on the distorted image before using it as input to the processing algorithm.

[0003] Currently, the distortion correction process performed on the YUV format images obtained by the computing platform is time-consuming.

[0004] With the widespread application of visual technology in various fields, effectively reducing the time consumption of image distortion correction processing has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides an image processing method and apparatus that can reduce the time required for distortion correction.

[0006] In a first aspect, embodiments of this application provide an image processing method, comprising: acquiring a first YUV image; acquiring a YUV mapping relationship, wherein the YUV mapping relationship is used to indicate the positional mapping relationship between pixels in an initial YUV image and each pixel in a target YUV image, wherein the target YUV image is an image obtained by performing distortion correction processing on the initial YUV image; and obtaining a second YUV image based on the YUV mapping relationship and the first YUV image.

[0007] The image processing method provided in this application, because the YUV mapping relationship indicates the positional mapping relationship between each pixel in the target YUV image and the pixels in the initial YUV image, can directly perform distortion correction on the first YUV image based on the YUV mapping relationship after acquiring it, without needing to first convert the first YUV image to an RGB image and then perform distortion correction on the RGB image based on the RGB mapping relationship. Therefore, the time consumed in the distortion correction process can be reduced. Furthermore, even for YUV and RGB images of the same size, since the YUV image uses less data than the RGB image, directly performing distortion correction on the YUV image can further reduce the time consumed in the distortion correction process. It is understood that the more YUV images that need distortion correction, the more significant the reduction in distortion correction time will be.

[0008] In conjunction with the first aspect, in one possible implementation, obtaining the YUV mapping relationship includes: obtaining an RGB mapping relationship, wherein the RGB mapping relationship indicates the positional mapping relationship between pixels in an initial RGB image and each pixel in a target RGB image, the initial RGB image is an image obtained by performing RGB conversion on the initial YUV image, the target RGB image is an image obtained by performing distortion correction processing on the initial RGB image, and the target RGB image and the target YUV image have the same size; and determining the YUV mapping relationship based on the RGB mapping relationship.

[0009] In conjunction with the first aspect, in one possible implementation, determining the YUV mapping relationship based on the RGB mapping relationship includes: determining a first target mapping relationship based on the RGB mapping relationship, the first target mapping relationship indicating the position of each pixel in the target YUV image in the Y channel image corresponding to the initial YUV image; determining a second target mapping relationship based on the RGB mapping relationship, the second target mapping relationship indicating the position of each pixel in the target image in the U channel image corresponding to the initial YUV image; and determining a third target mapping relationship based on the RGB mapping relationship, the third target mapping relationship indicating the position of each pixel in the target image in the V channel image corresponding to the initial YUV image. Correspondingly, obtaining the second YUV image based on the YUV mapping relationship and the first YUV image includes: determining the pixel value of each pixel in the second YUV image based on the pixel value at the position in the Y channel image corresponding to the first YUV image, the pixel value at the position in the U channel image corresponding to the first YUV image, and the pixel value at the position in the V channel image corresponding to the first YUV image.

[0010] In this implementation, based on the RGB mapping relationship, a first target mapping relationship can be determined to indicate the position of each pixel in the target YUV image in the Y-channel image corresponding to the initial YUV image; a second target mapping relationship can be determined to indicate the position of each pixel in the target image in the U-channel image corresponding to the initial YUV image; and a third target mapping relationship can be determined to indicate the position of each pixel in the target image in the V-channel image corresponding to the initial YUV image. It can be understood that after determining the positions of each pixel in the target image in the Y-channel image, U-channel image, and V-channel image corresponding to the initial YUV image, the pixel values ​​at those positions can be obtained. Furthermore, based on these pixel values, the pixel values ​​corresponding to each pixel in the target image can be obtained.

[0011] In conjunction with the first aspect, in one possible implementation, the first target mapping relationship is the same as the RGB mapping relationship; correspondingly, determining the second target mapping relationship based on the RGB mapping relationship includes: determining the horizontal and vertical coordinates of a first pixel in the target YUV image in the U channel image corresponding to the initial YUV image based on the RGB mapping relationship, wherein the first pixel is any one of the pixels; correspondingly, determining the third target mapping relationship based on the RGB mapping relationship includes: determining the horizontal and vertical coordinates of the first pixel in the V channel image corresponding to the initial YUV image based on the RGB mapping relationship.

[0012] In this implementation, the horizontal and vertical coordinates of the first pixel in the target YUV image in the U channel image corresponding to the initial YUV image, as well as the horizontal and vertical coordinates in the V channel image corresponding to the initial YUV image, can be obtained through the RGB mapping relationship to determine the position of the first pixel in the U channel image and the position in the V channel image.

[0013] In conjunction with the first aspect, in one possible implementation, the sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and the first YUV image is stored in a planar storage mode.

[0014] In conjunction with the first aspect, in one possible implementation, determining the abscissa and ordinate values ​​of the first pixel in the target YUV image within the U-channel image corresponding to the initial YUV image includes: according to the formula: Determine the horizontal coordinate value of the first pixel in the U channel image corresponding to the initial YUV image, where temp1 represents the sum of the horizontal coordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image, wherein the second, third, and fourth pixels are adjacent to the first pixel; according to the formula: Determine the ordinate value of the first pixel in the U-channel image corresponding to the initial YUV image, where src h The initial YUV image represents the height, and k represents the ratio. temp2 represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image; correspondingly, the x and y coordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image are determined, including: according to the formula: Determine the x-coordinate value of the first pixel in the V-channel image corresponding to the first image; according to the formula: Determine the ordinate value of the first pixel in the V channel image corresponding to the first image.

[0015] In conjunction with the first aspect, in one possible implementation, the sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and the first YUV image is stored in a packaged storage manner.

[0016] In conjunction with the first aspect, in one possible implementation, determining the abscissa and ordinate values ​​of the first pixel in the target YUV image within the U-channel image corresponding to the initial YUV image includes: according to the formula: Determine the horizontal coordinate value of the first pixel in the U channel image corresponding to the initial YUV image, where temp1 represents the sum of the horizontal coordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image, wherein the second, third, and fourth pixels are adjacent to the first pixel; according to the formula: Determine the ordinate value of the first pixel in the U-channel image corresponding to the initial YUV image, where src h The initial YUV image represents the height, and k represents the ratio. temp2 represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image; correspondingly, the x and y coordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image are determined, including: according to the formula: V x =U x +1, determine the x-coordinate value of the first pixel in the V channel image corresponding to the initial YUV image; according to the formula: Determine the ordinate value of the first pixel in the V channel image corresponding to the initial YUV image.

[0017] Secondly, this application provides an image processing apparatus, comprising: an acquisition module for acquiring a first YUV image and acquiring a YUV mapping relationship, wherein the YUV mapping relationship is used to indicate the positional mapping relationship between pixels in an initial YUV image and each pixel in a target YUV image, wherein the target YUV image is an image obtained by performing distortion correction processing on the initial YUV image; and a processing module for obtaining a second YUV image based on the YUV mapping relationship and the first YUV image.

[0018] In conjunction with the second aspect, in one possible implementation, the acquisition module is further configured to: acquire an RGB mapping relationship, wherein the RGB mapping relationship indicates the positional mapping relationship between pixels in the initial RGB image and each pixel in the target RGB image, wherein the initial RGB image is an image obtained by performing RGB conversion on the initial YUV image, and the target RGB image is an image obtained by performing distortion correction processing on the initial RGB image, wherein the target RGB image and the target YUV image are of equal size; the processing module is further configured to: determine the YUV mapping relationship based on the RGB mapping relationship.

[0019] In one possible implementation, the processing module is further configured to: determine a first target mapping relationship based on the RGB mapping relationship, the first target mapping relationship indicating the position of each pixel in the target YUV image in the Y channel image corresponding to the initial YUV image; determine a second target mapping relationship based on the RGB mapping relationship, the second target mapping relationship indicating the position of each pixel in the target image in the U channel image corresponding to the initial YUV image; determine a third target mapping relationship based on the RGB mapping relationship, the third target mapping relationship indicating the position of each pixel in the target image in the V channel image corresponding to the initial YUV image; correspondingly, the processing module is further configured to: determine the pixel value of each pixel in the second YUV image based on the pixel value of each pixel at its position in the Y channel image corresponding to the first YUV image, the pixel value at its position in the U channel image corresponding to the first YUV image, and the pixel value at its position in the V channel image corresponding to the first YUV image.

[0020] In one possible implementation, the first target mapping relationship is the same as the RGB mapping relationship; correspondingly, the processing module is further configured to: determine the horizontal and vertical coordinate values ​​of a first pixel in the target YUV image in the U channel image corresponding to the initial YUV image according to the RGB mapping relationship, wherein the first pixel is any one of the pixels; and determine the horizontal and vertical coordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image according to the RGB mapping relationship.

[0021] In one possible implementation, the sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and the first YUV image is stored in a planar storage mode.

[0022] In one possible implementation, the processing module is further configured to: according to the formula: Determine the horizontal coordinate value of the first pixel in the U channel image corresponding to the initial YUV image, where temp1 represents the sum of the horizontal coordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image, wherein the second, third, and fourth pixels are adjacent to the first pixel; according to the formula: Determine the ordinate value of the first pixel in the U-channel image corresponding to the initial YUV image, where srch The initial YUV image represents the height, and k represents the ratio. temp2 represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image; correspondingly, the processing module is further configured to: according to the formula: Determine the x-coordinate value of the first pixel in the V-channel image corresponding to the first image; according to the formula: Determine the ordinate value of the first pixel in the V channel image corresponding to the first image.

[0023] In one possible implementation, the sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and the first YUV image is stored in a packaged storage method.

[0024] In one possible implementation, the processing module is further configured to: according to the formula: Determine the horizontal coordinate value of the first pixel in the U channel image corresponding to the initial YUV image, where temp1 represents the sum of the horizontal coordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image, wherein the second, third, and fourth pixels are adjacent to the first pixel; according to the formula: Determine the ordinate value of the first pixel in the U-channel image corresponding to the initial YUV image, where src h The initial YUV image represents the height, and k represents the ratio. temp2 represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image; correspondingly, the processing module is further configured to: according to the formula: V x =U x +1, determine the x-coordinate value of the first pixel in the V channel image corresponding to the initial YUV image; according to the formula: Determine the ordinate value of the first pixel in the V channel image corresponding to the initial YUV image.

[0025] Thirdly, this application provides a computing platform that includes the apparatus described in the second aspect or any of the possible implementations thereof.

[0026] Fourthly, this application provides a mobile device that includes the computing platform described in the third aspect.

[0027] In conjunction with the fourth aspect, in one possible implementation, the mobile device includes an autonomous vehicle.

[0028] Fifthly, this application provides an image processing apparatus, comprising: a memory and a processor; the memory being used to store program instructions; the processor being used to invoke the program instructions in the memory to execute the image processing method as described in the first aspect or any possible implementation thereof.

[0029] In a sixth aspect, this application provides a chip including at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being configured to run a computer program or instructions to perform the image processing method as described in the first aspect or any of its possible implementations.

[0030] In a seventh aspect, this application provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing image processing as described in the first aspect or any of the possible implementations thereof.

[0031] Eighthly, this application provides a computer program product including instructions, the computer program product including computer program code, which, when run on a computer, causes the computer to perform the image processing method as described in the first aspect or any possible implementation thereof. Attached Figure Description

[0032] Figure 1 A structural schematic diagram of the YUV images corresponding to the three sampling methods provided in this application;

[0033] Figure 2 This application provides a structural schematic diagram of an image processing system;

[0034] Figure 3 A schematic diagram showing the distorted image provided in this application;

[0035] Figure 4 A schematic flowchart illustrating an image processing method provided in this application;

[0036] Figure 5 A structural diagram illustrating the storage of the 4x4 size color image provided in this application as an RGB image;

[0037] Figure 6 A structural diagram illustrating the storage of a 4x4 color image provided in this application as a YUV image;

[0038] Figure 7 A structural schematic diagram of bilinear interpolation provided in this application;

[0039] Figure 8 A schematic flowchart of an image processing apparatus provided in this application;

[0040] Figure 9 A schematic flowchart of an image processing apparatus provided in this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] To facilitate understanding, the relevant terms used in this application will be explained first.

[0043] 1. Distortion

[0044] The essence of distortion is the difference in pixel magnification caused by variations in lens refractive power and aperture stop position in different fields of view. Camera distortion can generally be classified into pincushion distortion, barrel distortion, and linear distortion.

[0045] Among them, pincushion distortion, also known as saddle distortion, refers to the magnification of the edge region in the field of view being much greater than that of the central region of the optical axis, and is commonly used in telephoto lenses.

[0046] Barrel distortion, the opposite of pincushion distortion, refers to a magnification in the central region of the optical axis that is much greater than that in the peripheral region. It is often seen in wide-angle lenses and fisheye lenses.

[0047] Linear distortion refers to the distortion caused by the optical axis not being orthogonal to the vertical plane of the object being photographed, such as a building. The far and near sides, which should be parallel, converge at different angles. This distortion is essentially a perspective transformation; at a given angle, any lens will produce similar distortion.

[0048] 2. RGB image

[0049] An RGB image is an image that uses the brightness of the three primary colors—red (R), green (G), and blue (B)—to quantitatively represent its value. R, G, and B are three base colors; adding them in different proportions produces a wide variety of colors. In an RGB image, each pixel has these three base colors.

[0050] 3. YUV images

[0051] YUV images are images encoded using luminance and chrominance. Y represents luminance, and U and V represent chrominance, which defines two aspects of color: hue and saturation.

[0052] Specifically, brightness represents the physical quantity that the human eye actually perceives as the intensity of light emitted or reflected from a light source or the surface of an illuminated object. When any two object surfaces are photographed and the final result is that they are equally bright, or appear equally bright to the eye, it means that their brightness is the same. Hue is represented as color in a color image.

[0053] Hue is the representation of the intensity of reflected and radiated energy of ground objects in an image. The attributes, geometric shapes, distribution ranges, and combination patterns of ground objects can all be reflected in remote sensing images through hue differences.

[0054] Saturation refers to the vividness or purity of a color. Saturation depends on the ratio of chromatic to achromatic components (gray) in the color. The higher the chromatic component, the higher the saturation; the higher the achromatic component, the lower the saturation. Pure colors are highly saturated, such as bright red and bright green. Colors mixed with white, gray, or other hues are unsaturated, such as deep purple, pink, and yellowish-brown. Completely unsaturated colors have no hue whatsoever, such as various shades of gray between black and white.

[0055] Based on the different sampling frequency ratios, YUV images are typically classified into three categories: YUV 4:4:4, YUV 4:2:2, and YUV 4:2:0. YUV 4:4:4 indicates no downsampling of the chroma channels, meaning one Y component corresponds to one U and one V component. YUV 4:2:2 indicates 2:1 horizontal downsampling with no vertical downsampling, meaning two Y components share one U and one V component. YUV 4:2:0 indicates 2:1 horizontal downsampling and 2:1 vertical downsampling, meaning every four Y components share one U and one V component.

[0056] For example, Figure 1 This is a structural diagram of the YUV images corresponding to the three sampling methods provided in this application. In this diagram, black dots represent the Y component corresponding to the sampled pixel, and hollow circles represent the UV component of the sampled pixel (i.e., a hollow circle simultaneously represents a U component and a V component).

[0057] in, Figure 1 (a) shows a schematic diagram of the YUV 4:4:4 structure. For example... Figure 1 As shown in (a), each pixel includes a black dot and a hollow circle, that is, one Y component corresponds to one U component and one V component.

[0058] in, Figure 1 (b) shows a schematic diagram of the YUV 4:2:2 structure. For example... Figure 1 As shown in (b), every two adjacent pixels share a single UV component. Specifically, with Figure 1 (b) Taking the two adjacent pixels within the dashed box as an example for illustration. Figure 1 In the dashed box in (b), the first pixel corresponds to a Y component represented by a black dot, and the second pixel corresponds to a Y component represented by a black dot. The first and second pixels share a hollow circle, that is, the two Y components share a U component and a V component.

[0059] in, Figure 1 (c) shows a structural schematic diagram of YUV 4:2:0. For example... Figure 1 As shown in (c), every four vertically adjacent pixels share a single UV component. Specifically, with Figure 1 (c) Taking the four adjacent pixels within the dashed box as an example for illustration. Figure 1 In the dashed box in (c), each of the four adjacent pixels corresponds to a black dot, that is, each of the four adjacent pixels corresponds to a Y component, and the four adjacent pixels share a hollow circle, that is, the four Y components share a U component and a V component.

[0060] In recent years, with the rapid development of artificial intelligence technology, computer vision technology has been widely applied in fields such as medicine, autonomous driving, and industry. Typically, in computer vision technology, images describing information about the external environment are first acquired through camera sensors, and then these images are processed based on algorithms within computer vision technology.

[0061] For example, Figure 2 This is a structural schematic diagram of the image processing system provided in this application. Figure 2 As shown, the image processing system 200 includes a camera sensor 201 and a processing module 202.

[0062] The camera sensor 201 is used to acquire images that describe information about the external environment and to input these images into the processing module 202. It should be understood that YUV and RGB formats are two common image formats. Images stored in YUV format are called YUV images, and images stored in RGB format are called RGB images. In most cases, to reduce storage pressure, the camera sensor 201 typically uses the YUV format to store the image information. For details regarding YUV and RGB images, please refer to the aforementioned terminology; further explanation is not provided here.

[0063] The processing module 202 is used to receive the image sent by the camera sensor 201 and process the image using a preset processing algorithm.

[0064] It should be noted that the specific form of the camera sensor 201 is not limited in the embodiments of this application. For example, it can be a fisheye camera, an eagle-eye camera, or a monocular or binocular camera, which does not constitute a limitation of this application.

[0065] It should also be noted that the embodiments of this application do not limit the specific application scenarios of the image processing system 200 described above. For example, the image processing system 200 can be applied to autonomous driving scenarios. Further, in this scenario, after receiving the image sent by the camera sensor 201, the processing module 202 uses a preset processing algorithm to perform processing based on the image, such as making further decisions on autonomous driving behavior based on the image, which does not constitute a limitation of this application. Of course, the embodiments of this application can also be applied to other systems that use visual recognition processing.

[0066] However, for Figure 2 The image processing system shown typically captures images with some degree of distortion from the camera sensor 201. For example, Figure 3 A schematic diagram illustrating image distortion is provided. Figure 3 (a) is a structural schematic diagram of pincushion deformity. Figure 3 (b) is a structural schematic diagram of barrel distortion. It can be seen that if this distorted image is directly used as input to the processing algorithm, it may affect the accuracy of the subsequent image processing results.

[0067] For example, in the field of autonomous driving, various types of camera sensors, such as fisheye cameras, eagle-eye cameras, or monocular and binocular cameras, are currently being used in autonomous driving devices. These devices can use computing platforms, such as mobile data centers (MDCs), domain controllers, and electronic control units, to train a decision-making model from the large number of images captured by the camera sensors using machine learning algorithms. This model is then used to make decisions based on the real-time images acquired by the camera sensors. However, because the images acquired by the camera sensors contain a certain degree of distortion, directly inputting distorted images into the machine learning algorithm during model training may affect the accuracy of the trained model. Furthermore, this could affect the accuracy of the decision-making results when the model makes decisions based on the real-time images acquired by the camera sensors.

[0068] Therefore, the distorted image is usually first processed to remove distortion, also known as correction. Then, the distorted image is used as the input of a preset processing algorithm to improve the accuracy of the processing results obtained by the subsequent preset processing algorithm when processing the image.

[0069] For example, distortion correction can be performed on the initial image captured by the camera sensor based on the RGB mapping relationship provided by the camera manufacturer. The RGB mapping relationship indicates the corresponding position of each pixel in the distorted target RGB image within the initial RGB image. However, since the initial image captured by the camera sensor is usually a YUV image, and the RGB mapping relationship refers to both the distorted target RGB image and the initial RGB image, the following steps can be taken: First, the initial YUV image captured by the camera sensor can be converted to its color gamut to obtain the initial RGB image; then, the RGB mapping relationship provided by the camera manufacturer can be obtained; finally, the pixel value of each pixel in the distorted target RGB image can be taken from its corresponding position in the initial RGB image, as indicated by the RGB mapping relationship, to obtain the pixel value of each pixel in the distorted target RGB image, thus obtaining the distorted image.

[0070] Specifically, when converting between the initial YUV image and the initial RGB image, the conversion can be performed based on formula (I).

[0071]

[0072] Among them, Y x U represents the value of the Y component at pixel x. x V represents the value of the U component at pixel x. x r represents the value of the V component at pixel x. xg represents the value of the R component at pixel x. x b represents the value of the G component at pixel x. x This represents the value of the Y component at pixel x.

[0073] However, converting an initial YUV image to an initial RGB image is very time-consuming, making the entire distortion correction process very time-consuming. For example, it typically takes 2 milliseconds (ms) to convert a 2-megabyte (M) initial YUV image to a 1024*768 initial RGB image, and 8 ms to convert an 8-megabyte (M) initial YUV image to a 1024*768 initial RGB image.

[0074] Therefore, how to reduce the time spent in the distortion removal process has become an urgent technical problem to be solved.

[0075] In view of this, this application proposes a novel distortion correction method, namely, a novel image processing method. In the technical solution proposed in this application, the processing module can first determine the position (i.e., YUV mapping relationship) of each pixel in the distorted image in the initial YUV image based on the RGB mapping relationship. Then, after acquiring the first YUV image, based on the YUV mapping relationship, the pixel value is taken from the corresponding position in the first YUV image, thereby obtaining the pixel value at each position in the distorted second YUV image, i.e., obtaining the distorted image.

[0076] It is understood that in the technical solution provided in this application, since the YUV mapping relationship directly indicates the position of the distorted target image on the initial YUV image, the conversion process from the initial YUV image to the initial RGB image is no longer required, thereby reducing the time consumed in the distortion removal process. Furthermore, it is understood that the more images that need distortion removal, the more significant the effect of the distortion removal method provided in this application in reducing the time consumed in the distortion removal process.

[0077] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0078] Figure 4 This is a schematic flowchart illustrating an image processing method according to one embodiment of this application. Figure 4 As shown, the method in this embodiment may include S401, S402, and S403. This image processing method can be... Figure 2 The processing module in the image processing system shown is used to perform the operation.

[0079] S401, Obtain the first YUV image.

[0080] It should be understood that most camera sensors typically use the YUV format to store images or videos to reduce storage pressure. In this embodiment, the first YUV image refers to an image captured by a camera sensor and stored in YUV format, such as an image captured by a fisheye camera, eagle-eye camera, or monocular / dual-lens camera. The concept of YUV can be found in descriptions in related technologies and will not be repeated here.

[0081] It should be noted that the embodiments of this application do not limit the method of obtaining the first YUV image, and can be determined according to the specific scenario.

[0082] For example, if the processing module needs to make decisions based on the external environment in real time, then as long as the camera sensor acquires the first YUV image, it can send the first YUV image to the processing module so that the processing module can process the first YUV image acquired by the camera sensor in real time.

[0083] For example, if the processing module only needs to make decisions based on the external environment during a specific time period, then the processing module can send a request message to the camera sensor during that specific time period. After receiving the request message from the processing module, the camera sensor will then send the acquired first YUV image to the processing module, so that the processing module can process the first YUV image acquired by the camera sensor during that specific time period.

[0084] It should be noted that in this embodiment of the application, the first YUV image is also referred to as the first original YUV image or the first initial YUV image.

[0085] S402. Obtain the YUV mapping relationship, which is used to indicate the position mapping relationship between the pixels in the initial YUV image and each pixel in the target YUV image. The target YUV image is the image obtained by performing distortion removal processing on the initial YUV image.

[0086] Typically, the initial image acquired by a camera sensor contains some degree of distortion, such as linear distortion, barrel distortion, or pincushion distortion. Therefore, in some image processing algorithms, distortion correction processing is usually performed on the initial image acquired by the camera sensor to avoid affecting the accuracy of subsequent processing results.

[0087] In this embodiment, if the camera sensor acquires an initial YUV image, the image obtained after distortion correction processing of the initial YUV image is referred to as the target YUV image. It should be noted that, in this application, the target YUV image is also referred to as the distortion-corrected YUV image.

[0088] In this embodiment, the YUV mapping relationship refers to the positional mapping relationship between each pixel in the initial YUV image and each pixel in the target YUV image (the distortion-free YUV image). That is, through the YUV mapping relationship, the specific position of each pixel in the distortion-free YUV image in the initial YUV image can be determined.

[0089] Understandably, since the YUV mapping relationship indicates the specific location of each pixel in the distorted YUV image within the initial YUV image, once the YUV mapping relationship is established, for each pixel in the distorted YUV image, the specific location of each pixel in the initial YUV image can be determined first; then, the pixel value at that specific location in the initial YUV image can be determined as the pixel value of that pixel in the target YUV image.

[0090] In one possible implementation, the YUV mapping relationship can be obtained through the RGB mapping relationship. Specifically, obtaining the YUV mapping relationship may include: obtaining the RGB mapping relationship, which indicates the positional mapping relationship between each pixel in the target RGB image and the pixels in the initial RGB image, wherein the initial RGB image is an image obtained by performing RGB conversion on the initial YUV image, and the target RGB image is an image obtained by performing distortion correction processing on the initial RGB image, wherein the target RGB image and the target YUV image have the same size; and determining the YUV mapping relationship based on the RGB mapping relationship.

[0091] In this embodiment, the equal size of the target RGB image and the target YUV image means that the number of pixels in the horizontal direction of the target RGB image is the same as the number of pixels in the horizontal direction of the target YUV image, and the number of pixels in the vertical direction of the target RGB image is the same as the number of pixels in the vertical direction of the target YUV image.

[0092] In this implementation scheme, the initial RGB image refers to the image corresponding to the initial YUV image acquired by the camera sensor after color gamut conversion. It should be noted that the method for converting a YUV image to an RGB image can be found in related technical documents and will not be repeated here.

[0093] Typically, camera manufacturers can almost predict the distortion that will occur in a camera sensor during manufacturing. Therefore, to address this distortion issue, they usually provide a mapping relationship (RGB mapping) to reflect the corresponding pixel positions in the initial RGB image and the distorted RGB image. For example, it can indicate the position of each pixel in the distorted RGB image (i.e., the target RGB image) within the initial RGB image, or vice versa. The position can be represented by coordinate values ​​or by the offset of each pixel in the initial or distorted RGB image relative to its ordinate or abscissa in another RGB image. Thus, given a potentially distorted initial RGB image, to obtain the distorted RGB image, the specific position of each pixel in the distorted RGB image within the initial RGB image can be determined based on the RGB mapping relationship. Then, the pixel value at that specific position in the initial RGB image is used as the pixel value of the corresponding pixel in the distorted RGB image.

[0094] It should be noted that the RGB mapping relationship may be different for different camera sensors.

[0095] It should also be noted that this embodiment does not limit the specific form of the RGB mapping relationship. For example, in one possible approach, the RGB mapping relationship can be represented in the form of a mapping table. Specifically, in implementation, one mapping table can indicate the horizontal coordinate value of each pixel in the distorted RGB image in the initial RGB image, while another mapping table can indicate the vertical coordinate value of each pixel in the distorted RGB image in the initial RGB image.

[0096] For example, taking a 2x4 RGB image after distortion correction as an example, meaning the RGB image after distortion correction includes 8 pixels, Table 1 indicates the x-coordinate value of each of these 8 pixels in the initial RGB image, as shown in Table 1. Table 2 indicates the y-coordinate value of each of these 8 pixels in the initial RGB image, as shown in Table 2.

[0097] Table 1

[0098]

[0099] Table 2

[0100]

[0101] In the RGB image after distortion correction, (X1, Y1) constitutes the first pixel of the first row, (X2, Y1) constitutes the second pixel of the first row, (X3, Y1) constitutes the third pixel of the first row, (X4, Y1) constitutes the fourth pixel of the first row, (X1, Y2) constitutes the first pixel of the second row, (X2, Y2) constitutes the second pixel of the second row, (X3, Y2) constitutes the third pixel of the second row, and (X4, Y2) constitutes the fourth pixel of the second row. It can be seen that the pixel value at each pixel in the distorted image can be determined using Tables 1 and 2. It should be noted that Tables 1 and 2 are merely examples and are not intended to be limiting. In one possible implementation, the vertical and horizontal coordinate values ​​can be combined into a single table; in other possible implementations, offset values ​​can be used to indicate the corresponding coordinate values.

[0102] It is understandable that RGB images can be divided into R-channel images, G-channel images, and B-channel images. Similarly, YUV images can be divided into Y-channel images, U-channel images, and V-channel images.

[0103] Typically, when storing RGB format images, the pixel values ​​of the R components of each pixel in the R channel image are stored first, then the pixel values ​​of the G components of each pixel in the G channel image are stored, and finally the pixel values ​​of the B components of each pixel in the B channel image are stored. YUV format images, on the other hand, are generally stored in two ways. The first is packed storage, which involves alternating the Y, U, and V components and storing them contiguously in units of pixels; the second is planar storage, which involves using three separate arrays to store the Y, U, and V components contiguously.

[0104] To facilitate understanding, we will use the storage of a 4x4 color image in RGB format and in YUV format as examples. It's clear that the first "4" in a 4x4 color image refers to the image comprising 4 pixels horizontally, and the second "4" refers to the image comprising 4 pixels vertically.

[0105] Figure 5 This is a structural diagram illustrating the storage of a 4x4 size color image provided in this application in RGB format. (See diagram below.) Figure 5As shown, when storing a 4x4 image in RGB format, the pixel values ​​of the R components of each pixel in the 4x4 R channel (i.e., R1 to R16 in the figure) are usually stored first, then the pixel values ​​of the G components of each pixel in the 4x4 G channel (i.e., G1 to G16 in the figure) are stored, and finally the pixel values ​​of the B components of each pixel in the 4x4 B channel (i.e., B1 to B16 in the figure) are stored.

[0106] Figure 6 This is a structural diagram illustrating the storage of a 4x4 size color image provided in this application in YUV format. It should be noted that... Figure 6 The following explanation uses a YUV image obtained when the sampling frequencies of the Y, U, and V components satisfy a 4:2:0 relationship. It should be understood that for YUV 4:2:0, each pixel corresponds to one Y component, and every four Y components share one UV component. Therefore, when storing a 4x4 color image in YUV format, it includes a total of 16 Y components, 4 U components, and 4 V components. Specifically, when using a planar storage method, as... Figure 6 As shown in (a), typically, the values ​​of 16 Y components are stored first. Let's assume these 16 Y components are named Y1, Y2, Y3, Y4, Y5, Y6, Y7, Y8, Y9, Y10, Y11, Y12, Y13, Y14, Y15, and Y16, respectively. After storing the values ​​of the 16 Y components, the values ​​of 4 U components are then stored. U1 is the U component shared by Y1, Y2, Y5, and Y6; U2 is the U component shared by Y3, Y4, Y7, and Y8; and U3 is the Y component shared by Y4, Y7, and Y8. 9. The U component shared by Y10, Y13, and Y14; U4 is the U component shared by Y11, Y12, Y15, and Y16. Finally, after storing the values ​​of the four U components, the values ​​of the four V components are stored. V1 is the V component shared by Y1, Y2, Y5, and Y6; V2 is the V component shared by Y3, Y4, Y7, and Y8; V3 is the V component shared by Y9, Y10, Y13, and Y14; and V4 is the V component shared by Y11, Y12, Y15, and Y16. When storing a 4x4 color image using a packing storage method, such as... Figure 6 As shown in (b), the values ​​of the 16 Y components are stored first, and then after storing the values ​​of the 16 Y components, the 4 U components and 4 V components are interleaved.

[0107] It should be noted that the 4*4 size color image described above is merely an example and does not constitute a limitation of this application.

[0108] For example, for any color image of size L*W, where L means the image includes L pixels in the horizontal direction and W means the image includes W pixels in the vertical direction.

[0109] Understandably, when storing a color image of size L*W using the RGB format, the pixel values ​​of the L*W R components are usually stored first, followed by the L*W G components, and finally the L*W B components.

[0110] When storing L*W size color images using the YUV format, both planar storage and packed storage methods can be used. It's understandable that in the YUV 4:2:0 case, since each pixel corresponds to one Y component and every four Y components share one UV component, an L*W color image includes a total of L*W Y components, L*W / 4 U components, and L*W / 4 V components. Therefore, when storing an L*W size color image using planar storage, the pixel values ​​of the L*W Y components are stored first, followed by the L*W / 4 U components, and finally the L*W / 4 V components. When storing an L*W size color image using packing storage, the pixel values ​​of the L*W Y components are still stored first, followed by the L*W / 4 U and L*W / 4 V components, which are then interleaved.

[0111] It can be seen that when storing images in YUV format, the storage method for the Y channel is the same regardless of whether a planar storage method or a packed storage method is used. It can also be seen that for RGB and YUV images, the storage method for the Y channel in YUV format is the same as the storage method for the R / G / B channels in RGB format. The difference lies in the storage methods for the U and V channels in YUV format, but there is a corresponding relationship between the storage methods for the U and V channels and the Y channel in YUV format.

[0112] Therefore, in one possible implementation, when determining the YUV mapping relationship, a first target mapping relationship, a second target mapping relationship, and a third target mapping relationship can be determined based on the RGB mapping relationship. The first target mapping relationship indicates the position of each pixel in the target YUV image within the Y channel image corresponding to the initial YUV image; the second target mapping relationship indicates the position of each pixel in the target image within the U channel image corresponding to the initial YUV image; and the third target mapping relationship indicates the position of each pixel in the target YUV image within the V channel image corresponding to the initial YUV image. Specifically, since the RGB mapping relationship indicates the position of each pixel in the target RGB image within the R / G / B channels of the initial RGB image, and the storage method of the Y channel in the YUV format is the same as that of the R / G / B channels in the RGB format, it can be directly determined that the first target mapping relationship is the same as the RGB mapping relationship, i.e., the position of each pixel in the target YUV image within the Y channel image corresponding to the initial YUV image can be directly determined.

[0113] Furthermore, for each pixel in the target YUV image, since there is a correspondence between the arrangement of the Y component and the U / V components, after determining that the first target mapping relationship is an RGB mapping relationship, the second target mapping relationship can be determined based on the RGB mapping relationship and the correspondence between the U and Y components, and the third target mapping relationship can be determined based on the RGB mapping relationship and the correspondence between the V and Y components. Specifically, based on the RGB mapping relationship and the correspondence between the U and Y components, the abscissa and ordinate values ​​of the first pixel in the target YUV image in the U channel image corresponding to the initial YUV image are determined, and based on the RGB mapping relationship and the correspondence between the V and Y components, the abscissa and ordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image are determined. Here, the first pixel is any pixel among all pixels.

[0114] For ease of understanding, Figure 5 and Figure 6 (a) To illustrate in conjunction. Now assume... Figure 5 The image shown is an RGB image after distortion correction based on the existing RGB mapping relationship. Figure 6 The image shown in (a) is a distortion-free YUV image, where the existing RGB mapping indicates... Figure 5 The position (horizontal and vertical coordinates) of each pixel in the distorted RGB image shown (corresponding one-to-one with R1 to R16 respectively) in the original initial RGB image.

[0115] like Figure 5 and6 As shown in (a), the storage method of each pixel in the distorted YUV image (corresponding one-to-one with Y1 to Y16) is exactly the same as the storage method of each pixel in the distorted RGB image (corresponding one-to-one with R1 to R16). Therefore, for each pixel represented by Y1 to Y16 in the distorted YUV image, its position in the initial YUV image is the same as the position of each pixel in the distorted RGB image in the original initial RGB image (the image after color gamut conversion of the initial YUV image). That is, the existing RGB mapping relationship can be directly used to indicate the position of each pixel in the distorted YUV image (corresponding one-to-one with R1 to R16) in the original initial YUV image. Then, based on the storage correspondence between the Y component, U component, and V component, the horizontal and vertical coordinate values ​​of each pixel in the U channel image corresponding to the initial YUV image, and the horizontal and vertical coordinate values ​​of each pixel in the V channel image corresponding to the initial YUV image can be determined.

[0116] For example, in order to determine Figure 6 The position (x-coordinate and y-coordinate) of the U1 component in the initial YUV image in (a) can be determined based on the positions of the Y1, Y2, Y5, and Y6 components in the initial YUV image, since the positions of the Y1, Y2, Y5, and Y6 components in the initial YUV image have already been determined according to the RGB mapping relationship.

[0117] S403. Obtain a second YUV image based on the YUV mapping relationship and the first YUV image.

[0118] Since the YUV mapping relationship indicates the positional mapping relationship between each pixel in the target YUV image and the pixels in the initial YUV image, it can be considered that the YUV mapping relationship indicates the position of each pixel in the target YUV image in the initial YUV image.

[0119] Therefore, in this embodiment, after acquiring the first YUV image, in order to obtain the distortion-free image (i.e., the second YUV image), the position of each pixel in the second YUV image in the first YUV image can be determined based on the YUV mapping relationship. Specifically, based on the YUV mapping relationship, the positions of each pixel in the second YUV image in the corresponding Y channel image, U channel image, and V channel image of the first YUV image are determined. Then, based on the pixel values ​​of each pixel in the second YUV image at its corresponding positions in the Y channel image, U channel image, and V channel image of the first YUV image, the pixel values ​​of each pixel in the second YUV image are obtained, thereby obtaining the second YUV image.

[0120] It should be noted that since the positions of some pixels indicated in the RGB mapping relationship are floating-point values ​​in the initial RGB image, bilinear interpolation is usually required to obtain the pixel values ​​at the corresponding positions. The concept and detailed implementation process of bilinear interpolation can be found in relevant technical descriptions, and will not be elaborated upon here.

[0121] The image processing method provided in this application, due to the positional mapping relationship between each pixel in the target YUV image and the pixels in the initial YUV image, can directly perform distortion correction on the first YUV image after acquisition, without needing to first convert the first YUV image to an RGB image and then perform distortion correction on the RGB image based on the RGB mapping relationship. Therefore, the time consumed in the distortion correction process can be reduced. It is understood that the more YUV images that need distortion correction, the more significant the reduction in distortion correction time will be.

[0122] As described in the foregoing embodiments, when storing RGB and YUV format images, the storage method of the R channel in the RGB image is the same as the storage method of the Y component in the YUV format. The difference lies in the storage methods of the U and V components in the YUV format image. Therefore, once the RGB mapping relationship is obtained, it can be considered that the RGB mapping relationship is equivalent to indicating the position of each pixel in the target YUV image in the Y channel image of the initial YUV image. Furthermore, it is only necessary to determine the position of each pixel in the target YUV image in the U channel image of the initial YUV image and the position of each pixel in the target YUV image in the V channel image of the initial YUV image based on the correspondence between the storage methods of the U and Y components in the YUV image, and based on the correspondence between the storage methods of the V and Y components in the YUV image. Specifically, based on the RGB mapping relationship, the x-coordinate and y-coordinate values ​​of the first pixel in the target YUV image in the U channel image corresponding to the initial YUV image, and the x-coordinate and y-coordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image, can be determined. Here, the first pixel is any pixel among all pixels.

[0123] The following examples illustrate how to obtain the YUV mapping relationship, with the sampling frequencies of the Y, U, and V components of the first YUV image satisfying a 4:2:0 relationship and the first YUV image being stored in a planar storage mode. A second example demonstrates how to obtain the YUV mapping relationship using a packed storage mode.

[0124] Next, we will introduce how the sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and how to obtain the YUV mapping relationship when the first YUV image is stored in a planar storage mode.

[0125] From the above embodiments Figure 5 and Figure 6 As can be seen, the arrangement of the R components is the same as that of the Y components. Therefore, for the first target mapping relationship that indicates the position of each pixel in the target YUV image in the Y channel image corresponding to the initial YUV image, the RGB mapping relationship can be used directly.

[0126] Furthermore, as can be seen from the above embodiments, when YUV images are stored in a planar storage manner, every four Y components share one U component and one V component. That is, it can be considered that one U component corresponds to four Y components in the Y channel, for example, as... Figure 6 As shown in (a), U1 corresponds to Y1, Y2, Y5 and Y6; U2 corresponds to Y3, Y4, Y7 and Y8; U3 corresponds to Y9, Y10, Y13 and Y14; and U4 corresponds to Y11, Y12, Y15 and Y16.

[0127] Therefore, in this embodiment, it can be based on the formula: Determine the x-coordinate value of the first pixel in the target YUV image within the corresponding U-channel image of the initial YUV image. Here, temp1 represents the sum of the x-coordinate values ​​of the first, second, third, and fourth pixels in the initial RGB image, where the second, third, and fourth pixels are adjacent to the first pixel. According to the formula: Determine the ordinate value of the first pixel in the U channel image corresponding to the initial YUV image, where src h The height of the initial YUV image is represented by k, and the ratio is represented by k. temp2 represents the sum of the ordinate values ​​of the first, second, third, and fourth pixels in the initial RGB image; correspondingly, the x and y coordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image are determined, including: according to the formula: Determine the x-coordinate value of the first pixel in the V channel image corresponding to the first image; according to the formula: Determine the ordinate value of the first pixel in the V channel image corresponding to the first image.

[0128] For example, for Figure 6 (a) Assuming the 4*4 YUV image is shown, the RGB mapping table indicates the positions of four pixels (let's call them pixel Y1, pixel Y2, pixel Y5, and pixel Y6) that correspond one-to-one with the Y1, Y2, Y5, and Y6 components in the initial RGB image. Let's assume that the specific position of pixel Y1 in the initial YUV image is represented as Y11, the specific position of pixel Y2 in the initial YUV image is represented as Y12, the specific position of pixel Y5 in the initial YUV image is represented as Y21, and the specific position of pixel Y6 in the initial YUV image is represented as Y22.

[0129] Since pixels Y1, Y2, Y5, and Y6 correspond to the same U and V components, the following formula can be used to determine the positions of the U1 and V1 components corresponding to pixels Y1, Y2, Y5, and Y6 in the initial YUV image:

[0130]

[0131] Among them, Y11 x Y1 represents the x-coordinate of pixel Y1 in the initial RGB image, and Y12 represents the x-coordinate of pixel Y1. x Y2 represents the x-coordinate of pixel Y2 in the initial RGB image, Y21 x Y5 represents the x-coordinate of pixel Y5 in the initial RGB image, Y22 x Y11 represents the x-coordinate of pixel Y6 in the initial RGB image. y Y1 represents the ordinate value of pixel Y1 in the initial RGB image, and Y12 represents the ordinate value of pixel Y1. y Y2 represents the ordinate value of pixel Y2 in the initial RGB image, Y21 y Y5 represents the ordinate value of pixel Y5 in the initial RGB image, Y22 y U represents the ordinate value of pixel Y6 in the initial RGB image. 1x This represents the x-coordinate values ​​of pixels Y1, Y2, Y5, and Y6 in the U-channel image corresponding to the initial YUV image. 1y V represents the ordinate values ​​of pixels Y1, Y2, Y5, and Y6 in the U-channel image corresponding to the initial YUV image. 1x This represents the x-coordinate values ​​of pixels Y1, Y2, Y5, and Y6 in the V channel image corresponding to the initial YUV image. 1y The src represents the ordinate values ​​of pixels Y1, Y2, Y5, and Y6 in the V channel image corresponding to the initial YUV image. h This indicates the height of the initial RGB image.

[0132] In this embodiment, when the YUV image is stored in a planar storage manner, the specific position of each pixel in the target YUV image in the Y channel, U channel and V channel of the initial YUV image can be determined based on the RGB mapping relationship. Thus, the distortion-free YUV image can be obtained without converting the initial YUV image to the initial RGB image.

[0133] The following section describes how the sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and how to obtain the YUV mapping relationship when the first YUV image is stored in a packaged storage mode.

[0134] pass Figure 6 (a) and Figure 6 (b) It can be seen that the difference between the packaged storage method and the planar storage method lies in the different arrangement of the U component and the V component.

[0135] As can be seen, for the packaged storage method, the U component and V component are arranged in an alternating manner. Therefore, the vertical coordinate values ​​of the U component and V component are the same, and the horizontal coordinate value of the U component and the vertical coordinate value of the V component for each pixel point differ by one pixel point.

[0136] Therefore, in this scenario, we can use the formula: Determine the x-coordinate value of the first pixel in the target YUV image within the corresponding U-channel image of the initial YUV image, where temp1 represents the sum of the x-coordinate values ​​of the first, second, third, and fourth pixels in the initial RGB image. The second, third, and fourth pixels are adjacent to the first pixel. According to the formula: Determine the ordinate value of the first pixel in the U-channel image corresponding to the initial YUV image, where srch represents the height of the initial YUV image and k represents the ratio. temp2 represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image; correspondingly, the x and y coordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image are determined, including: according to the formula: V x =U x +1, determine the x-coordinate value of the first pixel in the V channel image corresponding to the initial YUV image; according to the formula: Determine the ordinate value of the first pixel in the V channel image corresponding to the initial YUV image.

[0137] For example, for Figure 6(b) shows a 4*4 YUV image. Assume the RGB mapping table indicates the positions of four pixels (let's call them pixel Y1, pixel Y2, pixel Y5, and pixel Y6) corresponding to the Y1, Y2, Y5, and Y6 components in the initial RGB image. Assume the specific position of pixel Y1 in the initial YUV image is represented as Y11, pixel Y2 as Y12, pixel Y5 as Y21, and pixel Y6 as Y22. Since pixels Y1, Y2, Y5, and Y6 correspond to the same U1 and V1 components, the following formula can be used to determine the positions of the U1 and V1 components corresponding to pixels Y1, Y2, Y5, and Y6 in the initial YUV image:

[0138]

[0139] Among them, Y11 x Y1 represents the x-coordinate of pixel Y1 in the initial RGB image, and Y12 represents the x-coordinate of pixel Y1. x Y2 represents the x-coordinate of pixel Y2 in the initial RGB image, Y21 x Y5 represents the x-coordinate of pixel Y5 in the initial RGB image, Y22 x Y11 represents the x-coordinate of pixel Y6 in the initial RGB image. y Y1 represents the ordinate value of pixel Y1 in the initial RGB image, and Y12 represents the ordinate value of pixel Y1. y Y2 represents the ordinate value of pixel Y2 in the initial RGB image, Y21 y Y5 represents the ordinate value of pixel Y5 in the initial RGB image, Y22 y U represents the ordinate value of pixel Y6 in the initial RGB image. 1x This represents the x-coordinate values ​​of pixels Y1, Y2, Y5, and Y6 in the U-channel image corresponding to the initial YUV image. 1y V represents the ordinate values ​​of pixels Y1, Y2, Y5, and Y6 in the U-channel image corresponding to the initial YUV image. 1x This represents the x-coordinate values ​​of pixels Y1, Y2, Y5, and Y6 in the V channel image corresponding to the initial YUV image. 1y The src represents the ordinate values ​​of pixels Y1, Y2, Y5, and Y6 in the V channel image corresponding to the initial YUV image. h This indicates the height of the initial RGB image.

[0140] It should be understood that because the U and Y components corresponding to the packaged storage method are interleaved during storage, when the positions of the U and V components calculated through the above mapping relationship are floating-point values, these values ​​may have a certain degree of deviation during quantization. Moreover, since the difference between the values ​​of the U and V components is large, this error will lead to a large image output error. In view of this, in the embodiments of this application, using... Figure 7 For example, this application describes a method for solving the pixel value of the U component corresponding to the first pixel in the initial YUV image when the horizontal coordinate value and / or vertical coordinate value of the U component corresponding to the first pixel in the initial YUV image are floating-point values.

[0141] like Figure 7 As shown, the U1 component and the U2 component include the V1 component, and the U3 component and the U4 component include the V2 component. The position of the black circle indicates the position of the first pixel in the target YUV image obtained through the YUV mapping table, corresponding to the U component in the initial YUV image. Therefore, in this embodiment, it can be determined according to the formula:

[0142]

[0143] Obtain the pixel value of the U component corresponding to the first pixel in the initial YUV image.

[0144] Where x represents the horizontal coordinate of the U component corresponding to the first pixel in the initial YUV image, y represents the vertical coordinate of the U component corresponding to the first pixel in the initial YUV image, f(x,y) represents the pixel value of the U component corresponding to the first pixel in the initial YUV image, U1, U2, U3 and U4 are the positions of the four U components in the initial YUV image that are closest to the point corresponding to the position represented by (x,y), f(U1) represents the pixel value at U1, f(U2) represents the pixel value at U2, f(U3) represents the pixel value at U3, and f(U4) represents the pixel value at U4.

[0145] It should be noted that the above only describes how to calculate the pixel value of the U component corresponding to the first pixel in the initial YUV image when the x-coordinate and / or y-coordinate of the first pixel in the corresponding U channel image of the initial YUV image are floating-point values. It should be understood that when the x-coordinate and / or y-coordinate of the first pixel in the corresponding V channel image of the initial YUV image are floating-point values, the same approach as described above can be used to calculate the pixel value of the U component corresponding to the first pixel in the initial YUV image, and will not be elaborated further here.

[0146] The foregoing mainly describes the solutions provided by the embodiments of this application. Those skilled in the art should readily recognize that, based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0147] This application embodiment can divide each device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0148] When dividing each function into modules according to its corresponding function. Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this application. Figure 8 As shown, the device 800 includes an acquisition module 801 and a processing module 802.

[0149] The acquisition module 801 is used to acquire a first YUV image and acquire a YUV mapping relationship. The YUV mapping relationship is used to indicate the position mapping relationship between each pixel in the initial YUV image and each pixel in the target YUV image. The target YUV image is an image obtained by performing distortion removal processing on the initial YUV image. The processing module 802 is used to obtain a second YUV image based on the YUV mapping relationship and the first YUV image.

[0150] In one possible implementation, the acquisition module 801 is further configured to: acquire an RGB mapping relationship, wherein the RGB mapping relationship indicates the positional mapping relationship between pixels in the initial RGB image and each pixel in the target RGB image, wherein the initial RGB image is an image obtained by performing RGB conversion on the initial YUV image, and the target RGB image is an image obtained by performing distortion correction processing on the initial RGB image, wherein the target RGB image and the target YUV image are of equal size; the processing module 802 is further configured to: determine the YUV mapping relationship based on the RGB mapping relationship.

[0151] In one possible implementation, the processing module 802 is further configured to: determine a first target mapping relationship based on the RGB mapping relationship, wherein the first target mapping relationship indicates the position of each pixel in the target YUV image in the Y channel image corresponding to the initial YUV image; determine a second target mapping relationship based on the RGB mapping relationship, wherein the second target mapping relationship indicates the position of each pixel in the target image in the U channel image corresponding to the initial YUV image; determine a third target mapping relationship based on the RGB mapping relationship, wherein the third target mapping relationship indicates the position of each pixel in the target image in the V channel image corresponding to the initial YUV image; correspondingly, the processing module 802 is further configured to: determine the pixel value of each pixel in the second YUV image based on the pixel value of each pixel at its position in the Y channel image corresponding to the first YUV image, the pixel value at its position in the U channel image corresponding to the first YUV image, and the pixel value at its position in the V channel image corresponding to the first YUV image.

[0152] In one possible implementation, the first target mapping relationship is the same as the RGB mapping relationship; correspondingly, the processing module 802 is further configured to: determine the horizontal and vertical coordinate values ​​of a first pixel in the target YUV image in the U channel image corresponding to the initial YUV image according to the RGB mapping relationship, wherein the first pixel is any one of the pixels; and determine the horizontal and vertical coordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image according to the RGB mapping relationship.

[0153] In one possible implementation, the sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and the first YUV image is stored in a planar storage mode.

[0154] In one possible implementation, the processing module 802 is further configured to: according to the formula: Determine the horizontal coordinate value of the first pixel in the U channel image corresponding to the initial YUV image, where temp1 represents the sum of the horizontal coordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image, wherein the second, third, and fourth pixels are adjacent to the first pixel; according to the formula: Determine the ordinate value of the first pixel in the U-channel image corresponding to the initial YUV image, where srch The initial YUV image represents the height, and k represents the ratio. temp2 represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image; correspondingly, the processing module 802 is further configured to: according to the formula: Determine the x-coordinate value of the first pixel in the V-channel image corresponding to the first image; according to the formula: Determine the ordinate value of the first pixel in the V channel image corresponding to the first image.

[0155] In one possible implementation, the sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and the first YUV image is stored in a packaged storage method.

[0156] In one possible implementation, the processing module 802 is further configured to: according to the formula: Determine the horizontal coordinate value of the first pixel in the U channel image corresponding to the initial YUV image, where temp1 represents the sum of the horizontal coordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image, wherein the second, third, and fourth pixels are adjacent to the first pixel; according to the formula: Determine the ordinate value of the first pixel in the U-channel image corresponding to the initial YUV image, where src h The initial YUV image represents the height, and k represents the ratio. temp2 represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image; correspondingly, the processing module 802 is further configured to: according to the formula: V x =U x +1, determine the x-coordinate value of the first pixel in the V channel image corresponding to the initial YUV image; according to the formula: Determine the ordinate value of the first pixel in the V channel image corresponding to the initial YUV image.

[0157] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in another embodiment of this application. Figure 9 The apparatus shown can be used to perform the image processing method described in any of the foregoing embodiments.

[0158] like Figure 9 As shown, the device 900 in this embodiment includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, processor 902, and communication interface 903 are interconnected via the bus 904.

[0159] The memory 901 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 901 can store programs, and when the program stored in the memory 901 is executed by the processor 902, the processor 902 performs the execution... Figure 4 The steps of the method shown.

[0160] The processor 902 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, used to execute relevant programs to implement the methods in the various embodiments of this application.

[0161] The processor 902 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the method in the various embodiments of this application can be accomplished by the integrated logic circuitry in the hardware of the processor 902 or by instructions in software form.

[0162] The processor 902 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0163] The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 901. Processor 902 reads the information in memory 901 and, in conjunction with its hardware, completes the functions required by the units included in the temperature measuring device of this application. For example, it can execute... Figure 4 The various steps / functions of the illustrated embodiment.

[0164] The communication interface 903 can use, but is not limited to, transceivers to enable communication between the device 900 and other devices or communication networks.

[0165] Bus 904 may include a pathway for transmitting information between various components of device 900 (e.g., memory 901, processor 902, communication interface 903).

[0166] It should be understood that the device 900 shown in the embodiments of this application may be an electronic device, or it may be a chip configured in an electronic device.

[0167] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0168] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0169] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0170] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0171] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0172] It should be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0173] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0178] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0179] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, include: Acquire the first YUV image; Obtain the YUV mapping relationship, which is used to indicate the positional mapping relationship between pixels in the initial YUV image and each pixel in the target YUV image, wherein the target YUV image is the image obtained by performing distortion correction processing on the initial YUV image; A second YUV image is obtained based on the YUV mapping relationship and the first YUV image; The process of obtaining the YUV mapping relationship includes: Obtain the RGB mapping relationship, which indicates the position mapping relationship between each pixel in the initial RGB image and each pixel in the target RGB image. The initial RGB image is the image obtained by performing RGB conversion on the initial YUV image. The target RGB image is the image obtained by performing distortion correction processing on the initial RGB image. The target RGB image and the target YUV image are of the same size. The YUV mapping relationship is determined based on the RGB mapping relationship.

2. The method of claim 1, wherein, Determining the YUV mapping relationship based on the RGB mapping relationship includes: Based on the RGB mapping relationship, a first target mapping relationship is determined, wherein the first target mapping relationship indicates the position of each pixel in the target YUV image in the Y channel image corresponding to the initial YUV image; Based on the RGB mapping relationship, a second target mapping relationship is determined, which indicates the position of each pixel in the target YUV image in the U channel image corresponding to the initial YUV image. Based on the RGB mapping relationship, a third target mapping relationship is determined, which indicates the position of each pixel in the target YUV image in the V channel image corresponding to the initial YUV image; Accordingly, obtaining the second YUV image based on the YUV mapping relationship and the first YUV image includes: The pixel value of each pixel in the second YUV image is determined based on the pixel value of each pixel at its position in the Y channel image corresponding to the first YUV image, the pixel value at its position in the U channel image corresponding to the first YUV image, and the pixel value at its position in the V channel image corresponding to the first YUV image.

3. The method of claim 2, wherein, The first target mapping relationship is the same as the RGB mapping relationship; Accordingly, determining the second target mapping relationship based on the RGB mapping relationship includes: Based on the RGB mapping relationship, the horizontal and vertical coordinates of the first pixel in the target YUV image are determined in the U channel image corresponding to the initial YUV image, where the first pixel is any one of the pixels. Accordingly, determining the third target mapping relationship based on the RGB mapping relationship includes: Based on the RGB mapping relationship, the horizontal and vertical coordinates of the first pixel in the V channel image corresponding to the initial YUV image are determined.

4. The method according to any one of claims 1 to 3, characterized in that, The sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and the first YUV image is stored in a planar storage mode.

5. The method of claim 3, wherein, Determining the x-coordinate and y-coordinate values ​​of the first pixel in the target YUV image within the U-channel image corresponding to the initial YUV image includes: A horizontal coordinate value of the first pixel point in a U channel image corresponding to the initial YUV image is determined according to a formula: , wherein, represents a sum of a horizontal coordinate value of the first pixel point in the initial RGB image, a horizontal coordinate value of a second pixel point in the initial RGB image, a horizontal coordinate value of a third pixel point in the initial RGB image, and a horizontal coordinate value of a fourth pixel point in the initial RGB image, the second pixel point, the third pixel point, and the fourth pixel point being adjacent to the first pixel point. According to the formula: Determine the ordinate value of the first pixel in the U channel image corresponding to the initial YUV image, wherein, The initial YUV image represents the height, and k represents the ratio. The This represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image. Accordingly, determining the abscissa and ordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image includes: According to the formula: Determine the horizontal coordinate value of the first pixel in the V channel image corresponding to the initial YUV image; According to the formula: The vertical coordinate value of the first pixel in the V channel image corresponding to the initial YUV image is determined.

6. The method according to any one of claims 1 to 3, characterized in that, The sampling frequencies of the Y component, U component, and V component of the first YUV image satisfy a 4:2:0 relationship, and the first YUV image is stored in a packaged storage mode.

7. The method according to claim 3, characterized in that, Determining the x-coordinate and y-coordinate values ​​of the first pixel in the target YUV image within the U-channel image corresponding to the initial YUV image includes: According to the formula: Determine the horizontal coordinate value of the first pixel in the U channel image corresponding to the initial YUV image, wherein, This represents the sum of the horizontal coordinates of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image, wherein the second pixel, the third pixel, and the fourth pixel are adjacent to the first pixel. According to the formula: Determine the ordinate value of the first pixel in the U channel image corresponding to the initial YUV image, wherein, The initial YUV image represents the height, and k represents the ratio. The This represents the sum of the ordinate values ​​of the first pixel, the second pixel, the third pixel, and the fourth pixel in the initial RGB image. Accordingly, determining the abscissa and ordinate values ​​of the first pixel in the V channel image corresponding to the initial YUV image includes: According to the formula: Determine the horizontal coordinate value of the first pixel in the V channel image corresponding to the initial YUV image; According to the formula: The vertical coordinate value of the first pixel in the V channel image corresponding to the initial YUV image is determined.

8. An image processing apparatus, characterized in that, The apparatus includes a functional module for performing the method as described in any one of claims 1 to 7.

9. A computing platform, characterized in that, The computing platform includes the image processing apparatus of claim 8.

10. A mobile device, characterized in that, The mobile device includes the computing platform as described in claim 9.

11. The device according to claim 10, characterized in that, The mobile devices include autonomous vehicles.

12. An image processing apparatus, characterized in that, include: Memory and processor; The memory is used to store program instructions; The processor is used to invoke program instructions in the memory to execute the image processing method as described in any one of claims 1 to 7.

13. A chip, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being used to run a computer program or instructions to perform the method as described in any one of claims 1 to 7.

14. A computer-readable medium, characterized in that, The computer-readable medium stores program code for computer execution, the program code including instructions for performing the method as described in any one of claims 1 to 7.

15. A computer program product, said computer program product comprising computer program code, characterized in that, When the computer program code is run on a computer, the computer causes the computer to implement the method as described in any one of claims 1 to 7.

Images