Image processing method, device and computer equipment

Abstract

The present application relates to an image processing method, device and computer equipment. The method comprises: obtaining each initial image captured by each lens of a multi-eye device; determining each pixel in a correction reference area based on the position of each pixel to be corrected in each initial image; determining the interpolation coefficient of each pixel in the correction reference area based on each pixel in the correction reference area and the position of each pixel to be corrected; performing interpolation calculation on each pixel in the correction reference area based on each pixel in the correction reference area and the interpolation coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected, and obtaining the interpolation pixel value of each pixel to be corrected. The present method can solve the problem that the existing image processing method has a large amount of calculation and consumes a lot of computing resources in the process of removing water ripples.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus and computer device. Background Art

[0002] With the development of image processing technology, the application of 3D image display technology in the medical field has gradually attracted attention. 3D image display technology is mainly based on the 3D display of the left and right stereo image pairs collected by multi-eye devices to provide a three-dimensional sense of the observation area. When using 3D image display technology to observe the target area, there are high requirements for the image quality of the stereo display. Therefore, it is necessary to calibrate and correct the 3D imaging equipment.

[0003] When performing stereoscopic correction on a three-dimensional imaging device, the existence of distortion correction and epipolar correction may cause water ripples in areas with dense textures in the image content, affecting the display quality of the image. Therefore, it is necessary to remove the water ripples from the images captured by the three-dimensional imaging device to ensure the display quality of the image.

[0004] The existing image processing method mainly performs frequency domain transformation or problem pixel judgment on the image with water ripples, finds the area with water ripples, and then removes the water ripples separately. This method requires finding the water ripple area in the image first and then removing the water ripples, which is computationally intensive and consumes a lot of computing resources.

[0005] With regard to the existing image processing methods, in the process of removing water ripples in images, there is a problem of large amount of calculation and consumption of a large amount of computing resources, and no effective solution has been proposed yet. Summary of the invention

[0006] Based on this, it is necessary to provide an image processing method, apparatus and computer device to address the above technical issues.

[0007] In a first aspect, the present application provides an image processing method. The method comprises the following steps:

[0008] Acquire each initial image captured by each lens of the multi-eye device;

[0009] Based on the position of each pixel point to be corrected in each of the initial images, determining each pixel point in the correction reference area corresponding to each pixel point to be corrected in each of the initial images;

[0010] Based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determine the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected; the interpolation coefficient is a coefficient related to the direction between the pixel point in the correction reference area and the pixel point to be corrected;

[0011] Based on the pixel points in the correction reference area corresponding to the pixel points to be corrected, and the interpolation coefficients of the pixel points in the correction reference area corresponding to the pixel points to be corrected, interpolation calculation is performed on the pixel points in the correction reference area to obtain interpolated pixel values ​​of the pixel points to be corrected.

[0012] In one embodiment, the determining the interpolation coefficients of the pixels in the correction reference area corresponding to the pixels to be corrected based on the pixels in the correction reference area corresponding to the pixels to be corrected and the positions of the pixels to be corrected comprises:

[0013] Based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determine the basis function coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected; the gradient cosine coefficient is a coefficient related to the direction between the pixel point in the correction reference area and the pixel point to be corrected;

[0014] Based on the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, the interpolation coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected are determined.

[0015] In one of the embodiments, before determining each pixel point in the correction reference area corresponding to each pixel point to be corrected in each of the initial images based on the position of each pixel point to be corrected in each of the initial images, the following steps are included:

[0016] Based on the position of each pixel point of each of the initial images and the stereo correction table of the multi-eye device, the position of each pixel point to be corrected in each of the initial images is obtained; the stereo correction table of the multi-eye device is generated based on the internal parameters and external parameters of the multi-eye device, and is a correction table for stereo correction of the position of each pixel point of the initial image.

[0017] In one embodiment, the obtaining the position of each pixel point to be corrected of each initial image based on the position of each pixel point of each initial image and the stereo correction table of the multi-eye device comprises the following steps:

[0018] Determine the position corresponding to each pixel point of each of the initial images in the horizontal coordinate table in the stereo correction table of the multi-eye device as the horizontal coordinate of each of the pixel points to be corrected in each of the initial images;

[0019] Determine the corresponding position of each pixel point of each of the initial images in the longitudinal coordinate table in the stereo correction table of the multi-eye device as the longitudinal coordinate of each pixel point to be corrected in each of the initial images;

[0020] Based on the abscissa and the ordinate of each of the pixels to be corrected in each of the initial images, the position of each of the pixels to be corrected in each of the initial images is determined.

[0021] In one of the embodiments, determining each pixel point in the correction reference area corresponding to each pixel point to be corrected in each of the initial images based on the position of each pixel point to be corrected in each of the initial images comprises the following steps:

[0022] Determining the correction reference area corresponding to each of the pixel points to be corrected based on the position of each of the pixel points to be corrected in each of the initial images;

[0023] Based on the correction reference area corresponding to each of the pixel points to be corrected, each of the pixel points within the correction reference area corresponding to each of the pixel points to be corrected is determined.

[0024] In one of the embodiments, the method of determining the basis function coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected and the gradient cosine coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected based on each pixel in the correction reference area corresponding to each pixel to be corrected and the position of each pixel to be corrected comprises the following steps:

[0025] Determining the distance between each pixel in the correction reference area and the pixel to be corrected corresponding to the correction reference area based on each pixel in the correction reference area corresponding to each pixel to be corrected and the position of each pixel to be corrected;

[0026] Determining the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area;

[0027] Determining direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected;

[0028] Based on the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected is determined.

[0029] In one of the embodiments, the determining the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected based on each pixel in the correction reference area corresponding to each pixel to be corrected and the position of each pixel to be corrected comprises the following steps:

[0030] Determining the positions of the pixels in the correction reference area corresponding to the pixels to be corrected based on the pixels in the correction reference area corresponding to the pixels to be corrected;

[0031] Based on the positions of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the positions of each pixel point to be corrected, the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected is determined.

[0032] In one of the embodiments, the step of determining the gradient cosine coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected based on the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected and the distance between each pixel in the correction reference area and the pixel to be corrected corresponding to the correction reference area comprises the following steps:

[0033] Determine, based on the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected, the direction information of the correction reference area related to the gradient cosine coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected;

[0034] Based on the direction information of the correction reference area related to the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, the distances between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, and the size of the correction reference area corresponding to each pixel point to be corrected, the gradient cosine coefficients of each pixel point in the correction reference area are determined.

[0035] In one of the embodiments, the interpolation calculation is performed on each pixel in the correction reference area based on each pixel in the correction reference area corresponding to each pixel to be corrected and the interpolation coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected to obtain the interpolated pixel value of each pixel to be corrected, including the following steps:

[0036] Based on the pixel values ​​of each pixel in the correction reference area corresponding to each pixel to be corrected and the interpolation coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected, horizontal interpolation calculation is performed on each row of pixel in the correction reference area corresponding to each pixel to be corrected to obtain a horizontal interpolation result of each row of the correction reference area corresponding to each pixel to be corrected; the horizontal interpolation result of each row includes the position and pixel value of each row obtained by horizontal interpolation;

[0037] Determine, based on the horizontal interpolation result of each row of the correction reference area corresponding to each pixel point to be corrected, the interpolation coefficient of the pixel point obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel point to be corrected;

[0038] Based on the pixel values ​​of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel point to be corrected, and the interpolation coefficients of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel point to be corrected, vertical interpolation calculation is performed on the pixels obtained by horizontal interpolation of each row to obtain the interpolated pixel values ​​of each pixel point to be corrected.

[0039] In a second aspect, the present application also provides an image processing device. The device comprises:

[0040] An image acquisition module is used to acquire each initial image captured by each lens of the multi-eye device;

[0041] A pixel point determination module, used to determine each pixel point in a correction reference area corresponding to each pixel point to be corrected in each of the initial images based on the position of each pixel point to be corrected in each of the initial images;

[0042] A coefficient determination module, configured to determine, based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, an interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected; the interpolation coefficient is a coefficient related to the direction between the pixel point in the correction reference area and the pixel point to be corrected;

[0043] And an interpolation calculation module, which is used to perform interpolation calculation on each pixel point in the correction reference area based on each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the interpolation coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, so as to obtain the interpolated pixel value of each pixel point to be corrected.

[0044] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the image processing method described in the first aspect is implemented.

[0045] The above-mentioned image processing method, device and computer equipment determine the various pixel points in the correction reference area corresponding to the various pixel points to be corrected, and then use the various pixel points in the correction reference area corresponding to the various pixel points to be corrected and the positions of the various pixel points to be corrected to determine the interpolation coefficients of the various pixel points in the correction reference area corresponding to the various pixel points to be corrected. It corrects the various pixel points in the reference area for interpolation calculation based on information related to the direction and the pixel values ​​of the various pixel points in the correction reference area corresponding to the pixel points to be corrected, and obtains the interpolation pixel values ​​of the various pixel points to be corrected. It introduces information related to the direction into the calculation process of the interpolation pixel values ​​of the various pixel points to be corrected, and can effectively suppress water ripples without adding complex detection and filtering calculations, thereby solving the problem of large amount of calculation and consumption of a large amount of computing resources in the process of removing water ripples in the existing image processing methods.

[0046] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A flowchart of an image processing method provided by an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a correction reference area corresponding to a pixel point A to be corrected provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of an image processing method provided in a preferred embodiment of the present application;

[0051] Figure 4 A structural block diagram of an image processing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] In this embodiment, an image processing method is provided. Figure 1 is a flow chart of the image processing method of this embodiment, such as Figure 1 As shown, the process includes the following steps:

[0055] Step S110, obtaining each initial image captured by each lens of the multi-eye device.

[0056] The multi-eye device may be a device with two or more cameras, each camera is equipped with a lens. The initial images may be images processed by an ISP (Image Signal Processor), the format may be a YUV image, and the data bit width may be one of 8bit, 10bit or 12bit. The initial images captured by the lenses of the multi-eye device may be obtained by initializing the camera and setting parameters, for example, setting parameters such as synchronous photography, image format (such as YUV), bit width, etc.; processing the RAW image through the ISP, for example, the RAW image may be processed by the ISP for denoising, white balance, color correction, etc.; and outputting the processed image data in the YUV format. The RAW image may be an unprocessed data format directly obtained from an image sensor (such as a CMOS (Complementary Metal-Oxide-Semiconductor) or CCD (Charge-Coupled Device) sensor of a digital camera).

[0057] Step S120 , based on the position of each pixel point to be corrected in each initial image, determine each pixel point in the correction reference area corresponding to each pixel point to be corrected in each initial image.

[0058] Wherein, the above-mentioned determination of each pixel in the correction reference area corresponding to each pixel to be corrected of each initial image based on the position of each pixel to be corrected of each initial image may be based on the position of each pixel to be corrected of each initial image and the size of the preset correction reference area, to determine the correction reference area corresponding to each pixel to be corrected of each initial image, and then, based on the correction reference area corresponding to each pixel to be corrected of each initial image, to determine each pixel in the correction reference area corresponding to each pixel to be corrected of each initial image. Wherein, the correction reference area corresponding to the above-mentioned pixel to be corrected may be a range area of ​​a preset size centered on the pixel to be corrected. The above-mentioned determination of each pixel in the correction reference area corresponding to each pixel to be corrected of each initial image may be to determine the position and pixel value of each pixel in the correction reference area corresponding to each pixel to be corrected of each initial image.

[0059] Step S130, based on the pixels in the correction reference area corresponding to the pixels to be corrected and the positions of the pixels to be corrected, determine the interpolation coefficients of the pixels in the correction reference area corresponding to the pixels to be corrected; the interpolation coefficients are coefficients related to the direction between the pixels in the correction reference area and the pixels to be corrected.

[0060] The interpolation coefficient of the above-mentioned pixel point may be the product of the basis function coefficient of the pixel point and the gradient cosine coefficient of the pixel point. Therefore, the above-mentioned determination of the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected may be based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determining the basis function coefficient and gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and then determining the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on the basis function coefficient and gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected. The above-mentioned determination of the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected may be based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determining the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, and then determining the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area. It should be noted that the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area may include the horizontal distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, and the vertical distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area. The horizontal distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area may be the distance from the pixel point to be corrected corresponding to the correction reference area to the straight line determined by each pixel point in the column where each pixel point in the correction reference area is located. The vertical distance between each pixel point in the above-mentioned correction reference area and the pixel point to be corrected corresponding to the correction reference area can be the distance from the pixel point to be corrected corresponding to the correction reference area to the straight line determined by each pixel point in the row where each pixel point in the correction reference area is located.

[0061] In addition, the above-mentioned determination of the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected may be based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, to determine the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and specifically may be based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, to determine the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and then, based on the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected, determine the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected. The above-mentioned direction information of the pixel point may include one or more of horizontal direction information, vertical direction information, main diagonal direction information and anti-diagonal direction information. It should be noted that, because the position of each pixel in the correction reference area is fixed relative to the position of the pixel to be corrected, the direction information of each pixel in the correction reference area is one of horizontal direction information, vertical direction information, main diagonal direction information and anti-diagonal direction information. The above-mentioned main diagonal direction can be a direction from the upper left to the lower right. The above-mentioned anti-diagonal direction can be a direction from the upper right to the lower left. Based on this, the above-mentioned horizontal direction information is that the position of the pixel in the correction reference area is the horizontal direction relative to the position of the pixel to be corrected. The above-mentioned vertical direction information is that the position of the pixel in the correction reference area is the vertical direction relative to the position of the pixel to be corrected. The above-mentioned main diagonal direction information is that the position of the pixel in the correction reference area is the upper left to lower right direction or the lower right to upper left direction relative to the position of the pixel to be corrected. The above-mentioned anti-diagonal direction information is that the position of the pixel in the correction reference area is the upper right to lower left direction or the lower left to upper right direction relative to the position of the pixel to be corrected. The above-mentioned gradient cosine coefficient is a coefficient related to the direction between the pixel points in the correction reference area and the pixel points to be corrected. Therefore, the interpolation coefficient related to the gradient cosine coefficient is also a coefficient related to the direction between the pixel points in the correction reference area and the pixel points to be corrected.

[0062] Step S140, based on each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected, interpolation calculation is performed on each pixel point in the correction reference area to obtain the interpolated pixel value of each pixel point to be corrected.

[0063] The above-mentioned interpolation calculation is performed on each pixel in the correction reference area based on each pixel point to be corrected and the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected to obtain the interpolated pixel value of each pixel point to be corrected. It can be based on the pixel value of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the horizontal interpolation calculation is performed on each row in the correction reference area corresponding to each pixel point to be corrected to obtain the interpolated pixel value of each pixel point to be corrected. The horizontal interpolation result of each row of the correction reference area corresponding to each pixel to be corrected; the horizontal interpolation result includes the pixel value of each pixel obtained by horizontal interpolation and the position of each pixel obtained by horizontal interpolation; based on the horizontal interpolation result of each row of the correction reference area and the position of each pixel to be corrected, the interpolation coefficient of each pixel obtained by horizontal interpolation in each row is determined; based on the pixel value of each pixel obtained by horizontal interpolation in each row of the correction reference area corresponding to each pixel to be corrected and the interpolation coefficient of each pixel obtained by horizontal interpolation in each row, each pixel obtained by horizontal interpolation is vertically interpolated to obtain the interpolated pixel value of each pixel to be corrected. In addition, it should be noted that vertical interpolation calculation can also be performed on each column of the correction reference area corresponding to each pixel to be corrected first, to obtain the vertical interpolation result of each column of the correction reference area corresponding to each pixel to be corrected, and then, according to the vertical interpolation result of each column, each pixel obtained by vertical interpolation is horizontally interpolated to obtain the interpolated pixel value of each pixel to be corrected.

[0064] The above steps S110 to S140 determine the pixel points in the correction reference area corresponding to the pixel points to be corrected, and then determine the interpolation coefficients of the pixel points in the correction reference area corresponding to the pixel points to be corrected by using the pixel points in the correction reference area corresponding to the pixel points to be corrected and the positions of the pixel points to be corrected. The interpolation calculation is performed on the pixel points in the correction reference area according to the information related to the direction and the pixel values ​​of the pixel points in the correction reference area corresponding to the pixel points to be corrected, so as to obtain the interpolation pixel values ​​of the pixel points to be corrected. The interpolation calculation introduces the information related to the direction into the calculation process of the interpolation pixel values ​​of the pixel points to be corrected, so as to effectively suppress water ripples without adding complex detection and filtering calculations, thereby solving the problem of large amount of calculation and large amount of computing resources consumed in the process of removing water ripples in the existing image processing methods.

[0065] Specifically, in one embodiment, the above step S130, based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determines the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected, including:

[0066] Step S132, based on the pixels in the correction reference area corresponding to the pixels to be corrected and the positions of the pixels to be corrected, determine the basis function coefficients of the pixels in the correction reference area corresponding to the pixels to be corrected and the gradient cosine coefficients of the pixels in the correction reference area corresponding to the pixels to be corrected; the gradient cosine coefficient is a coefficient related to the direction between the pixels in the correction reference area and the pixels to be corrected.

[0067] Step S134, based on the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, determine the interpolation coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected.

[0068] In this step, the product of the basis function coefficient corresponding to each pixel point and the gradient cosine coefficient can be used as the interpolation coefficient corresponding to the current pixel point. The specific calculation process is as follows:

[0069]

[0070] Among them, f(r) is the interpolation coefficient of the pixel point in the correction reference area with a distance r between the pixel point to be corrected corresponding to the correction reference area, h(r) is the basis function coefficient of the pixel point in the correction reference area with a distance r between the pixel point to be corrected corresponding to the correction reference area, and g(r) is the gradient cosine coefficient of the pixel point in the correction reference area with a distance r between the pixel point to be corrected corresponding to the correction reference area.

[0071] In some cases, when the distance between the pixel points in the correction reference area corresponding to each pixel point to be corrected and the pixel point to be corrected is small (for example, less than a preset distance threshold), the influence of the gradient cosine coefficient on the interpolation calculation is small. In order to simplify the calculation process and reduce the amount of calculation, the gradient cosine coefficient can be set to 1.

[0072] In the above steps S132 to S134, the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected are determined, and the interpolation coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected are determined according to the basis function coefficients and the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected. The calculation of the interpolation coefficient facilitates the subsequent interpolation calculation of each pixel point in the correction reference area according to the interpolation coefficient of the pixel, so as to obtain the interpolated pixel value of each pixel point to be corrected.

[0073] In one embodiment, before step S120, the following steps are included:

[0074] Step S118, based on the position of each pixel point of each initial image and the stereo correction table of the multi-eye device, the position of each pixel point to be corrected in each initial image is obtained; the stereo correction table of the multi-eye device is a correction table generated based on the internal parameters and external parameters of the multi-eye device, and is used to perform stereo correction on the position of each pixel point of the initial image.

[0075] The stereo correction table of the multi-eye device may be a reverse mapping coordinate table obtained through a series of stereo correction calculations. The above-mentioned method for obtaining the stereo correction table of the multi-eye device is as follows: first, the camera of the multi-eye device is calibrated to obtain the internal and external parameters of the camera after calibration, and then, based on the internal and external parameters after the camera calibration, the stereo correction table of the multi-eye device is obtained. Specifically, the stereo correction table of the multi-eye device is obtained based on the internal and external parameters after the camera calibration, which may be to calculate the mapping function using the correction transformation matrix, and generate the stereo correction table according to the mapping function. The above-mentioned internal parameters may include the focal length, optical center, pixel size distortion coefficient, etc. of the camera. The above-mentioned external parameters may include the rotation matrix and the translation vector, etc. The above-mentioned calibration of the camera of the multi-eye device may be to calibrate the camera of the multi-eye device using the checkerboard method or other calibration board methods. The function of the stereo correction table of the multi-eye device is mainly to indicate how to map the pixels in the original image to the corrected image, that is, to indicate the position of each pixel in the original image to the corresponding position in the corrected image, and to perform stereo correction on the position of each pixel point of the initial image. Among them, the above-mentioned position of each pixel point to be corrected in each initial image is obtained based on the position of each pixel point of each initial image and the stereo correction table of the multi-eye device. It can be that according to the position of each pixel point of each initial image, the position corresponding to the position of each pixel point of the initial image in the stereo correction table of the multi-eye device is searched, and the position corresponding to the position of each pixel point of the initial image in the stereo correction table of the multi-eye device is used as the position of each pixel point to be corrected in each initial image.

[0076] Further, in one embodiment, step S118, based on the position of each pixel point of each initial image and the stereo correction table of the multi-eye device, obtains the position of each pixel point to be corrected in each initial image, including the following steps:

[0077] Step S1182, determining the position corresponding to each pixel point of each initial image in the horizontal coordinate table in the stereo correction table of the multi-eye device as the horizontal coordinate of each pixel point to be corrected in each initial image.

[0078] In order to obtain the position of each pixel to be corrected in each initial image by using the stereo correction table of the multi-eye device, the stereo correction table of the multi-eye device can be decomposed into a horizontal coordinate table and a vertical coordinate table.

[0079] Step S1184, determining the corresponding position of each pixel point of each initial image in the longitudinal coordinate table in the stereo correction table of the multi-eye device as the longitudinal coordinate of each pixel point to be corrected in each initial image.

[0080] Step S1186, determining the position of each pixel point to be corrected in each initial image based on the horizontal coordinate and the vertical coordinate of each pixel point to be corrected in each initial image.

[0081] In the above steps S1182 to S1186, the positions of the pixels to be corrected of each initial image are determined according to the corresponding positions of the pixels of each initial image in the horizontal coordinate table and the vertical coordinate table in the stereo correction table of the multi-eye device by disassembling the stereo correction table of the multi-eye device into the horizontal coordinate table and the vertical coordinate table. The positions of the pixels to be corrected of each initial image are determined to facilitate the subsequent determination of the pixels in the correction reference area corresponding to the pixels to be corrected of each initial image based on the positions of the pixels to be corrected of each initial image.

[0082] Specifically, in one embodiment, step S120, based on the position of each pixel point to be corrected in each initial image, determines each pixel point in the correction reference area corresponding to each pixel point to be corrected in each initial image, including the following steps:

[0083] Step S122: determining a correction reference area corresponding to each pixel to be corrected based on the position of each pixel to be corrected in each initial image.

[0084] Among them, the correction reference area corresponding to each pixel to be corrected is determined based on the position of each pixel to be corrected in each initial image. It can be that the pixel to be corrected is taken as the center, and an area of ​​N×N size around the pixel to be corrected is taken as the correction reference area corresponding to the pixel to be corrected. The above N can be a value such as 4, 6, 8, etc. N represents the number of pixels on each side of the square correction reference area. It should be noted that, in general, the size of N can be specifically set according to specific needs. As N increases, the correction reference area corresponding to the pixel to be corrected will also increase. The interpolation pixel value of the pixel to be corrected calculated based on this correction reference area is more accurate. However, the larger N is, the greater the amount of calculation of the interpolation pixel value of the pixel to be corrected. Therefore, based on the amount of calculation of the interpolation pixel value of the pixel to be corrected and the accuracy of the interpolation pixel value of the pixel to be corrected, a suitable N can be set after comprehensive consideration. In order to simply explain the process of calculating the interpolation pixel value of the pixel to be corrected, N is set to 4 in this embodiment.

[0085] Step S124, based on the correction reference area corresponding to each pixel point to be corrected, determining each pixel point in the correction reference area corresponding to each pixel point to be corrected.

[0086] In the above steps S122 to S124, the correction reference area corresponding to each pixel to be corrected is determined based on the position of each pixel to be corrected in each initial image, and each pixel in the correction reference area is obtained according to the correction reference area corresponding to each pixel to be corrected. By obtaining each pixel in each correction reference area, it is convenient to determine the basis function coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected and the gradient cosine coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected according to each pixel in each correction reference area.

[0087] Further, in one embodiment, based on step S132, based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determining the basis function coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected includes the following steps:

[0088] Step S1322, based on the pixels in the correction reference area corresponding to the pixels to be corrected and the positions of the pixels to be corrected, determine the distance between the pixels in the correction reference area and the pixels to be corrected corresponding to the correction reference area.

[0089] Because the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area includes the horizontal distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, as well as the vertical distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area. Therefore, the above-mentioned determination of the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected can be based on the position of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determining the first distance from each pixel point to be corrected to the straight line determined by each pixel point in the column where each pixel point in the correction reference area is located, determining the second distance from each pixel point to be corrected to the straight line determined by each pixel point in the row where each pixel point in the correction reference area is located, and using the first distance as the horizontal distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the over-corrected reference area, and using the second distance as the vertical distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the over-corrected reference area.

[0090] It should be noted that, because the distance from each pixel to be corrected to the straight line determined by each pixel in the column where each pixel in the correction reference area is located is the horizontal distance between each pixel in the correction reference area and the pixel to be corrected corresponding to the correction reference area, the horizontal distance between each column of pixels in the correction reference area and the pixel to be corrected is equal. Because the distance from each pixel to be corrected to the straight line determined by each pixel in the row where each pixel in the correction reference area is located is the vertical distance between each pixel in the correction reference area and the pixel to be corrected corresponding to the over-correction reference area, the vertical distance between each row of pixels in the correction reference area and the pixel to be corrected is equal.

[0091] Step S1324, determining basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area.

[0092] The above-mentioned method of determining the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area may be based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, and using a preset basis function determination method to determine the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected.

[0093] Furthermore, the above-mentioned preset basis function determination method may be to determine the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected by presetting a basis function coefficient correspondence table. Specifically, the above-mentioned determination of the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected by presetting a basis function coefficient correspondence table based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area may be to determine the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected by searching a pre-set basis function coefficient correspondence table based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area. The determination of the basis function coefficients by checking the edge can simplify the calculation process and reduce the amount of calculation.

[0094] The above-mentioned basis function coefficient correspondence table can be a correspondence table between the distance between the pixel points in different correction reference areas and the pixel points to be corrected corresponding to the correction reference area, and the basis function coefficients. For the convenience of expression, the distance between the pixel points in the correction reference area and the pixel points to be corrected corresponding to the correction reference area can be defined as parameter r. Then, r i Represents the distance between the i-th pixel in the correction reference area and the pixel to be corrected corresponding to the correction reference area. Specifically, the method of setting the basis function coefficient table can be to determine the interval range of the parameter r according to the size of the correction reference area corresponding to each pixel to be corrected, and then determine the number of basis function coefficients in the basis function coefficient corresponding table according to the precision of the parameter r taken, and the value of each parameter r, and calculate the basis function coefficient corresponding to each parameter r using the preset basis function coefficient calculation method to generate the basis function coefficient table. For example, if the size of the correction reference area corresponding to each pixel to be corrected is 4×4 (that is, N is equal to 4), then the range of parameter r is [-2, 2], the precision of parameter r is 1 decimal place, and the interval length between different parameters r is 0.1. Based on this, the basis function coefficients in the basis function coefficient corresponding table can be set to 41, which are the basis function coefficients obtained according to the parameter r, and the values ​​of the parameter r are: -2, -1.9, ..., -0.1, 0, 0.1, ...1.9, 2.

[0095] The above-mentioned method of calculating the basis function coefficients using the preset basis function coefficient calculation method to calculate the basis function coefficients corresponding to each parameter r may be to use a basis function model such as Bicubic or Lanczos to calculate the basis function coefficients corresponding to each parameter r.

[0096] Among them, the process of calculating the basis function coefficients using the Bicubic basis function model is as follows: The polynomial approximation formula of is:

[0097]

[0098] Where a is the width of the window in the basis function.

[0099] For example, when a=-1, the process of calculating the basis function coefficients using the Bicubic basis function model is:

[0100]

[0101] In addition, the process of calculating the basis function coefficients using the Lanczos basis function model is:

[0102]

[0103] Step S1326, based on the pixels in the correction reference area corresponding to the pixels to be corrected and the positions of the pixels to be corrected, determine the direction information of the pixels in the correction reference area corresponding to the pixels to be corrected.

[0104] The above-mentioned method of determining the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected may be based on the position of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of the pixel point to be corrected, to determine the directional relationship between each pixel point in the correction reference area and the pixel point to be corrected, and further, based on the directional relationship between each pixel point in the correction reference area and the pixel point to be corrected, to determine the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected.

[0105] Step S1328, based on the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, determine the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected.

[0106] Specifically, the direction information of the correction reference area related to the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected may be determined based on the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected; based on the direction information of the correction reference area related to the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected, the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area may be determined.

[0107] The above steps S1322 to S1328, by determining the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, facilitate the subsequent interpolation calculation of each pixel point in the correction reference area according to the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, so as to obtain the interpolated pixel value of each pixel point to be corrected.

[0108] In addition, in one embodiment, based on step S1326, based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determining the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected includes the following steps:

[0109] Step S13262, based on the pixels in the correction reference area corresponding to the pixels to be corrected, determine the positions of the pixels in the correction reference area corresponding to the pixels to be corrected.

[0110] Step S13264, based on the positions of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the positions of each pixel point to be corrected, determine the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected.

[0111] The specific process of determining the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected is described below by way of example. Figure 2 A schematic diagram of a correction reference area corresponding to a pixel point A to be corrected provided in an embodiment of the present application. Figure 2As shown, there are 16 pixels from s1 to s16 in the correction reference area of ​​the pixel point A to be corrected. Among them, the pixel point A to be corrected is located at the center of the correction reference area. If the direction information of the pixel point s1 in the correction reference area is to be determined, according to Figure 2 By combining the position of pixel point s1 in the correction reference area and the position of pixel point A to be corrected, it can be determined that the directional relationship between pixel point s1 in the correction reference area and pixel point A to be corrected is the upper left and lower right relationship, and further, the directional information of pixel point s1 in the correction reference area is determined to be the main diagonal directional information.

[0112] The above steps S13262 to S13264 determine the positions of each pixel in the correction reference area corresponding to each pixel to be corrected based on each pixel in the correction reference area corresponding to each pixel to be corrected, and then determine the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected based on the positions of each pixel in the correction reference area corresponding to each pixel to be corrected and the positions of each pixel to be corrected. By determining the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected, it is convenient to determine the gradient cosine coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected according to the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected.

[0113] In one embodiment, the above step S1328, based on the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected, and the distance between each pixel in the correction reference area and the pixel to be corrected corresponding to the correction reference area, determines the gradient cosine coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected, including:

[0114] Step S13282, based on the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected, determine the direction information of the correction reference area related to the gradient cosine coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected;

[0115] The above-mentioned direction information of the correction reference area corresponding to each pixel to be corrected is based on the direction information of the correction reference area corresponding to each pixel to be corrected, and the direction information of the correction reference area related to the gradient cosine coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected is determined. If the direction information of the pixel in the correction reference area corresponding to the pixel to be corrected is horizontal direction information / vertical direction information, then the direction information of the correction reference area related to the gradient cosine coefficient of the pixel in the correction reference area corresponding to the pixel to be corrected is the horizontal direction information of the correction reference area and the vertical direction information of the correction reference area; if the direction information of the pixel in the correction reference area corresponding to the pixel to be corrected is main diagonal direction information / anti-diagonal direction information, then the direction information of the correction reference area related to the gradient cosine coefficient of the pixel in the correction reference area corresponding to the pixel to be corrected is the main diagonal direction information of the correction reference area and the anti-diagonal direction information of the correction reference area.

[0116] Step S13284, determines the gradient cosine coefficient of each pixel point in the correction reference area based on the direction information of the correction reference area related to the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected, the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, and the size of the correction reference area corresponding to each pixel point to be corrected.

[0117] The calculation process of determining the gradient cosine coefficient of each pixel in the correction reference area based on the direction information of the correction reference area related to the gradient cosine coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected, the distance between each pixel in the correction reference area and the pixel to be corrected corresponding to the correction reference area, and the size of the correction reference area corresponding to each pixel to be corrected is as follows:

[0118] If the direction information of the pixel points in the correction reference area corresponding to the pixel points to be corrected is horizontal direction information, the calculation process of the gradient cosine coefficients of the pixel points in the correction reference area is as follows:

[0119]

[0120] Where gh(r) is the gradient cosine coefficient of the pixel point in the correction reference area whose distance from the pixel point to be corrected is r and whose direction information is horizontal direction information. c is a parameter that controls the gradient information. The larger c is, the greater the influence of the gradient information on the interpolation. The range of c is generally [0.5, 0.8], where the range of c is determined based on experience. For example, c can be set to 0.55. Where Gv is the vertical direction information of the correction reference area, and Gh is the horizontal direction information of the correction reference area.

[0121] Specifically, see Figure 2 , the calculation process of the vertical direction information Gv of the corrected reference area is as follows:

[0122] The calculation process of the vertical direction information Gv1 of the first column of pixels in the correction reference area is:

[0123]

[0124] Among them, s1 represents the position of pixel s1, s5 represents the position of pixel s5, s9 represents the position of pixel s9, and s13 represents the position of pixel s13.

[0125] The calculation process of the vertical direction information Gv2 of the second column of pixels in the correction reference area is:

[0126]

[0127] Among them, s2 represents the position of pixel s2, s6 represents the position of pixel s6, s10 represents the position of pixel s10, and s14 represents the position of pixel s14.

[0128] The calculation process of the vertical direction information Gv3 of the third column of pixels in the correction reference area is:

[0129]

[0130] Among them, s3 represents the position of pixel s3, s7 represents the position of pixel s7, s11 represents the position of pixel s11, and s15 represents the position of pixel s15.

[0131] The calculation process of the vertical direction information Gv4 of the fourth column of pixels in the correction reference area is:

[0132]

[0133] Among them, s4 represents the position of pixel s4, s8 represents the position of pixel s8, s12 represents the position of pixel s12, and s16 represents the position of pixel s16.

[0134] The calculation process of the vertical direction information Gv of the correction reference area is:

[0135]

[0136] The calculation process of the horizontal direction information Gh of the correction reference area is as follows:

[0137] The calculation process of the vertical direction information Gh1 of the first row of pixels in the correction reference area is:

[0138]

[0139] The calculation process of the vertical direction information Gh2 of the second row of pixels in the correction reference area is:

[0140]

[0141] The calculation process of the vertical direction information Gh3 of the third row of pixels in the correction reference area is:

[0142]

[0143] The calculation process of the vertical direction information Gh4 of the 4th row of pixels in the correction reference area is:

[0144]

[0145] The calculation process of correcting the vertical direction information Gh of the reference area is:

[0146]

[0147] If the direction information of the pixel points in the correction reference area corresponding to the pixel points to be corrected is vertical direction information, the calculation process of the gradient cosine coefficients of the pixel points in the correction reference area is as follows:

[0148]

[0149] Wherein, gv(r) is the gradient cosine coefficient of a pixel point in the correction reference area whose distance from the pixel point to be corrected is r and whose direction information is vertical direction information.

[0150] If the direction information of the pixel points in the correction reference area corresponding to the pixel points to be corrected is the main diagonal direction information, the calculation process of the gradient cosine coefficients of the pixel points in the correction reference area is as follows:

[0151]

[0152] Wherein, gdm(r) is the gradient cosine coefficient of the pixel point in the correction reference area whose distance from the pixel point to be corrected is r and whose direction information is the main diagonal direction information. Wherein, Gdn is the anti-diagonal direction information of the correction reference area, and Gdm is the main diagonal direction information of the correction reference area.

[0153] The calculation process of the anti-diagonal direction information Gdn of the corrected reference area is as follows:

[0154]

[0155]

[0156] The calculation process of the anti-diagonal direction information Gdm of the corrected reference area is as follows:

[0157]

[0158]

[0159] If the direction information of the pixel points in the correction reference area corresponding to the pixel points to be corrected is anti-diagonal direction information, the calculation process of the gradient cosine coefficients of the pixel points in the correction reference area is as follows:

[0160]

[0161] Wherein, gdn(r) is the gradient cosine coefficient of the pixel point whose distance from the pixel point to be corrected in the correction reference area is r and whose direction information is the anti-diagonal direction information. Figure 2 Taking a case where the horizontal direction information, vertical direction information, anti-diagonal direction information and main diagonal direction information of a correction reference area as an example, the specific calculation process can be adaptively adjusted according to the size of the correction reference area corresponding to the pixel point to be corrected.

[0162] The above steps S13282 to S13284 determine the gradient cosine coefficient of each pixel point in the correction reference area through the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected, the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, and the size of the correction reference area corresponding to each pixel point to be corrected. By determining the gradient cosine coefficient of each pixel point in the correction reference area, it is convenient to perform interpolation calculation on each pixel point in the correction reference area according to the gradient cosine coefficient of each pixel point in the correction reference area to obtain the interpolated pixel value of each pixel point to be corrected.

[0163] In addition, in one embodiment, the above step S140, based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected, performs interpolation calculation on each pixel point in the correction reference area to obtain the interpolated pixel value of each pixel point to be corrected, includes the following steps:

[0164] Step S142, based on the pixel values ​​of each pixel in the correction reference area corresponding to each pixel to be corrected, and the interpolation coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected, horizontal interpolation calculation is performed on each row of pixels in the correction reference area corresponding to each pixel to be corrected, to obtain the horizontal interpolation result of each row of the correction reference area corresponding to each pixel to be corrected; the horizontal interpolation result of each row includes the position and pixel value of the pixel obtained by horizontal interpolation of each row.

[0165] The above-mentioned horizontal interpolation calculation is performed on each row of pixel points in the correction reference area corresponding to each pixel point to be corrected to obtain the horizontal interpolation result of each row of the correction reference area corresponding to each pixel point to be corrected. It can be based on the interpolation coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the pixel values ​​of each row of pixel points in the correction reference area corresponding to each pixel point to be corrected are weighted summed to obtain the horizontal interpolation result of each row of the correction reference area.

[0166] Continue to see Figure 2 , if pixel s6, pixel s7, pixel s10 and pixel s11 are adjacent to pixel A to be corrected.

[0167] but, Figure 2 The calculation process of the horizontal interpolation result of each row in the correction reference area is as follows:

[0168]

[0169] Among them, v1 represents the pixel value of the horizontally interpolated pixel v1 in the first row of the correction reference area, rx1 is the horizontal distance between the pixel s1 and the pixel A to be corrected, and is also the horizontal distance between each pixel point in the column passing through s1 (such as s5, s9 and s13) and the pixel A to be corrected, rx2 is the horizontal distance between the pixel s2 and the pixel A to be corrected, and is also the horizontal distance between each pixel point in the column passing through s2 (such as s6, s10 and s14) and the pixel A to be corrected, rx3 is the horizontal distance between the pixel s3 and the pixel A to be corrected, and is also the horizontal distance between each pixel point in the column passing through s3 (such as s7, s11 and s15) and the pixel A to be corrected, rx4 is the horizontal distance between the pixel s4 and the pixel A to be corrected, and is also the horizontal distance between each pixel point in the column passing through s4 (such as s8, s12 and s16) and the pixel A to be corrected.

[0170]

[0171] Wherein, v2 represents the pixel value of the pixel point v2 after horizontal interpolation in the second row of the correction reference area.

[0172]

[0173] Wherein, v3 represents the pixel value of the horizontally interpolated pixel point v3 in the third row of the correction reference area.

[0174]

[0175] Wherein, v4 represents the pixel value of the horizontally interpolated pixel point v4 in the 4th row of the correction reference area.

[0176] It should be noted that, because the gradient cosine coefficients of the pixels in the correction reference area are related to the direction information of the pixels, the calculation process of the interpolation coefficients is different for different pixels.

[0177] For example, for pixel s1, the calculation process of its interpolation coefficient f(rx1) is as follows:

[0178]

[0179] For pixel s2, the calculation process of its interpolation coefficient f(rx2) is as follows:

[0180]

[0181] For pixel s3, the calculation process of its interpolation coefficient f(rx3) is as follows:

[0182]

[0183] For pixel s4, the calculation process of its interpolation coefficient f(rx4) is as follows:

[0184]

[0185] The interpolation coefficients of other pixel points are calculated in this way, which are not listed one by one in this embodiment.

[0186] Step S144, based on the horizontal interpolation result of each row of the correction reference area corresponding to each pixel to be corrected, determine the interpolation coefficient of the pixel point obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected.

[0187] The above-mentioned determination of the interpolation coefficients of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected based on the horizontal interpolation result of each row of the correction reference area corresponding to each pixel to be corrected may be based on the horizontal interpolation result of each row of the correction reference area corresponding to each pixel to be corrected, determining the basis function coefficients of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected, and the gradient cosine coefficients of the pixels obtained by horizontal interpolation of each row, and then, based on the basis function coefficients and gradient cosine coefficients of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected, determining the interpolation coefficients of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected. The calculation process of the interpolation coefficients in this embodiment is the same as step S134.

[0188] Specifically, it can be based on the horizontal interpolation result of each row of the correction reference area corresponding to each pixel to be corrected, to obtain the pixel value and position of the pixel point obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected; based on the position of the pixel point obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected and the position of each pixel to be corrected, determine the distance between the pixel point obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected and the pixel point to be corrected corresponding to the correction reference area; based on the distance between the pixel point obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected and the pixel point to be corrected corresponding to the correction reference area, determine the distance between the pixel point to be corrected and the pixel point to be corrected corresponding to the correction reference area using a preset basis function coefficient correspondence table. basis function coefficients of pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected; determining direction information of pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected based on the positions of pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected and the positions of each pixel to be corrected; determining gradient cosine coefficients of pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected based on the direction information of pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected and the distance between pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel to be corrected and the pixel to be corrected corresponding to the correction reference area.

[0189] Step S146, based on the pixel values ​​of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel point to be corrected, and the interpolation coefficients of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel point to be corrected, vertical interpolation calculation is performed on the pixels obtained by horizontal interpolation of each row to obtain the interpolated pixel values ​​of each pixel point to be corrected.

[0190] Continue to see Figure 2 , the process of performing vertical interpolation calculation on each row of horizontally interpolated pixels to obtain the interpolated pixel value d of the pixel point A to be corrected is as follows:

[0191]

[0192] Among them, ry1 is the vertical distance between the horizontally interpolated pixel v1 in the first row of the correction reference area and the pixel A to be corrected, and is also the vertical distance between each pixel point in the row passing through v1 (such as s1, s2 and s3, etc.) and the pixel A to be corrected. ry2 is the vertical distance between the horizontally interpolated pixel v2 in the second row of the correction reference area and the pixel A to be corrected. ry3 is the vertical distance between the horizontally interpolated pixel v3 in the third row of the correction reference area and the pixel A to be corrected. ry4 is the vertical distance between the horizontally interpolated pixel v4 in the fourth row of the correction reference area and the pixel A to be corrected.

[0193] In the above steps S142 to S146, the horizontal interpolation calculation is performed on the correction reference area corresponding to each pixel to be corrected, and the horizontal interpolation result of each row is obtained. Then, according to the horizontal interpolation result of each row, the pixel points obtained by the horizontal interpolation of each row are interpolated in the vertical direction to obtain the interpolated pixel value of each pixel to be corrected. In the process of interpolation calculation, the direction information is introduced into the bicubic interpolation process, and the intensity of the interpolation coefficient is automatically adjusted according to the gradient information of different directions of the image and the position relationship between pixels, so as to effectively suppress the water ripple problem and improve the clarity of the corrected image without increasing the complexity of detection and filtering calculation.

[0194] The present embodiment is described and illustrated below through preferred embodiments.

[0195] Figure 3 : is a flowchart of an image processing method provided by a preferred embodiment of the present application. Figure 3 As shown, the image processing method comprises the following steps:

[0196] Step S310, obtaining each initial image captured by each lens of the multi-eye device;

[0197] Step S320, based on the position of each pixel point of each initial image and the stereo correction table of the multi-eye device, obtain the position of each pixel point to be corrected in each initial image;

[0198] Step S330, based on the position of each pixel point to be corrected in each initial image, determining each pixel point in the correction reference area corresponding to each pixel point to be corrected in each initial image;

[0199] Step S340, based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, determining the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area;

[0200] Step S350, determining basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area;

[0201] Step S360, determining direction information of each pixel in the correction reference area corresponding to each pixel to be corrected based on each pixel in the correction reference area corresponding to each pixel to be corrected and the position of each pixel to be corrected;

[0202] Step S370, based on the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, determine the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected.

[0203] Step S380, interpolation calculation is performed on each pixel point in the correction reference area corresponding to each pixel point to be corrected, the position of each pixel point to be corrected, the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, to obtain the interpolated pixel value of each pixel point to be corrected.

[0204] The above steps S310 to S380 determine each pixel point in the correction reference area corresponding to each pixel point to be corrected, and then determine the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected by using each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected. It interpolates each pixel point in the correction reference area according to the gradient information related to the direction and the pixel value of each pixel point in the correction reference area corresponding to the pixel point to be corrected, so as to obtain the interpolated pixel value of each pixel point to be corrected. It introduces the gradient information related to the direction into the calculation process of the interpolated pixel value of each pixel point to be corrected, so as to effectively suppress water ripples without increasing complex detection and filtering calculations, thereby solving the problem of large amount of calculation and large consumption of computing resources in the process of removing water ripples in the existing image processing method.

[0205] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0206] Based on the same inventive concept, an image processing device is also provided in this embodiment, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0207] In one embodiment, Figure 4 is a structural block diagram of an image processing device provided by an embodiment of the present application, such as Figure 4 As shown, the image processing device comprises:

[0208] An image acquisition module 42 is used to acquire each initial image captured by each lens of the multi-eye device;

[0209] A pixel point determination module 44 is used to determine each pixel point in the correction reference area corresponding to each pixel point to be corrected in each initial image based on the position of each pixel point to be corrected in each initial image;

[0210] The coefficient determination module 46 is used to determine the interpolation coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected based on each pixel in the correction reference area corresponding to each pixel to be corrected and the position of each pixel to be corrected; the interpolation coefficient is a coefficient related to the direction between the pixel in the correction reference area and the pixel to be corrected;

[0211] And an interpolation calculation module 48 is used to perform interpolation calculation on each pixel in the correction reference area based on each pixel in the correction reference area corresponding to each pixel to be corrected, and the interpolation coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected, so as to obtain the interpolated pixel value of each pixel to be corrected.

[0212] The above-mentioned image processing device determines each pixel point in the correction reference area corresponding to each pixel point to be corrected, and then determines the interpolation coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected by using each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected. It performs interpolation calculation on each pixel point to be corrected according to information related to the direction and the pixel value of each pixel point in the correction reference area corresponding to the pixel point to be corrected to obtain the interpolation pixel value of each pixel point to be corrected. By introducing information related to the direction into the calculation process of the interpolation pixel value of each pixel point to be corrected, it can effectively suppress water ripples without adding complex detection and filtering calculations, thereby solving the problem of large amount of calculation and consumption of a large amount of computing resources in the process of removing water ripples in existing image processing methods.

[0213] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0214] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any one of the image processing methods in the above embodiments is implemented.

[0215] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0216] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0217] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire each initial image captured by each lens of the multi-eye device; Based on the position of each pixel point to be corrected in each of the initial images, determining each pixel point in the correction reference area corresponding to each pixel point to be corrected in each of the initial images; Based on the pixel points in the correction reference area corresponding to the pixel points to be corrected and the positions of the pixel points to be corrected, basis function coefficients of the pixel points in the correction reference area corresponding to the pixel points to be corrected and gradient cosine coefficients of the pixel points in the correction reference area corresponding to the pixel points to be corrected are determined; the gradient cosine coefficient is a coefficient related to the direction between the pixel points in the correction reference area and the pixel points to be corrected; based on the basis function coefficients of the pixel points in the correction reference area corresponding to the pixel points to be corrected and the gradient cosine coefficients of the pixel points in the correction reference area corresponding to the pixel points to be corrected, interpolation coefficients of the pixel points in the correction reference area corresponding to the pixel points to be corrected are determined; Based on the pixel points in the correction reference area corresponding to the pixel points to be corrected, and the interpolation coefficients of the pixel points in the correction reference area corresponding to the pixel points to be corrected, interpolation calculation is performed on the pixel points in the correction reference area to obtain interpolated pixel values ​​of the pixel points to be corrected.

2. The image processing method according to claim 1, characterized in that: Before determining each pixel point in a correction reference area corresponding to each pixel point to be corrected in each of the initial images based on the position of each pixel point to be corrected in each of the initial images, the method includes: Based on the position of each pixel point of each of the initial images and the stereo correction table of the multi-eye device, the position of each pixel point to be corrected in each of the initial images is obtained; the stereo correction table of the multi-eye device is generated based on the internal parameters and external parameters of the multi-eye device, and is a correction table for stereo correction of the position of each pixel point of the initial image.

3. The image processing method according to claim 2, characterized in that: The obtaining the position of each pixel point to be corrected of each initial image based on the position of each pixel point of each initial image and the stereo correction table of the multi-eye device comprises: Determine the position corresponding to each pixel point of each of the initial images in the horizontal coordinate table in the stereo correction table of the multi-eye device as the horizontal coordinate of each of the pixel points to be corrected in each of the initial images; Determine the corresponding position of each pixel point of each of the initial images in the longitudinal coordinate table in the stereo correction table of the multi-eye device as the longitudinal coordinate of each pixel point to be corrected in each of the initial images; Based on the abscissa and the ordinate of each of the pixels to be corrected in each of the initial images, the position of each of the pixels to be corrected in each of the initial images is determined.

4. The image processing method according to claim 1, characterized in that: The step of determining, based on the positions of the pixels to be corrected in the initial images, the pixels in the correction reference area corresponding to the pixels to be corrected in the initial images comprises: Determining the correction reference area corresponding to each of the pixel points to be corrected based on the position of each of the pixel points to be corrected in each of the initial images; Based on the correction reference area corresponding to each of the pixel points to be corrected, each of the pixel points within the correction reference area corresponding to each of the pixel points to be corrected is determined.

5. The image processing method according to claim 1, characterized in that: The determining, based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected, basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected and gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, comprises: Determining the distance between each pixel in the correction reference area and the pixel to be corrected corresponding to the correction reference area based on each pixel in the correction reference area corresponding to each pixel to be corrected and the position of each pixel to be corrected; Determining the basis function coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area; Determining direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected based on each pixel point in the correction reference area corresponding to each pixel point to be corrected and the position of each pixel point to be corrected; Based on the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the distance between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, the gradient cosine coefficient of each pixel point in the correction reference area corresponding to each pixel point to be corrected is determined.

6. The image processing method according to claim 5, characterized in that: The determining, based on the pixel points in the correction reference area corresponding to the pixel points to be corrected and the positions of the pixel points to be corrected, the direction information of the pixel points in the correction reference area corresponding to the pixel points to be corrected includes: Determining the positions of the pixels in the correction reference area corresponding to the pixels to be corrected based on the pixels in the correction reference area corresponding to the pixels to be corrected; Based on the positions of each pixel point in the correction reference area corresponding to each pixel point to be corrected and the positions of each pixel point to be corrected, the direction information of each pixel point in the correction reference area corresponding to each pixel point to be corrected is determined.

7. The image processing method according to claim 5, characterized in that: The step of determining the gradient cosine coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected based on the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected and the distance between each pixel in the correction reference area and the pixel to be corrected corresponding to the correction reference area comprises: Determine, based on the direction information of each pixel in the correction reference area corresponding to each pixel to be corrected, the direction information of the correction reference area related to the gradient cosine coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected; Based on the direction information of the correction reference area related to the gradient cosine coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, the distances between each pixel point in the correction reference area and the pixel point to be corrected corresponding to the correction reference area, and the size of the correction reference area corresponding to each pixel point to be corrected, the gradient cosine coefficients of each pixel point in the correction reference area are determined.

8. The image processing method according to claim 1, characterized in that: The interpolation calculation is performed on each pixel in the correction reference area based on each pixel in the correction reference area corresponding to each pixel to be corrected and the interpolation coefficient of each pixel in the correction reference area corresponding to each pixel to be corrected to obtain the interpolated pixel value of each pixel to be corrected, including: Based on the pixel values ​​of each pixel in the correction reference area corresponding to each pixel to be corrected and the interpolation coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected, horizontal interpolation calculation is performed on each row of pixel in the correction reference area corresponding to each pixel to be corrected to obtain a horizontal interpolation result of each row of the correction reference area corresponding to each pixel to be corrected; the horizontal interpolation result of each row includes the position and pixel value of each row obtained by horizontal interpolation; Determine, based on the horizontal interpolation result of each row of the correction reference area corresponding to each pixel point to be corrected, the interpolation coefficient of the pixel point obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel point to be corrected; Based on the pixel values ​​of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel point to be corrected, and the interpolation coefficients of the pixels obtained by horizontal interpolation of each row of the correction reference area corresponding to each pixel point to be corrected, vertical interpolation calculation is performed on the pixels obtained by horizontal interpolation of each row to obtain the interpolated pixel values ​​of each pixel point to be corrected.

9. An image processing device, characterized in that: The device comprises: An image acquisition module is used to acquire each initial image captured by each lens of the multi-eye device; A pixel point determination module, used to determine each pixel point in a correction reference area corresponding to each pixel point to be corrected in each of the initial images based on the position of each pixel point to be corrected in each of the initial images; A coefficient determination module, used to determine the basis function coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected and the gradient cosine coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected based on each pixel in the correction reference area corresponding to each pixel to be corrected and the positions of each pixel to be corrected; the gradient cosine coefficient is a coefficient related to the direction between the pixel in the correction reference area and the pixel to be corrected; based on the basis function coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected and the gradient cosine coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected, determine the interpolation coefficients of each pixel in the correction reference area corresponding to each pixel to be corrected; And an interpolation calculation module, which is used to perform interpolation calculation on each pixel point in the correction reference area based on each pixel point in the correction reference area corresponding to each pixel point to be corrected, and the interpolation coefficients of each pixel point in the correction reference area corresponding to each pixel point to be corrected, so as to obtain the interpolated pixel value of each pixel point to be corrected.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the image processing method according to any one of claims 1 to 8 are implemented.

Images