Image processing method, device and electronic equipment

Abstract

The present application provides an image processing method, device and equipment, the method comprising: for each camera, obtaining n frames of original images collected by the camera; wherein, when the coding pattern is projected n times to the object to be measured by the projection optical machine, the n frames of original images correspond to the n coding patterns one by one; generating a decoding image based on the n frames of original images, obtaining a binocular depth map based on the decoding image corresponding to the camera and the decoding image corresponding to another camera; obtaining a to-be-compensated image based on the decoding image, and obtaining a binocular compensation image based on the binocular depth map; determining fitting parameters based on the to-be-compensated image and the binocular compensation image; compensating the to-be-compensated image based on the fitting parameters to obtain a compensated image; obtaining the target depth map of the camera based on the compensated image. Through the scheme of the present application, the error is compensated in real time and dynamically, the measurement accuracy and environmental adaptability are improved, and the quality and accuracy of the target depth map are improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to an image processing method, device and electronic device. Background Art

[0002] The three-dimensional imaging device may include a laser and a camera (such as a video camera). The laser is used to project line structured light onto the surface of the object to be measured (i.e., the target to be measured), and the camera is used to shoot the object to be measured to obtain an image with line structured light, i.e., a line structured light image. After obtaining the line structured light image, the center line of the light strip of the line structured light image can be obtained, and the center line of the light strip can be converted according to pre-calibrated parameters to obtain the spatial coordinates of the object to be measured at the current position (i.e., three-dimensional coordinates). Based on the spatial coordinates of the object to be measured at the current position, three-dimensional reconstruction (i.e., three-dimensional reconstruction) of the object to be measured can be achieved.

[0003] In order to achieve 3D reconstruction of the object under test, it is necessary to collect line structured light images at different positions of the object under test, that is, the laser projects line structured light to different positions of the object under test, each position corresponds to a line structured light image, and the camera only collects the line structured light image corresponding to one position each time. When completing 3D reconstruction based on multiple line structured light images collected by the camera, there are problems such as poor 3D reconstruction effect. Summary of the invention

[0004] The present application provides an image processing method, which is applied to a three-dimensional imaging device, wherein the three-dimensional imaging device includes a projection optical machine and at least two cameras, and the method includes: for each camera, obtaining n frames of original images collected by the camera; wherein, when the coded pattern is projected n times onto the object to be measured by the projection optical machine, the n frames of original images correspond to the n coded patterns one by one; generating a decoding image based on the n frames of original images, and obtaining a binocular depth map based on the decoding image corresponding to the camera and the decoding image corresponding to another camera; obtaining a to-be-compensated image based on the decoding image, and obtaining a binocular compensation image based on the binocular depth map; determining fitting parameters based on the to-be-compensated image and the binocular compensation image; compensating the to-be-compensated image based on the fitting parameters to obtain a compensated image; and obtaining a target depth map of the camera based on the compensated image.

[0005] The present application provides an image processing device, which is applied to a three-dimensional imaging device, wherein the three-dimensional imaging device includes a projection optical machine and at least two cameras, and the device includes: an acquisition module, which is used to acquire n frames of original images collected by each camera; wherein, when the coded pattern is projected n times onto the object to be measured by the projection optical machine, the n frames of original images correspond to the n coded patterns one by one; a decoding image is generated based on the n frames of original images, and a binocular depth map is acquired based on the decoding image corresponding to the camera and the decoding image corresponding to another camera; a to-be-compensated image is acquired based on the decoding image, and a binocular compensation image is acquired based on the binocular depth map; a determination module, which is used to determine fitting parameters based on the to-be-compensated image and the binocular compensation image; the to-be-compensated image is compensated based on the fitting parameters to obtain a compensated image; and a processing module, which is used to acquire a target depth map of the camera based on the compensated image.

[0006] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method of the above example of the present application.

[0007] It can be seen from the above technical solutions that in the embodiment of the present application, the coding pattern is projected n times to the object to be measured by a projection optical machine, and the n frames of original images corresponding to the coding pattern are collected by a camera, and a decoding image is generated based on the n frames of original images. In this way, a binocular depth map can be obtained based on the decoding images corresponding to multiple cameras, and three-dimensional reconstruction can be achieved based on the binocular depth map. The three-dimensional reconstruction effect is very good, the reconstruction accuracy is higher, and the robustness is better. A binocular compensation map can be obtained based on the binocular depth map, and the compensation map can be compensated based on the binocular compensation map to obtain a compensated image, and a target depth map can be obtained based on the compensated image, so as to perform real-time dynamic compensation for errors, improve measurement accuracy and environmental adaptability, and improve the quality and accuracy of the target depth map. When three-dimensional reconstruction is achieved based on the target depth map, the influence of errors can be eliminated to obtain an accurate and reliable three-dimensional reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a flowchart of an image processing method in one embodiment of the present application;

[0009] Figure 2 is a schematic diagram of monocular system reconstruction and binocular system reconstruction in one embodiment of the present application;

[0010] Figure 3 is a flowchart of an image processing method in one embodiment of the present application;

[0011] Figure 4 is a flowchart of an image processing method in one embodiment of the present application;

[0012] Figure 5 is a schematic diagram of a sorting and matching method in one embodiment of the present application;

[0013] Figure 6 is a flowchart of an image processing method in one embodiment of the present application;

[0014] Figure 7 is a flowchart of an image processing method in one embodiment of the present application;

[0015] Figure 8 is a schematic structural diagram of an image processing device in one embodiment of the present application;

[0016] Fig. 9 It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0017] In an embodiment of the present application, an image processing method is proposed, which can be applied to a three-dimensional imaging device, and the three-dimensional imaging device can include a projection light machine and at least two cameras. Figure 1 FIG. 1 is a flow chart of the image processing method, which may include:

[0018] Step 101: for each camera, obtain n frames of original images captured by the camera; wherein, when the coding pattern is projected n times onto the object to be measured by the projection optical machine, the n frames of original images correspond to the n coding patterns one by one. The n frames of original images are all images of the object to be measured, and the original images include the coding patterns.

[0019] Step 102: Generate a decoded image based on n frames of original images, and obtain a binocular depth image based on the decoded image corresponding to the camera and the decoded image corresponding to another camera. The binocular depth image may include a distance between a position point in the physical space and the camera, and the position point is a position point in the overlapping area of ​​the field of view of the two cameras. For example, the binocular depth image may include multiple pixels, and the value of each pixel represents a depth value. For each depth value, the depth value represents the distance between the position point in the physical space and the camera.

[0020] Step 103: Obtain a to-be-compensated image based on the decoded image, and obtain a binocular compensation image based on the binocular depth image. For example, the to-be-compensated image may include a first pixel corresponding to a field of view overlap area and a second pixel corresponding to a non-field of view overlap area, and the binocular compensation image may include a first pixel corresponding to a field of view overlap area.

[0021] Step 104: determine fitting parameters (i.e., fitting parameters of the compensation function) based on the image to be compensated and the binocular compensation image; and compensate the image to be compensated based on the fitting parameters to obtain a compensated image.

[0022] Exemplarily, based on the image to be compensated and the binocular compensation image, determining the fitting parameters may include but is not limited to: determining the compensation difference of the first pixel based on the pixel value corresponding to the first pixel in the image to be compensated and the pixel value corresponding to the first pixel in the binocular compensation image; determining the fitting parameters of the compensation function based on the compensation differences of multiple first pixels. Compensating the image to be compensated based on the fitting parameters to obtain a compensated image may include but is not limited to: determining the compensation difference of the second pixel based on the compensation function. Using the compensation difference of the first pixel to compensate the corresponding pixel value in the image to be compensated, using the compensation difference of the second pixel to compensate the corresponding pixel value in the image to be compensated, to obtain a compensated image.

[0023] Step 105: Obtain a target depth map of the camera based on the compensated image, the target depth map including the distance between a position point in the physical space and the camera, the position point being a position point within the field of view of the camera. For example, the target depth map may include a plurality of pixels, each pixel value representing a depth value, and for each depth value, the depth value represents the distance between the position point in the physical space and the camera.

[0024] Exemplarily, the binocular compensation map may be a binocular depth map, the map to be compensated may be a monocular depth map corresponding to the decoded map, the compensated image may be a compensated monocular depth map, and the target depth map may be a compensated monocular depth map. Alternatively, the binocular compensation map may be a binocular disparity map corresponding to the binocular depth map, the map to be compensated may be a monocular disparity map corresponding to the decoded map, the compensated image may be a compensated monocular disparity map, and the target depth map may be a monocular depth map corresponding to the compensated monocular disparity map. Alternatively, the binocular compensation map may be a binocular decoded map corresponding to the binocular depth map, the map to be compensated may be a decoded map, the compensated image may be a compensated decoded map, and the target depth map may be a monocular depth map corresponding to the compensated decoded map.

[0025] Exemplarily, determining the fitting parameters of the compensation function based on the compensation differences of multiple first pixel points includes, but is not limited to: determining the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image and the compensation differences of the multiple first pixel points; or, determining the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image, the pixel values ​​corresponding to the multiple first pixel points in the monocular depth map corresponding to the decoded image, and the compensation differences of the multiple first pixel points; or, determining the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image, the pixel values ​​corresponding to the multiple first pixel points in the decoded image corresponding to another camera, and the compensation differences of the multiple first pixel points.

[0026] Exemplarily, the compensation function includes but is not limited to: A polynomial function of degree, is a positive integer greater than 1, The polynomial function is expressed as: ;in, , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, represents the compensation difference of the first pixel; or, for the pixel value in the decoded image The polynomial function and the pixel value for the monocular depth map A polynomial function of degree, and is a positive integer greater than 1, polynomial function and The polynomial function is expressed as: ;in, , , , , , , , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, Indicates the pixel value corresponding to the first pixel in the monocular depth map corresponding to the decoded image, represents the compensation difference of the first pixel; or, for the pixel value in the decoded image The polynomial function and the pixel value in the decoded image corresponding to another camera A polynomial function of degree, and is a positive integer greater than 1, polynomial function and The polynomial function is expressed as: ; , , , , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, Indicates the pixel value corresponding to the first pixel in the decoded image corresponding to another camera. Indicates the compensation difference of the first pixel.

[0027] Exemplarily, determining the compensation difference of the second pixel point based on the compensation function may include but is not limited to: inputting the pixel value corresponding to the second pixel point in the decoded image and the fitting parameters into the compensation function to obtain the compensation difference of the second pixel point; or, inputting the pixel value corresponding to the second pixel point in the decoded image, the pixel value corresponding to the second pixel point in the monocular depth map corresponding to the decoded image, and the fitting parameters into the compensation function to obtain the compensation difference of the second pixel point; or, inputting the pixel value corresponding to the second pixel point in the decoded image, the pixel value corresponding to the second pixel point in the decoded image corresponding to another camera, and the fitting parameters into the compensation function to obtain the compensation difference of the second pixel point.

[0028] Exemplarily, after acquiring the target depth map of the camera based on the compensated image, the first point cloud data of the position points in the overlapping field of view and the second point cloud data of the position points in the non-overlapping field of view of the camera can be determined based on the target depth map of the camera; the third point cloud data of the position points in the overlapping field of view and the fourth point cloud data of the position points in the non-overlapping field of view of the other camera can be determined based on the target depth map of the other camera. On this basis, the first point cloud data, the second point cloud data and the fourth point cloud data can be fused to obtain fused point cloud data, and a fused depth map can be generated based on the fused point cloud data; or, the first point cloud data and the third point cloud data can be weighted to obtain weighted point cloud data, and the weighted point cloud data, the second point cloud data and the fourth point cloud data can be fused to obtain fused point cloud data, and a fused depth map can be generated based on the fused point cloud data.

[0029] Exemplarily, obtaining a binocular depth map based on a decoded image corresponding to the camera and a decoded image corresponding to another camera may include, but is not limited to: obtaining a key point pair corresponding to a pixel point in a first decoded image, wherein the key point pair may include the pixel point and an associated pixel point corresponding to the pixel point in a second decoded image, the first decoded image being the decoded image corresponding to the camera, and the second decoded image being the decoded image corresponding to another camera; obtaining a binocular depth map based on multiple key point pairs. Among them, when obtaining the associated pixel point corresponding to the pixel point in the second decoding image, if the associated pixel point is an integer pixel point, the associated pixel point can be traversed from the second decoding image, and the pixel value of the pixel point is the same as the pixel value of the associated pixel point; or, if the associated pixel point is a sub-pixel point, the left adjacent pixel point and the right adjacent pixel point can be traversed from the second decoding image, the pixel value of the pixel point is greater than the pixel value of the left adjacent pixel point, and the pixel value of the pixel point is less than the pixel value of the right adjacent pixel point; based on the pixel value of the left adjacent pixel point, the coordinates of the left adjacent pixel point, the pixel value of the right adjacent pixel point, the coordinates of the right adjacent pixel point, and the pixel value of the pixel point, the coordinates of the associated pixel point are calculated.

[0030] It can be seen from the above technical solutions that in the embodiment of the present application, the coding pattern is projected n times to the object to be measured by a projection optical machine, and the n frames of original images corresponding to the coding pattern are collected by a camera, and a decoding image is generated based on the n frames of original images. In this way, a binocular depth map can be obtained based on the decoding images corresponding to multiple cameras, and three-dimensional reconstruction can be achieved based on the binocular depth map. The three-dimensional reconstruction effect is very good, the reconstruction accuracy is higher, and the robustness is better. A binocular compensation map can be obtained based on the binocular depth map, and the compensation map can be compensated based on the binocular compensation map to obtain a compensated image, and a target depth map can be obtained based on the compensated image, so as to perform real-time dynamic compensation for errors, improve measurement accuracy and environmental adaptability, and improve the quality and accuracy of the target depth map. When three-dimensional reconstruction is achieved based on the target depth map, the influence of errors can be eliminated to obtain an accurate and reliable three-dimensional reconstructed image.

[0031] The image processing method of the embodiment of the present application is described below in conjunction with specific application scenarios.

[0032] In order to achieve three-dimensional reconstruction of the object to be measured, a monocular depth camera can be used to achieve three-dimensional reconstruction of the object to be measured. For example, the three-dimensional imaging device may include a projection optical machine and a single camera. If the decoding result of a pixel on the camera is clear, the pixel coordinates corresponding to the pixel on the projection optical machine are known. Based on the pixel coordinates of the pixel on the camera image and the pixel coordinates corresponding to the pixel on the projection optical machine, the three-dimensional coordinates corresponding to the pixel can be calculated by triangulation to complete the three-dimensional reconstruction. In order to achieve three-dimensional reconstruction of the object to be measured, a dual monocular depth camera can be used to achieve three-dimensional reconstruction of the object to be measured. For example, the three-dimensional imaging device may include a projection optical machine and two cameras. Each camera combined with the projection optical machine can form a set of monocular depth cameras, and each camera can complete three-dimensional reconstruction alone. In addition, the two cameras can form a binocular system. The projection optical machine only provides pattern texture information. The binocular matching of the two cameras can also complete the reconstruction, but it can only reconstruct the area observed by the two cameras at the same time.

[0033] See also Figure 2 As shown in the figure, it is a schematic diagram of the reconstruction of two monocular systems and the reconstruction of a binocular system. After the reconstruction results of the two monocular systems are fused, they are richer and more complete than those of the binocular system. For example, CAM1 represents the left camera, CAM2 represents the right camera, and PROJ represents the projection optical machine. The reconstruction of the two monocular systems means that the data of CAM1 and PROJ complete the reconstruction of the left system, and the reconstruction results include the reconstruction results of areas such as a, b, c, and e. The data of CAM2 and PROJ complete the reconstruction of the right system, and the reconstruction results include the reconstruction results of areas such as a, b, c, and d. After the reconstruction results of the left system and the right system are fused, the fused reconstruction results include areas such as a, b, c, d, and e.

[0034] Binocular system reconstruction means that the data of CAM1 and CAM2 complete the reconstruction of the binocular system, and the reconstruction result includes the reconstruction result of the overlapping area, such as the reconstruction result includes the reconstruction results of areas a, b, c, etc.

[0035] In summary, dual monocular depth cameras can be used to achieve 3D reconstruction of the object being measured, that is, obtain the reconstruction results of two monocular systems, and then fuse the reconstruction results of the two monocular systems. For example, obtain the reconstruction results of the left eye system and the right eye system, and fuse the two reconstruction results.

[0036] When using dual monocular depth cameras to achieve 3D reconstruction of the object being measured, the data of CAM1 and PROJ complete the reconstruction of the left eye system, and the data of CAM2 and PROJ complete the reconstruction of the right eye system. Obviously, the 3D reconstruction process needs to rely on the stability of the projection optical machine (PROJ), that is, the data accuracy of the projection optical machine needs to be guaranteed. However, the projection optical machine itself is a power device, and the heat generation of the projection optical machine is serious, which can easily cause the temperature drift of the lens structure parameters of the projection optical machine, and the temperature drift change will cause the data of the projection optical machine to be wrong. When the data of the projection optical machine is wrong, the reconstruction result of the left eye system is wrong, and the reconstruction result of the right eye system is wrong, which leads to errors in the reconstruction result after fusion, and the 3D reconstruction effect is poor.

[0037] In view of the above findings, in the embodiments of the present application, it is possible to perform real-time dynamic compensation for the errors caused by the state changes of the projection optical machine, improve the measurement accuracy and environmental adaptability, make the reconstruction results of the left-eye system correct, make the reconstruction results of the right-eye system correct, make the reconstruction results after fusion correct, and achieve better three-dimensional reconstruction effects. The error compensation in this embodiment refers to: due to environmental factors (such as ambient temperature, structural changes caused by transportation vibration, the angle with the direction of gravity during installation, the stress exerted by fasteners on the camera, etc.), the state of the projection optical machine is inconsistent with the factory calibrated state, resulting in a difference between the measurement results of the projection optical machine and the true value. In order to improve the adaptability and stability of three-dimensional imaging equipment in different environments, the errors caused by the state changes of the projection optical machine are compensated for in real time and dynamically to correct the measurement errors caused by the state changes.

[0038] In an embodiment of the present application, an image processing method is proposed, which can be applied to a three-dimensional imaging device, and the three-dimensional imaging device may include a projection light machine (such as at least one projection light machine) and at least two cameras (such as a video camera). The at least two cameras may be two cameras, or three cameras or more cameras, and there is no limitation to this. In the following, two cameras are taken as an example, and the two cameras are called a left-eye camera and a right-eye camera.

[0039] The projection optical machine is used to project a specific pattern (recorded as a coded pattern) onto the object to be measured. The coded image is projected onto the object to be measured, and is affected (modulated) by the height of the object to be measured and changes. The camera records the changed image and processes it. For example, the projection optical machine may include but is not limited to speckle DOE (Diffractive Optical Elements), DLP (Digital Light Processing) projector, LCD (Liquid Crystal Display) projector, LCOS (Liquid Crystal On Silicon) projector, and laser galvanometer optical machine. Of course, the above is only an example of a projection optical machine, and there is no limitation to this. The laser galvanometer optical machine will be used as an example for explanation later.

[0040] For example, a laser galvanometer optical machine is composed of a galvanometer and a laser. The optical machine composed of a galvanometer and a laser is used to project a specific coded pattern to provide image features. Regarding the working process of the laser galvanometer optical machine, the galvanometer is a small mirror that can be electrically controlled to rotate. A single line laser (the line laser emitted by the laser) is incident on the small mirror and reflected onto the object to be measured. While the galvanometer rotates, the switch / energy intensity of the laser is synchronously controlled. During the entire process of the galvanometer rotating from the starting point to the end point, the camera is continuously exposed, and the camera obtains a scan pattern.

[0041] See also Figure 3 FIG. 1 is a flow chart of an image processing method, which may include:

[0042] Step 301: for each camera, obtain n frames of original images captured by the camera, and generate a decoding image based on the n frames of original images. For example, when the coding pattern is projected n times onto the object to be measured by a projection optical machine, n frames of original images are collected, and the n frames of original images correspond to the n coding patterns one by one. The n frames of original images are all images of the object to be measured, and for each frame of original images, the original image includes a coding pattern.

[0043] For example, when the coding pattern a1 is projected onto the object to be measured by the projection optical machine, the camera synchronously collects the original image b1 of the object to be measured, and the original image b1 may include the coding pattern a1. When the coding pattern a2 is projected onto the object to be measured by the projection optical machine, the camera synchronously collects the original image b2 of the object to be measured, and the original image b2 may include the coding pattern a2. By analogy, n frames of original images may be obtained. After obtaining n frames of original images, a decoding image may be generated based on the n frames of original images.

[0044] Exemplarily, the projection light machine and the camera need to be processed synchronously. For example, the moment when the projection light machine starts projecting is the same as the moment when the camera starts exposing, and the moment when the projection light machine ends projecting is the same as the moment when the camera ends exposing. In this way, n frames of original images correspond one-to-one to n coded patterns.

[0045] Exemplarily, when the projection light machine projects the coding pattern onto the object to be measured, the coding pattern may be a Gray code, a phase shift code, other stripe patterns or a combination of various patterns, and there is no limitation on this coding pattern.

[0046] Exemplarily, after obtaining n frames of original images, a decoding image is generated based on the n frames of original images. For example, the three-dimensional imaging device includes a processor (such as an embedded processor), and the embedded processor generates a decoding image based on the n frames of original images. Alternatively, the three-dimensional imaging device can also send the n frames of original images to a host industrial computer, and the host industrial computer generates a decoding image based on the n frames of original images, and sends the decoding image to the three-dimensional imaging device. Of course, the above is only an example, and there is no limitation on the execution subject of the generation of the decoding image.

[0047] Regarding the coding pattern and the decoding map, the coding pattern can be used for projection, such as Gray code pattern, phase shift pattern, speckle pattern, etc. Various schemes of these pattern combinations can be projected in sequence. For the pixels on a single camera, the brightness corresponding to each frame of the original image is different, so the gray value of the pixel changes over time, forming a set of codes. The code can use a specific algorithm to calculate the decoding result of each pixel through the pattern design rules of various schemes, which is the decoding map.

[0048] Exemplarily, a process of generating a decoding graph based on n frames of original images may include:

[0049] For each pixel in the decoded image, the decoded value corresponding to the pixel in the decoded image is determined based on the pixel value corresponding to the pixel in each frame of the original image. For example, for the pixel (x, y) in the decoded image, the decoded value corresponding to the pixel (x, y) in the decoded image can be determined based on the pixel value corresponding to the pixel (x, y) in the original image b1, the pixel value corresponding to the pixel (x, y) in the original image b2, ..., the pixel value corresponding to the pixel (x, y) in the original image bn. After obtaining the decoded value corresponding to each pixel in the decoded image, the decoded values ​​corresponding to all the pixels can be combined to obtain a decoded image, that is, the decoded image can include the decoded value corresponding to each pixel.

[0050] Taking the coding pattern as an 8-order Gray code as an example, the projection optical machine can project the coding pattern (8-order Gray code) 8 times onto the object to be measured, and collect 8 frames of original images through the camera. For each original image, the original image is binarized, that is, the pixel value of each pixel of the original image is 0 or 1. On this basis, for the pixel point (x, y) in the decoding image, assuming that the pixel values ​​corresponding to the pixel point (x, y) in the 8 frames of the original image (that is, 8 independent binary values) are: 0, 1, 0, 1, 1, 0, 1, 1, then the corresponding decoding value of the pixel point (x, y) in the decoding image can be determined based on these 8 binary values. For example, these 8 binary values ​​are arranged in order and spliced ​​into a binary value of 01011011, and then this binary value 01011011 is converted into a Gray code, and this Gray code is used as the corresponding decoding value of the pixel point (x, y) in the decoding image. Of course, the above is just an example of determining the decoding value, and there is no limitation to this.

[0051] In order to convert the binary value 01011011 into Gray code, the mapping relationship between Gray code and binary code can be pre-configured, and there is no restriction on this mapping relationship. On this basis, by querying the mapping relationship through the binary value 01011011, the Gray code corresponding to the binary value 01011011 can be obtained. Regarding how to generate a decoding image based on n frames of original images, the decoding image generation method is determined according to the coding pattern, and there is no restriction in this embodiment. For example, when the coding pattern is an 8th-order Gray code, the above embodiment is adopted. When the coding pattern is a phase shift code, the least squares phase shift method can be used to generate a decoding image. When the coding pattern is other stripe patterns, there is also a corresponding decoding image generation method.

[0052] In a possible implementation, taking the left camera and the right camera as an example, see Figure 4 FIG. 3 is a flow chart of an image processing method. In step 301, a left-eye acquisition image (i.e., n frames of original images) acquired by a left-eye camera is obtained, and a left-eye decoded image is obtained based on the left-eye acquisition image. A right-eye acquisition image acquired by a right-eye camera is obtained, and a right-eye decoded image is obtained based on the right-eye acquisition image.

[0053] Step 302: For each camera, a binocular depth map is obtained based on a decoded map corresponding to the camera and a decoded map corresponding to another camera. For example, binocular matching can be performed based on the decoded maps corresponding to the two cameras to obtain a binocular depth map. Exemplarily, the binocular depth map may include a distance between a position point in a physical space and the camera, and the position point is a position point in an overlapping area of ​​the field of view of the two cameras.

[0054] Exemplarily, the decoded image corresponding to the camera can be called the first decoded image, and the decoded image corresponding to the other camera can be called the second decoded image. Based on this, when binocular matching is performed based on the first decoded image and the second decoded image to obtain a binocular depth map, the key point pairs corresponding to the pixel points (such as multiple pixel points) in the first decoded image can be obtained, and the key point pairs can include the pixel points and the associated pixel points corresponding to the pixel points in the second decoded image. After obtaining the key point pairs corresponding to the multiple pixel points, the binocular depth map can be obtained based on the multiple key point pairs. For example, for the binocular matching process, the main goal is to determine the correspondence between the pixel points of the first decoded image and the pixel points of the second decoded image, so as to calculate the binocular depth map.

[0055] Exemplarily, in order to achieve binocular matching, as long as the key point pair can be found, the depth value can be calculated by triangulation, and the depth values ​​corresponding to all pixels in the first decoded image constitute the binocular depth map. There is no restriction on this process. The process of obtaining the key point pair is explained below.

[0056] Exemplarily, after obtaining the first decoding image and the second decoding image, the first decoding image and the second decoding image can be subjected to epipolar line correction, and no restriction is imposed on the epipolar line correction process, so that the first decoding image and the second decoding image satisfy the epipolar line correction constraint, that is, a row of pixels of the first decoding image corresponds to a row of pixels of the second decoding image, such as the 27th row of pixels of the first decoding image corresponds to the 27th row of pixels of the second decoding image.

[0057] For example, for a certain position point in the physical space, assuming that the position point corresponds to pixel point 1 in the first decoding image, and the position point corresponds to pixel point 2 in the second decoding image, then pixel point 1 and pixel point 2 correspond to the same row, and the pixel value of pixel point 1 is the same as the pixel value of pixel point 2, such as the pixel value of pixel point 1 and the pixel value of pixel point 2 are both Gray codes corresponding to the binary value 01011011.

[0058] On this basis, for each pixel point in the first decoding image, the associated pixel point corresponding to the pixel point can be traversed from the second decoding image, and the pixel value of the pixel point is the same as the pixel value of the associated pixel point. For example, assuming that pixel point 1 is located in the 27th row of pixels in the first decoding image, each pixel point in the 27th row of the second decoding image is traversed in turn. If the pixel value of the traversed pixel point is the same as the pixel value of pixel point 1, the pixel point is used as the associated pixel point of pixel point 1. If the pixel value of the traversed pixel point is different from the pixel value of pixel point 1, continue to traverse the next pixel point. Obviously, after traversing the second decoding image, the associated pixel point of each pixel point in the first decoding image can be obtained, and the pixel point in the first decoding image and the associated pixel point of the pixel point in the second decoding image constitute a key point pair.

[0059] Exemplarily, for each pixel point in the first decoding diagram, if the associated pixel point corresponding to the pixel point in the second decoding diagram is an integer pixel point, the associated pixel point can be traversed from the second decoding diagram, and the pixel value of the pixel point is the same as the pixel value of the associated pixel point, and this process will not be described in detail. If the associated pixel point corresponding to the pixel point in the second decoding diagram is a sub-pixel point, a sorting matching algorithm can also be used to sort the pixels in the second decoding diagram, and the associated pixel point corresponding to the pixel point in the second decoding diagram is found based on the sorting result. For example, the sorting matching algorithm may include but is not limited to a merge sort matching algorithm, a bucket sort matching algorithm, a block sort matching algorithm, etc., and there is no restriction on this sorting algorithm. Taking the coding pattern as Gray code as an example, the sorting matching process of the pixels in the second decoding diagram is introduced.

[0060] See also Figure 5 As shown, it is a schematic diagram of the sorting matching method. Gray code (i.e., the pixel value of the pixel point in the decoding image) is unique and monotonically increasing on a single row of pixel points. When determining the mapping relationship, the pixel points of the corresponding rows of the first decoding image and the second decoding image are sorted together, and the decoding image to which each pixel point belongs is recorded while sorting. After the sorting is completed, for the pixel point P (belonging to the first decoding image) in the sorting result, the pixel point M and the pixel point N closest to the pixel point P are searched forward or backward. The pixel point M belongs to the second decoding image and is located in front of the pixel point P, and the pixel point N belongs to the second decoding image and is located behind the pixel point P. Obviously, the pixel coordinates of the pixel point P are between the pixel coordinates of the pixel point M and the pixel coordinates of the pixel point N. On this basis, linear interpolation is performed based on the pixel value of the pixel point M and the pixel value of the pixel point N, and the sub-pixel coordinates of the associated pixel point (sub-pixel point) corresponding to the pixel point P in the second decoding image can be obtained.

[0061] In summary, for the pixel point P in the first decoding image, the left adjacent pixel point (pixel point M) and the right adjacent pixel point (pixel point N) can be traversed from the second decoding image, and the pixel value of the pixel point P is greater than the pixel value of the left adjacent pixel point, and the pixel value of the pixel point P is less than the pixel value of the right adjacent pixel point. Then, based on the pixel value of the left adjacent pixel point, the coordinates of the left adjacent pixel point, the pixel value of the right adjacent pixel point, the coordinates of the right adjacent pixel point, and the pixel value of the pixel point P, the coordinates (i.e., sub-pixel coordinates) of the associated pixel point corresponding to the pixel point P in the second decoding image are calculated.

[0062] For example, the sub-pixel coordinates of the associated pixel points can be determined using the following formula (1):

[0063] Formula (1)

[0064] In formula (1), Represents the pixel value of the adjacent pixel on the left. Represents the coordinates of the adjacent pixel on the left. Represents the pixel value of the adjacent pixel on the right. Represents the coordinates of the adjacent pixel on the right. represents the pixel value of pixel point P, Indicates the sub-pixel coordinates of the associated pixel.

[0065] After obtaining the associated pixel point corresponding to the pixel point P in the second decoding image, the pixel point P and the associated pixel point may be combined into a key point pair, that is, the associated pixel points of the key point pair are sub-pixel coordinates.

[0066] In a possible implementation, after obtaining the key point pair, a binocular depth map can be obtained based on the key point pair. For example, the disparity value of the pixel point can be determined based on the key point pair, and the binocular depth map can be obtained based on the disparity value of the pixel point. For example, taking pixel point P as an example, the key point pair includes the pixel coordinates of pixel point P. The pixel coordinates of the pixel point associated with pixel point P , based on this, the disparity value of pixel point P The calculation method is: On this basis, in order to obtain the depth value corresponding to the pixel point P in the binocular depth map, the following formula (2) can be used to determine the depth value:

[0067] Formula (2)

[0068] In formula (2), Indicates the depth value corresponding to the pixel point P in the binocular depth map, Represents the distance between the two cameras. If the binocular depth map corresponding to the left camera is obtained, then represents the focal length of the left camera, Indicates the principal point coordinates of the left camera. If the binocular depth map corresponding to the right camera is obtained, then represents the focal length of the right camera, Indicates the coordinates of the principal point of the right camera.

[0069] The principal point coordinates and focal length are the internal parameters of the left camera (or right camera), and the internal parameters of the left camera (or right camera) can be calibrated in advance, and there is no restriction on the calibration process of the internal parameters.

[0070] In a possible implementation, a left camera and a right camera are used as an example for description. Figure 4 As shown, in step 302, based on the left-eye decoding image and the right-eye decoding image, a binocular matching algorithm can be used to perform binocular matching to obtain a binocular depth map. This embodiment does not limit the binocular matching algorithm.

[0071] Step 303: For each camera, obtain a monocular depth map based on the decoded map corresponding to the camera.

[0072] In a possible implementation, a left camera and a right camera are used as an example for description. Figure 4 As shown, in step 303, monocular reconstruction is performed based on the left-eye decoded image, the calibration parameters of the left-eye camera, and the calibration parameters of the projection optical machine, and a monocular depth map corresponding to the left-eye decoded image can be obtained. Monocular reconstruction is performed based on the right-eye decoded image, the calibration parameters of the right-eye camera, and the calibration parameters of the projection optical machine, and a monocular depth map corresponding to the right-eye decoded image can be obtained. This embodiment does not limit this monocular reconstruction algorithm.

[0073] Exemplarily, taking the process of obtaining a monocular depth map by monocular reconstruction based on a left-eye decoded image as an example, the monocular reconstruction process is introduced. The process of obtaining a monocular depth map by monocular reconstruction based on a right-eye decoded image is similar.

[0074] Exemplarily, in order to achieve monocular reconstruction, the correspondence between the pixel points on the left-eye decoded image and the optomechanical pixel points (i.e., the pixel points of the coded pattern) is determined to obtain multiple key point pairs, that is, the key point pairs include the pixel points on the left-eye decoded image and the optomechanical pixel points. Based on the key point pairs, the camera intrinsic parameters of the left-eye camera, the optomechanical intrinsic parameters of the projection optical machine, and the external parameters between the left-eye camera and the projection optical machine, the monocular depth map is calculated.

[0075] Regarding the correspondence between the pixel points on the left-eye decoding image and the optomechanical pixel points: Since the coding pattern is artificially designed and consists of multiple optomechanical pixel points, the decoding value (i.e., pixel value) of each optomechanical pixel point is known and the decoding value of the optomechanical pixel point is fixed.

[0076] Taking Gray code as an example, assume that the pixel value of pixel P on the left decoded image is , the phase value on the projection optical machine The coordinates of the optical machine pixel points Satisfies a known relationship, expressed as , its inverse function can be expressed as , F represents the phase value The coordinates of the optical machine pixel The functional relationship is known and can be configured based on experience. Based on this, the optical-mechanical pixel corresponding to the pixel point P on the left-eye decoded image is recorded as ,but , that is, the pixel value of pixel point P Substituting into the inverse function , we can get the optical-mechanical pixel point corresponding to the pixel point P. Obviously, after performing the above processing on each pixel point on the left-eye decoded image, we can get the optical-mechanical pixel point corresponding to each pixel point (that is, the pixel coordinates of the pixel point in the optical machine), so that we can get multiple key point pairs.

[0077] The camera intrinsic parameters of the left-eye camera, the optical and mechanical intrinsic parameters of the projection optical machine, and the external parameters between the left-eye camera and the projection optical machine can all be calibrated in advance, and there is no restriction on this calibration method.

[0078] After knowing multiple key point pairs, the camera intrinsic parameters of the left eye camera, the optical machine intrinsic parameters of the projection optical machine, and the external parameters between the left eye camera and the projection optical machine, monocular reconstruction can be performed using triangulation to obtain the monocular depth map corresponding to the left eye decoding image. There is no restriction on this monocular reconstruction process.

[0079] Step 304: Use the binocular depth map as a binocular compensation map, calculate fitting parameters of the compensation function corresponding to the monocular depth map based on the binocular depth map, and obtain the compensation function corresponding to the monocular depth map.

[0080] For example, a compensation function corresponding to a monocular depth map can be pre-configured, and the fitting parameters of the compensation function are unknown. The fitting parameters of the compensation function can be calculated based on the binocular depth map, that is, the fitting parameters of the compensation function are known. After the fitting parameters of the compensation function are known, the compensation function can be obtained.

[0081] For example, when performing monocular reconstruction to obtain a monocular depth map, the change in the state of the projection optical machine will affect the coordinates of the optical machine pixel points corresponding to the pixel points (i.e., the pixel points on the left-eye decoded image or the pixel points on the right-eye decoded image), thereby affecting the result of the monocular 3D reconstruction, that is, there is an error in the monocular depth map. When performing binocular matching to obtain a binocular depth map, since the binocular depth map is obtained based on the first decoded image and the second decoded image, and has no direct relationship with the coordinates of the optical machine pixel points, the change in the state of the projection optical machine will not affect the result of the binocular matching, that is, there is no error in the binocular depth map.

[0082] For example, in the binocular matching process, it is necessary to establish a matching relationship between the left camera and the right camera. When the state of the projection optical machine changes, it will not affect the matching relationship between the left camera and the right camera, so that the change in the state of the projection optical machine will not affect the result of the binocular matching.

[0083] In summary, when the state of the projection optical machine changes (that is, the state parameters of the projection optical machine change), the binocular depth map does not change, but the monocular depth map will change, resulting in errors in the reconstruction result of the monocular depth map. Based on this, the fitting parameters of the compensation function corresponding to the monocular depth map can be calculated based on the binocular depth map, and the compensation function corresponding to the monocular depth map can be obtained. Then, the monocular depth map can be corrected based on the compensation function, and the error can be compensated in real time and dynamically, thereby improving the measurement accuracy and environmental adaptability.

[0084] In a possible implementation, the process of calculating the fitting parameters of the compensation function corresponding to the monocular depth map based on the binocular depth map to obtain the compensation function corresponding to the monocular depth map may include:

[0085] Step S11, obtaining a monocular depth map and a binocular depth map. The method for obtaining the monocular depth map can refer to step 303, and the method for obtaining the binocular depth map can refer to step 302, which will not be repeated here.

[0086] Exemplarily, a monocular depth map may include a first pixel corresponding to a field of view overlap area and a second pixel corresponding to a non-field of view overlap area, and a binocular depth map may include a first pixel corresponding to a field of view overlap area. For example, taking the monocular depth map of the left camera as an example, the field of view overlap area refers to the field of view overlap area between the left camera and the right camera, such as Figure 2 The non-overlapping areas of a, b, and c in the figure are located in the field of view of the left camera and not in the field of view of the right camera. Figure 2 The e area in .

[0087] For another example, taking the monocular depth map of the right camera as an example, the field of view overlap area refers to the field of view overlap area of ​​the left camera and the right camera, such as Figure 2 The non-overlapping areas of a, b, and c in the figure are located in the field of view of the right camera and not in the field of view of the left camera. Figure 2 The d region in .

[0088] Exemplarily, for a monocular depth map, the pixel points in the overlapping field of view may be recorded as the first pixel point (such as multiple first pixel points), and the pixel points in the non-overlapping field of view may be recorded as the second pixel point (such as multiple second pixel points). For a binocular depth map, the pixel points in the overlapping field of view may be recorded as the first pixel point, and the first pixel point of the binocular depth map corresponds to the first pixel point of the monocular depth map.

[0089] Step S12: determining a compensation difference of the first pixel based on a pixel value corresponding to the first pixel in the monocular depth map and a pixel value corresponding to the first pixel in the binocular depth map.

[0090] For example, for a certain position point in the physical space, the position point corresponds to the first pixel point in the monocular depth map, and the position point corresponds to the first pixel point in the binocular depth map. Based on the difference between the pixel value corresponding to the first pixel point in the monocular depth map and the pixel value corresponding to the first pixel point in the binocular depth map, the compensation difference of the first pixel point can be determined. For example, the compensation difference can be determined using formula (3):

[0091] Formula (3)

[0092] In formula (3), Indicates the pixel value corresponding to the first pixel in the binocular depth map (that is, the depth value, used to indicate the distance between the position point and the left camera / right camera), Indicates the pixel value corresponding to the first pixel in the monocular depth map (that is, the depth value, used to indicate the distance between the position point and the left camera / right camera), represents the compensation difference of the first pixel. compensate , you can make the inaccurate depth value Depth value close to that obtained by binocular matching , and the depth value It is an accurate depth value, so that the depth value becomes a depth result that is not affected by the optical machine state, that is, the real value.

[0093] When compensating the monocular depth map of the left camera, Indicates the distance between the position point and the left camera. Indicates the distance between the position point and the left camera. For any first pixel in the monocular depth map (the physical position corresponding to the first pixel can be observed by both cameras at the same time, and the corresponding pixel value is normal, without overexposure and other abnormal phenomena), the difference can be compensated Therefore, a compensation function can be constructed based on multiple first pixel points in the monocular depth map to fit the compensation difference .

[0094] When compensating the monocular depth map of the right camera, Indicates the distance between the position point and the right camera. Indicates the distance between the position point and the right camera. For any first pixel in the monocular depth map, the difference can be compensated , thereby constructing a compensation function based on multiple first pixel points.

[0095] Step S13: determining fitting parameters of the compensation function based on the compensation differences of the plurality of first pixel points.

[0096] In a possible implementation, the fitting parameters of the compensation function can be determined based on the corresponding pixel values ​​of the first pixels in the decoded image (when compensating the monocular depth image of the left camera, the decoded image is the left decoded image, and when compensating the monocular depth image of the right camera, the decoded image is the right decoded image) and the compensation differences of the first pixels. For example, the compensation difference of each first pixel It can be expressed as the decoded value of the first pixel (i.e., the pixel value corresponding to the first pixel in the decoded image) Based on this, the compensation function can be a function of the pixel value in the decoded image. A polynomial function of degree, is a positive integer greater than 1, The polynomial function can be shown in formula (4).

[0097] Formula (4)

[0098] In formula (4), , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, Indicates the compensation difference of the first pixel. The pixel value corresponding to the first pixel in the decoded image is known, the compensation difference of the first pixel is known, and the pixel values ​​corresponding to the first pixels are and compensation difference After substituting into formula (4), we can get the fitting parameters , , , and the fitting parameters are obtained , , After taking the value of , we can get the compensation function shown in formula (4).

[0099] In a possible implementation, the fitting parameters of the compensation function may be determined based on the pixel values ​​corresponding to the plurality of first pixels in the decoded image (such as the left-eye decoded image or the right-eye decoded image), the pixel values ​​corresponding to the plurality of first pixels in the monocular depth image corresponding to the decoded image (such as the monocular depth image corresponding to the left-eye decoded image or the monocular depth image corresponding to the right-eye decoded image) (used to represent the depth value), and the compensation difference values ​​of the plurality of first pixels. For example, the compensation difference value of each first pixel It can be expressed as the monocular depth value of the first pixel (i.e. the pixel value corresponding to the first pixel in the monocular depth map) Polynomial functions and decoded values (i.e. the pixel value corresponding to the first pixel in the decoded image) Based on this, the compensation function can be a combination of the pixel values ​​in the decoded image The polynomial function and the pixel value for the monocular depth map A polynomial function of degree, and can be positive integers greater than 1. polynomial function and The polynomial function can be shown in formula (5).

[0100] Formula (5)

[0101] In formula (5), , , , , , , , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, Indicates the pixel value corresponding to the first pixel in the monocular depth map corresponding to the decoded image, Indicates the compensation difference of the first pixel. The pixel value corresponding to the first pixel in the decoded image It is known that the pixel value corresponding to the first pixel in the monocular depth map is is known, the compensation difference of the first pixel is known, and the pixel values ​​corresponding to the first pixels are , pixel value and compensation difference After substituting into formula (5), we can get the fitting parameters , , , , , , , , , after obtaining the values ​​of these fitting parameters, we can get the compensation function shown in formula (5).

[0102] In a possible implementation, the fitting parameters of the compensation function can be determined based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image (such as the left-eye decoded image or the right-eye decoded image), the pixel values ​​corresponding to the multiple first pixel points in the decoded image corresponding to another camera (such as the right-eye decoded image or the left-eye decoded image), and the compensation difference values ​​of the multiple first pixel points. For example, when compensating the monocular depth map of the left camera, the decoded image is the left-eye decoded image, and the decoded image corresponding to the other camera is the right-eye decoded image. When compensating the monocular depth map of the right camera, the decoded image is the right-eye decoded image, and the decoded image corresponding to the other camera is the left-eye decoded image. For example, the compensation difference value of each first pixel point It can be expressed as the decoded value of the first pixel (such as the pixel value corresponding to the first pixel in the left decoded image) The polynomial function and the decoded value of the first pixel (such as the pixel value corresponding to the first pixel in the right decoded image) Based on this, the compensation function can be a combination of the pixel values ​​in the left decoded image The polynomial function and the pixel value of the right eye decoded image A polynomial function of degree, and is a positive integer greater than 1, polynomial function and The polynomial function can be shown in formula (6).

[0103] Formula (6)

[0104] In formula (6), , , , , , represents the fitting parameters, Indicates the pixel value corresponding to the first pixel in the left decoded image, Indicates the pixel value corresponding to the first pixel in the right decoded image, Indicates the compensation difference of the first pixel. The pixel value corresponding to the first pixel in the left decoded image is known, the pixel value corresponding to the first pixel in the right decoded image is known, the compensation difference of the first pixel is known, and the pixel values ​​corresponding to the first pixels are , pixel value and compensation difference After substituting into formula (6), we can get the fitting parameters , , , , , , after obtaining the values ​​of these fitting parameters, the compensation function shown in formula (6) can be obtained.

[0105] In formula (4), formula (5) and formula (6), in order to obtain the fitting parameters, the least squares equation can be constructed to solve the fitting parameters, and there is no restriction on this process. For example, the RanSAC (Random Sample Consensus) strategy can be used to optimize the least squares method, and multiple least squares iterations can be performed to calculate the fitting parameters. In this way, by using the RanSAC strategy, the influence of abnormal decoding results that may appear in the monocular reconstruction results on the accuracy of the fitting parameters can be eliminated as much as possible.

[0106] Since the number of first pixel points that can participate in the calculation is relatively large, the image row direction can also be downsampled, that is, the number of first pixel points that can participate in the calculation is reduced. In this way, a small number of first pixel points can be selected to participate in the calculation to obtain fitting parameters, thereby greatly improving the overall calculation efficiency.

[0107] In a possible implementation, a left camera and a right camera are used as an example for description. Figure 4 As shown, in step 304, the fitting parameters of the compensation function corresponding to the monocular depth map (left eye) are calculated based on the binocular depth map to obtain the compensation function corresponding to the monocular depth map. The fitting parameters of the compensation function corresponding to the monocular depth map (right eye) are calculated based on the binocular depth map to obtain the compensation function corresponding to the monocular depth map.

[0108] Step 305: Compensate the corresponding pixel value in the monocular depth map using the compensation difference of the first pixel point, and compensate the corresponding pixel value in the monocular depth map using the compensation difference of the second pixel point to obtain a compensated image, which may be a compensated monocular depth map, and the compensated image may be used as a target depth map. The target depth map may include the distance between a position point in the physical space and the camera (i.e., the pixel value in the target depth map is the depth value), and the position point is a position point within the field of view of the camera.

[0109] Exemplarily, the monocular depth map may include a first pixel corresponding to the field of view overlapping area and a second pixel corresponding to the non-field of view overlapping area. The compensation difference of the first pixel is determined by: based on the pixel value corresponding to the first pixel in the monocular depth map and the pixel value corresponding to the first pixel in the binocular depth map, the compensation difference of the first pixel is determined. This process can be referred to step S12 and will not be repeated here. Alternatively, the compensation difference of the first pixel can be determined using the compensation function corresponding to the monocular depth map. The compensation difference of the second pixel is determined by: using the compensation function corresponding to the monocular depth map to determine the compensation difference of the second pixel. The process of determining the compensation difference of the second pixel is described below.

[0110] In a possible implementation, the pixel value corresponding to the second pixel in the decoded image and the fitting parameter can be input into the compensation function to obtain the compensation difference of the second pixel. For example, when compensating the monocular depth image of the left camera, the pixel value corresponding to the second pixel in the left decoded image (i.e., the decoded value ) and fitting parameters , , Substitute into formula (4) to obtain the compensation difference of the second pixel When compensating the monocular depth map of the right camera, the pixel value corresponding to the second pixel point in the right decoded map (i.e., the decoded value ) and the fitting parameters are substituted into formula (4) to obtain the compensation difference of the second pixel point .

[0111] Alternatively, the pixel value corresponding to the second pixel in the decoded image, the pixel value corresponding to the second pixel in the monocular depth image corresponding to the decoded image, and the fitting parameter are input into the compensation function to obtain the compensation difference of the second pixel. For example, when compensating the monocular depth image of the left camera, the pixel value corresponding to the second pixel in the left decoded image (i.e., the decoded value ), the pixel value corresponding to the second pixel in the monocular depth map corresponding to the left-eye decoded image (i.e., the monocular depth value ) and fitting parameters , , , , , , , , Substituting into formula (5), we can get the compensation difference of the second pixel: When compensating the monocular depth map of the right camera, the pixel value corresponding to the second pixel point in the right decoded map (i.e., the decoded value ), the pixel value corresponding to the second pixel in the monocular depth map corresponding to the right-eye decoded image (i.e., the monocular depth value ) and the fitting parameters are substituted into formula (5) to obtain the compensation difference of the second pixel point .

[0112] Alternatively, the pixel value corresponding to the second pixel in the decoded image, the pixel value corresponding to the second pixel in the decoded image corresponding to another camera, and the fitting parameter are input into the compensation function to obtain the compensation difference of the second pixel. For example, when compensating the monocular depth image of the left camera, the pixel value corresponding to the second pixel in the left decoded image (i.e., the decoded value ), the pixel value corresponding to the second pixel in the right-eye decoding image (i.e., the decoding value ) and fitting parameters , , , , , Substitute into formula (6) to get the compensation difference of the second pixel When compensating the monocular depth map of the right camera, the pixel value corresponding to the second pixel in the right decoded image, the pixel value corresponding to the second pixel in the left decoded image and the fitting parameter are substituted into formula (6) to obtain the compensation difference of the second pixel .

[0113] The monocular depth map may include a first pixel corresponding to the field of view overlapping area and a second pixel corresponding to the non-field of view overlapping area. For each first pixel, the pixel value of the first pixel in the monocular depth map can be compensated based on the compensation difference of the first pixel, that is, the pixel value is added to the compensation difference to obtain the compensated pixel value of the first pixel. For each second pixel, the pixel value of the second pixel in the monocular depth map can be compensated based on the compensation difference of the second pixel, that is, the pixel value is added to the compensation difference to obtain the compensated pixel value of the second pixel. After compensating the pixel values ​​of all first pixels and second pixels, the compensated pixel values ​​of all first pixels and the compensated pixel values ​​of all second pixels constitute a compensated image, and this compensated image serves as the target depth map.

[0114] In a possible implementation, a left camera and a right camera are used as an example for description. Figure 4 As shown, in step 305, the monocular depth map (left eye) is compensated based on the compensation function corresponding to the left eye camera to obtain a compensated depth map (left eye), and this compensated depth map is used as the target depth map output by the left eye camera. The monocular depth map (right eye) is compensated based on the compensation function corresponding to the right eye camera to obtain a compensated depth map (right eye), and this compensated depth map is used as the target depth map output by the right eye camera.

[0115] For example, the pixel points under the left camera For example, if the pixel The first pixel corresponding to the overlapping area of ​​the visual field can be based on the pixel point Corresponding pixel values ​​and pixel points in the monocular depth map Determine the pixel point by the corresponding pixel value in the binocular depth map The compensation difference can be used to determine the pixel point based on the compensation function. The above process does not require any information from the right camera. The second pixel point corresponding to the non-view overlap area, that is, the right camera cannot detect the pixel point Normal decoding (such as pixel The right camera cannot observe the pixel due to occlusion, or The corresponding physical position is reflected, overexposed, too dark, etc. under the right camera, then the pixel point is determined based on the compensation function compensation difference.

[0116] Step 306: Generate a fused depth map based on the target depth maps of multiple cameras. For example, the target depth map of the left camera and the target depth map of the right camera can be fused into a binocular depth map.

[0117] In a possible implementation, for each camera, first point cloud data of position points in the overlapping field of view and second point cloud data of position points in the non-overlapping field of view of the camera are determined based on the target depth map of the camera. Third point cloud data of position points in the overlapping field of view and fourth point cloud data of position points in the non-overlapping field of view of the other camera are determined based on the target depth map of the other camera. On this basis, the first point cloud data, the second point cloud data and the fourth point cloud data can be fused to obtain fused point cloud data, and a fused depth map can be generated based on the fused point cloud data.

[0118] For example, when generating a fused depth map for the left camera, the first point cloud data of the position points in the overlapping area of ​​the field of view (i.e., the overlapping area of ​​the field of view of the left camera and the right camera) and the second point cloud data of the position points in the non-overlapping area of ​​the field of view (i.e., belonging to the field of view of the left camera and not belonging to the field of view of the right camera) are determined based on the target depth map of the left camera. For example, since the target depth map includes the depth values ​​of each position point in the overlapping area of ​​the field of view, the depth values ​​of these position points constitute the first point cloud data. Since the target depth map includes the depth values ​​of each position point in the non-overlapping area of ​​the field of view, the depth values ​​of these position points constitute the second point cloud data. In addition, based on the target depth map of the right camera, the third point cloud data of the position points in the overlapping area of ​​the field of view and the fourth point cloud data of the position points in the non-overlapping area of ​​the field of view (i.e., belonging to the field of view of the right camera and not belonging to the field of view of the left camera) are determined.

[0119] On this basis, the first point cloud data, the second point cloud data and the fourth point cloud data can be fused to obtain fused point cloud data (i.e., the fused point cloud data includes the first point cloud data, the second point cloud data and the fourth point cloud data), and a fused depth map is generated based on the fused point cloud data. For example, the fused depth map includes the depth values ​​of multiple position points corresponding to the first point cloud data, the depth values ​​of multiple position points corresponding to the second point cloud data, and the depth values ​​of multiple position points corresponding to the fourth point cloud data. Obviously, the fused depth map includes the position points of the field of view of the left camera, the position points of the field of view of the right camera, and the position points of the overlapping area of ​​the field of view of the left camera and the right camera, thereby expanding the field of view of the left camera.

[0120] When generating a fused depth map for the right camera, first point cloud data of position points in the overlapping area of ​​the field of view and second point cloud data of position points in the non-overlapping area of ​​the field of view are determined based on the target depth map of the right camera. Fourth point cloud data of position points in the non-overlapping area of ​​the field of view are determined based on the target depth map of the left camera. On this basis, the first point cloud data, the second point cloud data and the fourth point cloud data can be fused to obtain fused point cloud data, and a fused depth map is generated based on the fused point cloud data.

[0121] In a possible implementation, for each camera, first point cloud data of position points in the overlapping field of view and second point cloud data of position points in the non-overlapping field of view of the camera are determined based on the target depth map of the camera. Third point cloud data of position points in the overlapping field of view and fourth point cloud data of position points in the non-overlapping field of view of the other camera are determined based on the target depth map of the other camera. On this basis, a weighted operation can be performed on the first point cloud data and the third point cloud data to obtain weighted point cloud data, and the weighted point cloud data, the second point cloud data and the fourth point cloud data can be fused to obtain fused point cloud data, and a fused depth map can be generated based on the fused point cloud data.

[0122] For example, when generating a fused depth map for the left camera, first point cloud data of the position points in the overlapping field of view and second point cloud data of the position points in the non-overlapping field of view are determined based on the target depth map of the left camera. Third point cloud data of the position points in the overlapping field of view and fourth point cloud data of the position points in the non-overlapping field of view are determined based on the target depth map of the right camera.

[0123] On this basis, since the first point cloud data and the third point cloud data are point cloud data of position points within the overlapping field of view, they can be understood as point cloud data of the same position point under different cameras. Therefore, the first point cloud data and the third point cloud data can be weighted to obtain weighted point cloud data. For example, a weighted operation is performed based on the first point cloud data, the first weight coefficient of the first point cloud data, the third point cloud data, and the second weight coefficient of the third point cloud data to obtain weighted point cloud data. The first weight coefficient and the second weight coefficient can be configured based on experience. The first weight coefficient can be greater than the second weight coefficient, the first weight coefficient can also be equal to the second weight coefficient, and the first weight coefficient can also be less than the second weight coefficient.

[0124] Then, the weighted point cloud data, the second point cloud data, and the fourth point cloud data may be fused to obtain fused point cloud data (i.e., the fused point cloud data includes the weighted point cloud data, the second point cloud data, and the fourth point cloud data), and a fused depth map may be generated based on the fused point cloud data. For example, the fused depth map may include depth values ​​of multiple position points corresponding to the weighted point cloud data, depth values ​​of multiple position points corresponding to the second point cloud data, and depth values ​​of multiple position points corresponding to the fourth point cloud data.

[0125] In a possible implementation, a left camera and a right camera are used as an example for description. Figure 4 As shown, in step 306, depth map fusion is performed based on the compensated depth map corresponding to the left camera and the compensated depth map corresponding to the right camera to obtain a fused depth map (ie, a dual-monocular fused depth map).

[0126] In a possible implementation, after obtaining the left-eye decoded image A and the right-eye decoded image B, a resolution reduction operation is performed on the left-eye decoded image A to obtain a left-eye decoded image A1, and the resolution of the left-eye decoded image A1 is lower than the resolution of the left-eye decoded image A. A resolution reduction operation is performed on the right-eye decoded image B to obtain a right-eye decoded image B1, and the resolution of the right-eye decoded image B1 is lower than the resolution of the right-eye decoded image B. Based on the left-eye decoded image A1 and the right-eye decoded image B1, binocular matching is performed to obtain a low-resolution binocular depth image C1.

[0127] In addition, based on the left eye decoded image A, the calibration parameters of the left eye camera and the calibration parameters of the projection optical machine, a monocular reconstruction is performed to obtain a full-resolution monocular depth map D. Based on the right eye decoded image B, the calibration parameters of the right eye camera and the calibration parameters of the projection optical machine, a monocular reconstruction is performed to obtain a full-resolution monocular depth map E.

[0128] Based on the binocular depth map C1, the compensation difference (i.e., the compensation depth value, which will be recorded as the monocular compensation depth value later) is determined. Since the monocular compensation depth value is not obtained by binocular matching (i.e., the full-resolution binocular depth map C is not obtained based on the left-eye decoding map A and the right-eye decoding map B, and the monocular compensation depth value is determined based on the binocular depth map C), but is obtained using the low-resolution binocular depth map C1, the accuracy of the monocular compensation depth value is low. Based on this, the three-dimensional coordinates of each pixel point can also be calculated based on the intrinsic parameters of the left camera and the monocular compensation depth value, and the three-dimensional coordinates of the pixel point are projected into the image of the right camera based on the intrinsic parameters of the left camera, the intrinsic parameters of the right camera, and the external parameters between the left camera and the right camera, and the corresponding binocular matching points are found in the local area to calculate a more accurate binocular matching depth value.

[0129] On this basis, for the pixel points where no binocular matching points are found, the monocular compensation depth value with lower accuracy can be directly used, and for the pixel points where binocular matching points are found, the binocular matching depth value with higher accuracy can be used. In summary, the binocular matching depth value and the monocular compensation depth value complement each other to become compensation information. This solution can replace the sorting matching method in the above steps and has higher efficiency.

[0130] Furthermore, the full-resolution monocular depth map D can be compensated based on the monocular compensation depth value and the binocular matching depth value to obtain a compensated monocular depth map D'. For example, for a pixel in the monocular depth map D, if the binocular matching point of the pixel is not found, the depth value of the pixel is compensated using the monocular compensation depth value. For a pixel in the monocular depth map D, if the binocular matching point of the pixel is found, the depth value of the pixel is compensated using the binocular matching depth value.

[0131] See also Figure 6 FIG. 1 is a flow chart of an image processing method, which may include:

[0132] Step 601: For each camera, obtain n frames of original images captured by the camera, and generate a decoding image based on the n frames of original images. For example, when the coding pattern is projected n times onto the object to be measured by a projection optical machine, n frames of original images are collected, and the n frames of original images correspond to the n coding patterns one by one. The n frames of original images are all images of the object to be measured, and for each frame of original images, the original image includes a coding pattern.

[0133] Step 602: For each camera, a binocular depth map is obtained based on a decoded map corresponding to the camera and a decoded map corresponding to another camera. Exemplarily, the binocular depth map may include a distance between a position point in a physical space and the camera, where the position point is a position point in an overlapping area of ​​the fields of view of the two cameras.

[0134] Step 603: For each camera, obtain a monocular depth map based on the decoded map corresponding to the camera.

[0135] Step 604: Obtain a binocular disparity map corresponding to the binocular depth map, and use the binocular disparity map as a binocular compensation map. Also, obtain a monocular disparity map corresponding to the monocular depth map.

[0136] Exemplarily, the pixel values ​​in the binocular depth map represent the depth values, and the pixel values ​​in the binocular disparity map represent the disparity values. The conversion relationship between the depth value and the disparity value can be pre-configured, and there is no restriction on this conversion relationship. After obtaining the binocular depth map, the mapping relationship is queried based on the depth value in the binocular depth map to obtain the disparity value, and then the binocular disparity map is generated based on the disparity value. After obtaining the monocular depth map, the mapping relationship is queried based on the depth value in the monocular depth map to obtain the disparity value, and then the monocular disparity map is generated based on the disparity value.

[0137] Step 605 : Calculate fitting parameters of the compensation function corresponding to the monocular disparity map based on the binocular disparity map to obtain the compensation function corresponding to the monocular disparity map. For this process, refer to step 304 .

[0138] For example, in step S12, the compensation difference can be determined using formula (3): represents the pixel value (i.e., disparity value) corresponding to the first pixel in the binocular disparity map. represents the pixel value (i.e., disparity value) corresponding to the first pixel in the monocular disparity map. Indicates the compensation difference of the first pixel. Based on the compensation difference , the fitting parameters can be determined using formula (4), formula (5) or formula (6).

[0139] Step 606: Use the compensation difference of the first pixel point to compensate the corresponding pixel value in the monocular disparity map, and use the compensation difference of the second pixel point to compensate the corresponding pixel value in the monocular disparity map to obtain a compensated image, which can be a compensated monocular disparity map. Then, determine the monocular depth map corresponding to the compensated monocular disparity map, and use the monocular depth map as the target depth map. The target depth map includes the distance between the position point in the physical space and the camera, and the position point is the position point within the field of view of the camera.

[0140] In order to determine the monocular depth map corresponding to the compensated monocular disparity map, the conversion relationship between the depth value and the disparity value can be pre-configured, and there is no restriction on this conversion relationship. The mapping relationship is queried based on the disparity value in the compensated monocular disparity map to obtain the depth value, and then the monocular depth map is generated based on the depth value.

[0141] Step 607: Generate a fused depth map based on the target depth maps of multiple cameras. For example, the target depth map of the left camera and the target depth map of the right camera can be fused into a binocular depth map.

[0142] See also Figure 7 FIG. 1 is a flow chart of an image processing method, which may include:

[0143] Step 701: for each camera, obtain n frames of original images captured by the camera, and generate a decoding image based on the n frames of original images. For example, when the coding pattern is projected n times onto the object to be measured by a projection optical machine, n frames of original images are collected, and the n frames of original images correspond to the n coding patterns one by one. The n frames of original images are all images of the object to be measured, and for each frame of original images, the original image includes a coding pattern.

[0144] Step 702: For each camera, a binocular depth map is obtained based on a decoded map corresponding to the camera and a decoded map corresponding to another camera. Exemplarily, the binocular depth map may include a distance between a position point in a physical space and the camera, where the position point is a position point in an overlapping area of ​​the fields of view of the two cameras.

[0145] Step 703: Obtain a binocular decoded image corresponding to the binocular depth image, and use the binocular decoded image as a binocular compensation image, and use the decoded image (such as a left-eye decoded image or a right-eye decoded image) as an image to be compensated.

[0146] Exemplarily, the pixel value in the binocular depth map represents the depth value. For each pixel point in the binocular depth map (taking pixel point P as an example), the depth value of pixel point P is obtained based on the binocular depth map. .

[0147] The pixel point P is a three-dimensional point, and the coordinates of the pixel point P are three-dimensional coordinates. The three-dimensional coordinates of the pixel point P can be reprojected to the pixel plane of the projection machine based on the intrinsic parameters of the camera (for example, when compensating the left-eye decoded image, the camera is the left-eye camera, and when compensating the right-eye decoded image, the camera is the right-eye camera) and the intrinsic parameters of the projection machine. Assume that the projected pixel coordinates of the pixel point P are The present embodiment does not limit the manner of reprojecting the three-dimensional coordinates to the pixel plane. For example, a transformation matrix may be constructed based on the intrinsic parameters of the camera and the intrinsic parameters of the projection optical machine, and the three-dimensional coordinates may be mapped to the pixel plane based on the transformation matrix.

[0148] The mapping function between the optical-mechanical pixel coordinates and the decoded values ​​can be pre-configured. There is no restriction on this mapping function, and it can be configured based on experience. For example, the mapping function can be recorded as , represents the optical-mechanical pixel coordinates, Represents the decoded value, Represents the mapping function. Obviously, when the projection pixel coordinates of pixel point P are Substituting this mapping function, we can get the decoded value corresponding to the pixel point P .

[0149] In summary, for each pixel in the binocular depth map, the decoding value corresponding to the pixel can be obtained by the above method, and the decoding values ​​corresponding to all pixels constitute a binocular decoding map.

[0150] Step 704: Calculate the fitting parameters of the compensation function corresponding to the left-eye decoding image (or the right-eye decoding image) based on the binocular decoding image to obtain the compensation function corresponding to the left-eye decoding image. For this process, refer to step 304.

[0151] For example, in step S12, the compensation difference can be determined using formula (3): Indicates the pixel value (i.e., decoded value) corresponding to the first pixel in the binocular decoding image. Indicates the pixel value (decoded value) corresponding to the first pixel in the left-view decoded image. Indicates the compensation difference of the first pixel. Based on the compensation difference , the fitting parameters can be determined using formula (4), formula (5) or formula (6).

[0152] Step 705: Use the compensation difference of the first pixel point to compensate the corresponding pixel value in the left-eye decoded image, and use the compensation difference of the second pixel point to compensate the corresponding pixel value in the left-eye decoded image to obtain a compensated image, which can be a compensated decoded image. Then, determine the monocular depth map corresponding to the compensated decoded image, and use the monocular depth map as the target depth map. The target depth map includes the distance between the position point in the physical space and the camera, and the position point is the position point within the field of view of the camera.

[0153] For the process of determining the monocular depth map corresponding to the compensated decoded image, please refer to step 303.

[0154] Step 706: Generate a fused depth map based on the target depth maps of multiple cameras. For example, the target depth map of the left camera and the target depth map of the right camera can be fused into a binocular depth map.

[0155] It can be seen from the above technical solutions that in the embodiment of the present application, a binocular depth map can be obtained based on the decoded images corresponding to multiple cameras, and three-dimensional reconstruction can be achieved based on the binocular depth map. The three-dimensional reconstruction effect is very good, the reconstruction accuracy is higher, and the robustness is better. A binocular compensation map can be obtained based on the binocular depth map, and the compensated image can be obtained by compensating the compensation map based on the binocular compensation map. The target depth map is obtained based on the compensated image, so as to perform real-time dynamic compensation for the error, improve the measurement accuracy and environmental adaptability, and improve the quality and accuracy of the target depth map. When three-dimensional reconstruction is achieved based on the target depth map, the error effect can be eliminated to obtain an accurate and reliable three-dimensional reconstructed image. A dynamic error compensation method is proposed. By calculating the difference between the binocular depth value and the monocular depth value, a global fitting function of the difference is fitted to correct the drift of the monocular depth value due to the change of the optical and mechanical parameters under the influence of the environment, thereby improving the quality and accuracy of the dual-monocular fusion depth map. The scheme adopts the least squares fitting method to fit the global fitting parameters, which can downsample the image, has simple calculation and high efficiency, and is suitable for real-time and dynamic correction of the dual-monocular fusion depth map.

[0156] Based on the same application concept as the above method, an image processing device is proposed in the embodiment of the present application, which is applied to a three-dimensional imaging device, wherein the three-dimensional imaging device includes a projection light machine and at least two cameras, see Figure 8As shown, it is a structural schematic diagram of the image processing device, and the device may include: an acquisition module 81, which is used to acquire n frames of original images collected by each camera; wherein, when the coding pattern is projected n times to the object to be measured by the projection optical machine, the n frames of original images correspond to the n coding patterns one by one; a decoding image is generated based on the n frames of original images, and a binocular depth map is acquired based on the decoding image corresponding to the camera and the decoding image corresponding to another camera, wherein the binocular depth map includes the distance between a position point in the physical space and the camera, and the position point is a position point in the overlapping area of ​​the field of view of the two cameras; a to-be-compensated image is acquired based on the decoding image, and a binocular compensation image is acquired based on the binocular depth map; a determination module 82, which is used to determine fitting parameters based on the to-be-compensated image and the binocular compensation image; the to-be-compensated image is compensated based on the fitting parameters to obtain a compensated image; a processing module 83, which is used to acquire a target depth map of the camera based on the compensated image.

[0157] Exemplarily, the image to be compensated includes a first pixel corresponding to the field of view overlapping area and a second pixel corresponding to the non-field of view overlapping area, and the binocular compensation image includes the first pixel corresponding to the field of view overlapping area; the determination module 82 determines fitting parameters based on the image to be compensated and the binocular compensation image; compensates the image to be compensated based on the fitting parameters to obtain a compensated image, which is specifically used to: determine the compensation difference of the first pixel based on the pixel value corresponding to the first pixel in the image to be compensated and the pixel value corresponding to the first pixel in the binocular compensation image; determine the fitting parameters of the compensation function based on the compensation differences of multiple first pixels, and determine the compensation difference of the second pixel based on the compensation function;

[0158] The compensation difference of the first pixel point is used to compensate the corresponding pixel value in the image to be compensated, and the compensation difference of the second pixel point is used to compensate the corresponding pixel value in the image to be compensated, so as to obtain a compensated image.

[0159] Exemplarily, the binocular compensation map may be the binocular depth map, the map to be compensated may be the monocular depth map corresponding to the decoded map, the compensated image may be the compensated monocular depth map, and the target depth map may be the compensated monocular depth map; or, the binocular compensation map may be the binocular disparity map corresponding to the binocular depth map, the map to be compensated may be the monocular disparity map corresponding to the decoded map, the compensated image may be the compensated monocular disparity map, and the target depth map may be the monocular depth map corresponding to the compensated monocular disparity map; or, the binocular compensation map may be the binocular decoded map corresponding to the binocular depth map, the map to be compensated may be the decoded map, the compensated image may be the compensated decoded map, and the target depth map may be the monocular depth map corresponding to the compensated decoded map.

[0160] Exemplarily, when the determination module 82 determines the fitting parameters of the compensation function based on the compensation differences of multiple first pixels, it is specifically used to: determine the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixels in the decoded image and the compensation differences of the multiple first pixels; or, determine the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixels in the decoded image, the pixel values ​​corresponding to the multiple first pixels in the monocular depth map corresponding to the decoded image, and the compensation differences of the multiple first pixels; or, determine the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixels in the decoded image, the pixel values ​​corresponding to the multiple first pixels in the decoded image corresponding to the other camera, and the compensation differences of the multiple first pixels.

[0161] Exemplarily, the compensation function includes: for the pixel value in the decoded image A polynomial function of degree, is a positive integer greater than 1, The polynomial function is expressed as: ; , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, represents the compensation difference of the first pixel; or, for the pixel value in the decoded image The polynomial function and the pixel value for the monocular depth map A polynomial function of degree, and is a positive integer greater than 1, polynomial function and The polynomial function is expressed as: ; , , , , , , , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, Indicates the pixel value corresponding to the first pixel in the monocular depth map corresponding to the decoded image, represents the compensation difference of the first pixel; or, for the pixel value in the decoded image The polynomial function and the pixel value in the decoded image corresponding to another camera A polynomial function of degree, and is a positive integer greater than 1, polynomial function and The polynomial function is expressed as: ; , , , , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, represents the pixel value corresponding to the first pixel in the decoded image corresponding to the other camera, Indicates the compensation difference of the first pixel.

[0162] Exemplarily, when the determination module 82 determines the compensation difference of the second pixel point based on the compensation function, it is specifically used to: input the pixel value corresponding to the second pixel point in the decoded image and the fitting parameter into the compensation function to obtain the compensation difference of the second pixel point; or, input the pixel value corresponding to the second pixel point in the decoded image, the pixel value corresponding to the second pixel point in the monocular depth map corresponding to the decoded image, and the fitting parameter into the compensation function to obtain the compensation difference of the second pixel point; or, input the pixel value corresponding to the second pixel point in the decoded image, the pixel value corresponding to the second pixel point in the decoded image corresponding to the other camera, and the fitting parameter into the compensation function to obtain the compensation difference of the second pixel point.

[0163] Exemplarily, the processing module 83 is further configured to, after acquiring the target depth map of the camera based on the compensated image, determine first point cloud data of the position points in the field of view overlapping area and second point cloud data of the position points in the non-field of view overlapping area of ​​the camera based on the target depth map of the camera; determine third point cloud data of the position points in the field of view overlapping area and fourth point cloud data of the position points in the non-field of view overlapping area of ​​the other camera based on the target depth map of the other camera;

[0164] The first point cloud data, the second point cloud data and the fourth point cloud data are fused to obtain fused point cloud data, and a fused depth map is generated based on the fused point cloud data; or, a weighted operation is performed on the first point cloud data and the third point cloud data to obtain weighted point cloud data, the weighted point cloud data, the second point cloud data and the fourth point cloud data are fused to obtain fused point cloud data, and a fused depth map is generated based on the fused point cloud data.

[0165] Exemplarily, when the acquisition module 81 acquires the binocular depth map based on the decoded image corresponding to the camera and the decoded image corresponding to another camera, it is specifically used to: acquire a key point pair corresponding to a pixel point in the first decoded image, the key point pair including the pixel point and an associated pixel point corresponding to the pixel point in the second decoded image, the first decoded image is the decoded image corresponding to the camera, and the second decoded image is the decoded image corresponding to another camera; acquire the binocular depth map based on multiple key point pairs; wherein, when acquiring the associated pixel point corresponding to the pixel point in the second decoded image, if the associated pixel point is an integer pixel point, then Traverse to the associated pixel point, and the pixel value of the pixel point is the same as the pixel value of the associated pixel point; or, if the associated pixel point is a sub-pixel point, traverse to the left adjacent pixel point and the right adjacent pixel point from the second decoding image, the pixel value of the pixel point is greater than the pixel value of the left adjacent pixel point, and the pixel value of the pixel point is less than the pixel value of the right adjacent pixel point; calculate the coordinates of the associated pixel point based on the pixel value of the left adjacent pixel point, the coordinates of the left adjacent pixel point, the pixel value of the right adjacent pixel point, the coordinates of the right adjacent pixel point, and the pixel value of the pixel point.

[0166] Based on the same application concept as the above method, an electronic device (such as a three-dimensional imaging device) is proposed in the embodiment of the present application, see Fig. 9 As shown, it includes: a processor 91 and a machine-readable storage medium 92, the machine-readable storage medium 92 stores machine-executable instructions that can be executed by the processor 91; the processor 91 is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of this application.

[0167] The aforementioned electronic device may be a 3D camera or a back-end image processing device, such as an image acquisition card, a PC, a server, etc. For example, the projection optical machine is used to project the coded pattern onto the object to be measured, and each camera is used to collect the original image of the object to be measured, and provide the original image to the image processing device, and the image processing device implements the image processing method disclosed in the above example based on these original images.

[0168] Based on the same application concept as the above method, an electronic device is proposed in the embodiment of the present application, which can be a three-dimensional imaging device (also called a three-dimensional reconstruction device), and the three-dimensional imaging device can include a projection optical machine, at least two cameras and a processor. For example, the projection optical machine is used to project a coded pattern onto the object to be measured, and each camera is used to collect the original image of the object to be measured. Based on these original images, the processor is used to execute machine executable instructions to implement the image processing method disclosed in the above example of the present application.

[0169] The at least two cameras may be at least two monocular depth cameras, that is, each camera combined with a projection optical machine may constitute a monocular depth camera. Alternatively, the at least two cameras may be binocular depth cameras, that is, at least two cameras constitute a binocular depth camera, and the projection optical machine only provides pattern texture information.

[0170] For example, see Figure 2 As shown, CAM1 represents the left eye camera, CAM2 represents the right eye camera, PROJ represents the projection optical machine, and the two monocular depth cameras refer to: the data of CAM1 and PROJ complete the reconstruction of the left eye system, and the data of CAM2 and PROJ complete the reconstruction of the right eye system.

[0171] Binocular depth camera means: the data of CAM1 and CAM2 complete the reconstruction of the binocular system.

[0172] As for the projection optical machine, the projection optical machine is used to project a specific pattern onto the object to be measured, and the projection optical machine may include but is not limited to speckle DOE, DLP projector, LCD projector, LCOS projector, laser galvanometer optical machine, etc. Of course, the above are just a few examples of projection optical machines, and there is no limitation to this.

[0173] Based on the same application concept as the above method, a depth camera is proposed in the embodiment of the present application, and the depth camera can be deployed in a three-dimensional imaging device. The depth camera can capture the original image and provide the original image to the three-dimensional imaging device so that the three-dimensional imaging device performs relevant processing. For example, the depth camera can be at least two monocular depth cameras or binocular depth cameras, and there is no limitation on this.

[0174] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above example of the present application can be implemented.

[0175] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0176] Based on the same application concept as the above method, an embodiment of the present application further provides a computer program product, which may include a computer program, and when the computer program is executed by a processor, it implements the image processing method disclosed in the above example of the present application.

[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0178] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: Applied to a three-dimensional imaging device, the three-dimensional imaging device includes a projection optical machine and at least two cameras, and the method includes: For each camera, n frames of original images captured by the camera are obtained; wherein when the n-times coded patterns are projected onto the object to be measured by the projection optical machine, the n-frames of original images correspond to the n-times coded patterns one by one; Generate a decoding image based on the n frames of original images, and obtain a binocular depth image based on the decoding image corresponding to the camera and the decoding image corresponding to another camera; Acquire a to-be-compensated image based on the decoded image, and acquire a binocular compensation image based on the binocular depth image; Determining fitting parameters based on the image to be compensated and the binocular compensation image; compensating the image to be compensated based on the fitting parameters to obtain a compensated image; Acquire a target depth map of the camera based on the compensated image; Wherein, the image to be compensated includes a first pixel point corresponding to a field of view overlapping area and a second pixel point corresponding to a non-field of view overlapping area, and the binocular compensation image includes a first pixel point corresponding to the field of view overlapping area; the fitting parameters are determined based on the image to be compensated and the binocular compensation image; the image to be compensated is compensated based on the fitting parameters to obtain a compensated image, including: determining a compensation difference of the first pixel point based on a pixel value corresponding to the first pixel point in the image to be compensated and a pixel value corresponding to the first pixel point in the binocular compensation image; determining a fitting parameter of a compensation function based on the compensation differences of multiple first pixels, and determining a compensation difference of a second pixel point based on the compensation function; using the compensation difference of the first pixel point to compensate the corresponding pixel value in the image to be compensated, and using the compensation difference of the second pixel point to compensate the corresponding pixel value in the image to be compensated, to obtain a compensated image; Among them, the determining the fitting parameters of the compensation function based on the compensation differences of multiple first pixel points includes: determining the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image and the compensation differences of the multiple first pixel points; or, determining the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image, the pixel values ​​corresponding to the multiple first pixel points in the monocular depth map corresponding to the decoded image, and the compensation differences of the multiple first pixel points; or, determining the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image, the pixel values ​​corresponding to the multiple first pixel points in the decoded image corresponding to the other camera, and the compensation differences of the multiple first pixel points.

2. The method according to claim 1, characterized in that The binocular compensation map is the binocular depth map, the image to be compensated is the monocular depth map corresponding to the decoded image, the compensated image is the compensated monocular depth map, and the target depth map is the compensated monocular depth map; Alternatively, the binocular compensation image is a binocular disparity image corresponding to the binocular depth image, the image to be compensated is a monocular disparity image corresponding to the decoded image, the compensated image is a compensated monocular disparity image, and the target depth map is a monocular depth map corresponding to the compensated monocular disparity image; Alternatively, the binocular compensation image is a binocular decoded image corresponding to the binocular depth image, the image to be compensated is the decoded image, the compensated image is the compensated decoded image, and the target depth map is a monocular depth map corresponding to the compensated decoded image.

3. The method according to claim 1, characterized in that The compensation function includes: For the pixel values ​​in the decoded image A polynomial function of degree, is a positive integer greater than 1, The polynomial function is expressed as: ;in, , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, represents the compensation difference of the first pixel; Or, for the pixel values ​​in the decoded image The polynomial function and the pixel value for the monocular depth map A polynomial function of degree, and is a positive integer greater than 1, polynomial function and The polynomial function is expressed as: ;in, , , , , , , , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, Indicates the pixel value corresponding to the first pixel in the monocular depth map corresponding to the decoded image, represents the compensation difference of the first pixel; Or, for the pixel values ​​in the decoded image The polynomial function and the pixel value in the decoded image corresponding to another camera A polynomial function of degree, and is a positive integer greater than 1, polynomial function and The polynomial function is expressed as: ;in, , , , , , represents the fitting parameters, represents the pixel value corresponding to the first pixel in the decoded image, represents the pixel value corresponding to the first pixel in the decoded image corresponding to the other camera, Indicates the compensation difference of the first pixel.

4. The method according to claim 1, characterized in that: The determining the compensation difference of the second pixel point based on the compensation function includes: Inputting the pixel value corresponding to the second pixel point in the decoded image and the fitting parameter into the compensation function to obtain a compensation difference value of the second pixel point; or, Inputting the pixel value corresponding to the second pixel in the decoded image, the pixel value corresponding to the second pixel in the monocular depth image corresponding to the decoded image, and the fitting parameter into the compensation function to obtain a compensation difference value of the second pixel; or The pixel value corresponding to the second pixel in the decoded image, the pixel value corresponding to the second pixel in the decoded image corresponding to the other camera and the fitting parameter are input into the compensation function to obtain the compensation difference of the second pixel.

5. The method according to claim 1, characterized in that After acquiring the target depth map of the camera based on the compensated image, the method further includes: Determine first point cloud data of position points in the field of view overlap area and second point cloud data of position points in the non-field of view overlap area of ​​the camera based on the target depth map of the camera; Determine, based on the target depth map of the other camera, third point cloud data of the position points in the field of view overlapping area and fourth point cloud data of the position points in the non-field of view overlapping area of ​​the other camera; fusing the first point cloud data, the second point cloud data and the fourth point cloud data to obtain fused point cloud data, and generating a fused depth map based on the fused point cloud data; Alternatively, a weighted operation is performed on the first point cloud data and the third point cloud data to obtain weighted point cloud data, the weighted point cloud data, the second point cloud data and the fourth point cloud data are fused to obtain fused point cloud data, and a fused depth map is generated based on the fused point cloud data.

6. The method according to claim 1, characterized in that The obtaining of a binocular depth map based on a decoded map corresponding to the camera and a decoded map corresponding to another camera includes: Obtain a key point pair corresponding to a pixel point in a first decoded image, the key point pair including the pixel point and an associated pixel point corresponding to the pixel point in a second decoded image, the first decoded image is a decoded image corresponding to the camera, and the second decoded image is a decoded image corresponding to another camera; obtain a binocular depth image based on multiple key point pairs; When obtaining the associated pixel point corresponding to the pixel point in the second decoding image, If the associated pixel is an integer pixel, the associated pixel is traversed from the second decoding graph, and the pixel value of the pixel is the same as the pixel value of the associated pixel; or, If the associated pixel point is a sub-pixel point, then traverse the left adjacent pixel point and the right adjacent pixel point from the second decoding image, the pixel value of the pixel point is greater than the pixel value of the left adjacent pixel point, and the pixel value of the pixel point is less than the pixel value of the right adjacent pixel point; based on the pixel value of the left adjacent pixel point, the coordinates of the left adjacent pixel point, the pixel value of the right adjacent pixel point, the coordinates of the right adjacent pixel point, and the pixel value of the pixel point, calculate the coordinates of the associated pixel point.

7. An image processing device, characterized in that: Applied to a three-dimensional imaging device, the three-dimensional imaging device includes a projection light machine and at least two cameras, and the device includes: An acquisition module is used to acquire, for each camera, n frames of original images captured by the camera; wherein, when the coding pattern is projected n times onto the object to be measured by the projection optical machine, the n frames of original images correspond to the n coding patterns one by one; a decoding image is generated based on the n frames of original images, and a binocular depth image is acquired based on the decoding image corresponding to the camera and the decoding image corresponding to another camera; Acquire a to-be-compensated image based on the decoded image, and acquire a binocular compensation image based on the binocular depth image; A determination module, configured to determine fitting parameters based on the image to be compensated and the binocular compensation image; and to compensate the image to be compensated based on the fitting parameters to obtain a compensated image; A processing module, configured to obtain a target depth map of the camera based on the compensated image; Wherein, the image to be compensated includes a first pixel point corresponding to a field of view overlapping area and a second pixel point corresponding to a non-field of view overlapping area, and the binocular compensation image includes a first pixel point corresponding to the field of view overlapping area; the determination module determines fitting parameters based on the image to be compensated and the binocular compensation image; compensates the image to be compensated based on the fitting parameters, and when obtaining a compensated image, is specifically used to: determine a compensation difference of a first pixel point based on a pixel value corresponding to the first pixel point in the image to be compensated and a pixel value corresponding to the first pixel point in the binocular compensation image; determine a fitting parameter of a compensation function based on the compensation differences of multiple first pixels, and determine a compensation difference of a second pixel point based on the compensation function; use the compensation difference of the first pixel point to compensate the corresponding pixel value in the image to be compensated, and use the compensation difference of the second pixel point to compensate the corresponding pixel value in the image to be compensated, to obtain a compensated image; Among them, when the determination module determines the fitting parameters of the compensation function based on the compensation differences of multiple first pixel points, it is specifically used to: determine the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image and the compensation differences of the multiple first pixel points; or, determine the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image, the pixel values ​​corresponding to the multiple first pixel points in the monocular depth map corresponding to the decoded image, and the compensation differences of the multiple first pixel points; or, determine the fitting parameters of the compensation function based on the pixel values ​​corresponding to the multiple first pixel points in the decoded image, the pixel values ​​corresponding to the multiple first pixel points in the decoded image corresponding to the other camera, and the compensation differences of the multiple first pixel points.

8. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-6.

Images