A depth map extraction method based on RGBIR binocular camera

Abstract

The present invention relates to the field of image processing technology, and in particular to a depth map extraction method based on an RGBIR binocular camera, comprising obtaining two left and right original viewpoint images of the same scene, and interpolating the two left and right original viewpoint images into one color visible light image and the other black-and-white near-infrared image; generating corresponding initial disparity images and confidence maps according to the color visible light image and the black-and-white near-infrared image respectively; fusing the generated disparity images and confidence maps into a final disparity image; and converting the final disparity image into a depth map. The present invention can make full use of color information so that no misjudgment occurs when stereo matching is achieved. In addition, the present application fuses near-infrared to obtain an object image with clearer texture, so that the final depth map has higher reliability.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a depth map extraction method based on an RGBIR binocular camera. Background Art

[0002] Calculating depth maps based on binocular stereo matching is one of the mainstream methods for obtaining depth maps. The principle is that a binocular camera takes two left and right viewpoint images of the same scene, uses a stereo matching algorithm to generate a disparity map of the two images, and finally converts the disparity map into a depth map based on the camera parameters.

[0003] Binocular stereo matching depth map is widely used in virtual reality, car assisted driving, robotics, drones, security and other industries, mainly to achieve three-dimensional imaging, liveness detection, body analysis and obstacle avoidance functions.

[0004] The RGBIR image sensor is a dual-spectrum image sensor that can both sense near-infrared light and restore colors similar to those perceived by the human eye. Compared with ordinary image sensors that only sense visible light, it has stronger photosensitivity in low-light environments, and compared with ordinary near-infrared image sensors, it can restore colors similar to those perceived by the human eye. Therefore, it is widely used in automobiles, security, robotics and other fields.

[0005] At present, the depth map of binocular stereo matching is generated based on the images taken by ordinary color cameras that only perceive visible light or black and white cameras that can perceive near-infrared light. For example, the invention with application number CN201010235455.3 generates depth maps based on black and white images. This solution does not use color information and is prone to misjudgment when implementing stereo matching, especially for objects with similar gray values ​​but different color values. Application number CN201310634040.7 generates depth maps based on ordinary color cameras. Because near-infrared light is filtered out, the texture of objects is not clear in low light and is also prone to misjudgment, especially in some scenes that can be supplemented with infrared light. Summary of the invention

[0006] (I) Purpose of the invention

[0007] In order to solve the technical problems existing in the background technology, the present invention proposes a depth map extraction method based on an RGBIR binocular camera.

[0008] (II) Technical solution

[0009] To solve the above problems, the present invention provides a depth map extraction method based on an RGBIR binocular camera, comprising the steps of:

[0010] Step S100, obtaining two left and right original viewpoint images of the same scene, and interpolating the two left and right original viewpoint images into one color visible light image and the other black and white near infrared image;

[0011] Step S200, generating corresponding initial disparity images and confidence maps according to the color visible light image and the black-and-white near-infrared image respectively;

[0012] Step S300, fusing the generated disparity image and the confidence map into a final disparity image;

[0013] Step S400: converting the final disparity image into a depth map.

[0014] As a technical solution of the present invention, in step S100, the two original viewpoint images on the left and right are interpolated into a color visible light image and a black and white near infrared image, and an RGBIR image demosaicing algorithm is used, including the steps of:

[0015] Step S101, judging the color of the current pixel at the coordinate position according to the coordinate position of the current pixel; the color of the current pixel includes red R, green G, blue B, infrared IR, setting the RGBI R4X4 color array, and the first pixel in the upper left corner is blue B;

[0016] According to the horizontal and vertical coordinate positions i and j of the current pixel, the color is marked as follows:

[0017]

[0018] In the formula, Cfa ij is the color mark of coordinate position i,j;

[0019] Step S102: taking the current pixel P ij As the center, take the current pixel P ij The horizontal and vertical gradient values ​​of the pixels in the 5x5 area are calculated respectively to determine the edge direction of the current pixel;

[0020] Step S103, calculating the green G pixel value according to the results of step S101 and step S102, and if the current pixel is a green pixel, outputting the original value; if the current pixel is not green, interpolating the value using adjacent pixels according to the edge direction;

[0021] Step S104, calculating the red R and blue B pixel values ​​according to the results of step S101 and step S103, and combining them with the green G pixel value in step S103, outputting a color visible light image;

[0022] Step S105 , calculating the infrared IR pixel value according to the results of step S101 and step S103 , and outputting a black and white near infrared image.

[0023] As a technical solution of the present invention, in step S200, a stereo matching method is used to respectively generate corresponding initial disparity images and confidence maps, including the steps of:

[0024] Step S201, calculate the sum of the differences between the pixel block in the 5x5 area of ​​the color visible light image centered on the current pixel of the left view and the right view; the formula is as follows:

[0025]

[0026]

[0027]

[0028] SRGB i,j,p =SR i,j,p +SG i,j,p +SB i,j,p

[0029] In the above formula, SR i,j,p is the disparity value of the red component at coordinates i, j and disparity p, SG i,j,p is the green component disparity value, SB i,j,p is the blue component disparity value, SRGB i,j,p is the disparity value of visible light color, RL is the left view of the red component, RR is the right view of the red component, GL is the left view of the green component, GR is the right view of the green component, BL is the left view of the blue component, BR is the right view of the blue component, m is the horizontal offset relative to the current pixel coordinate position, and n is the vertical offset relative to the current pixel coordinate position;

[0030] Step S202, finding the disparity position of the color visible light image with the minimum disparity value, which is the disparity of the current pixel, to generate an initial disparity image of the color visible light image; as shown in the following formula:

[0031] Index

[0032] In the above formula, Pmax is the disparity search range, and the minimum disparity value is found within the disparity range [0pmax];

[0033] Step S203, calculating a confidence map of the initial disparity image of the color visible light image, includes:

[0034] Get the minimum disparity value;

[0035]

[0036] Calculate the average difference ERGB of the current pixel in the left view of the color visible light image i,j ;

[0037]

[0038] In the above formula, is the 5x5 area average of the red component of the current pixel, is the mean of the green component, blue component mean;

[0039] The confidence CRGB of the current pixel of the color visible light image is obtained according to the minimum disparity value and the average difference i,j ;

[0040]

[0041] In the above formula, CRGB i,j The larger the value, the more reliable the disparity value of the color visible light image;

[0042] Step S204, calculating the difference between the pixel block in the 5x5 area of ​​the black and white near-infrared image centered on the current pixel of the left view and the pixel block in the right view, the formula is as follows:

[0043]

[0044] In the above formula, SIR i,j,p is the disparity value of the black-and-white near-infrared image with coordinates i, j and disparity p, I RL is the left view of the black-and-white near-infrared image, and I RR is the right view of the black-and-white near-infrared image;

[0045] Step S205, finding the parallax position of the black-and-white near-infrared image with the minimum parallax value, which is the parallax of the current pixel, to generate an initial parallax image of the black-and-white near-infrared image; as shown in the following formula:

[0046]

[0047] In the above formula, pmax is the disparity search range, and the minimum disparity value is found within the disparity range [0pmax];

[0048] Step S206, calculating the confidence map of the initial disparity image of the black and white near-infrared image, including:

[0049] (1) Obtain the minimum disparity value SIRmin i,j :

[0050]

[0051] (4) Calculate the mean difference EIR of the current pixel in the left view of the black and white near-infrared image i,j:

[0052]

[0053] In the above formula, The mean value of the 5x5 area of ​​the near-infrared image of the current pixel in black and white;

[0054] (5) The confidence CIR of the current pixel of the black-and-white near-infrared image is obtained based on the minimum disparity value and the average difference i,j :

[0055]

[0056] Among them, CIR i,j The larger the value, the more reliable the parallax value of the black and white near-infrared image.

[0057] As a technical solution of the present invention, in step S300, the generated disparity image and the confidence map are fused into a final disparity image, including:

[0058] Step S301, filter the current pixel of the confidence map of the color visible light image and the black and white near infrared image respectively, the formula is as follows:

[0059]

[0060]

[0061] In the above formula, flt is a 5x5 constant filter;

[0062] Step S302: compare the confidence map of the colored visible light image and the filtered black and white near infrared image pixel by pixel, and select the parallax value with the larger confidence as the final parallax image Pout after fusion. i,j , the formula is as follows:

[0063]

[0064] As a technical solution of the present invention, in step S400, the final disparity image is converted into a depth map by the following formula:

[0065] D i,j =fx*dx / Pout i,j

[0066] In the above formula, fx is the focal length of the lens, and dx is the distance between the optical centers of the left and right cameras.

[0067] The above technical solution of the present invention has the following beneficial technical effects:

[0068] The present invention uses an RGBIR binocular camera to capture two left and right original viewpoint images of the same scene, and interpolates the two left and right original viewpoint images into a color visible light image and a black-and-white near-infrared image, and uses a stereo matching algorithm to respectively generate disparity images and confidence maps of the visible light color image and the near-infrared black-and-white image, and uses a disparity map fusion algorithm to fuse them into a pair of disparity images, and converts the finally generated disparity image into a depth map. Therefore, the present application can make full use of color information so that there will be no misjudgment when implementing stereo matching. In addition, the present application fuses near-infrared to obtain an object image with clearer texture, so the final depth map has higher reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of the method of the present invention;

[0070] Figure 2 This is a schematic diagram of the RGBI R4x4 array in the present invention;

[0071] Figure 3 A schematic diagram of a formula for outputting an RGB color image in one embodiment of the present invention;

[0072] Figure 4 Schematic diagram of the formula for outputting an IR graph in one embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0074] refer to Figure 1-2 To solve the above problems, this application adopts the following technical solutions:

[0075] The present invention provides a depth map extraction method based on an RGBIR binocular camera, comprising the steps of:

[0076] Step S100, obtaining two left and right original viewpoint images of the same scene, and interpolating the two left and right original viewpoint images into one color visible light image and the other black and white near infrared image;

[0077] Specifically, the RGBIR image demosaicing algorithm is used, including the following steps:

[0078] Step S101, determine the color of the current pixel, that is, determine the color of the current pixel at the coordinate position according to the coordinate position of the current pixel. The color of the current pixel is red R, green G, blue B, infrared IR. The present invention supports the RGBIR4X4 color array, and the first pixel in the upper left corner is blue B. The specific color array is as follows: Figure 2 As shown; according to the horizontal and vertical coordinate positions i and j of the current pixel, the color is marked as follows:

[0079]

[0080] In the formula, Cfa ij is the color mark of coordinate position i,j;

[0081] Step S102: determine the edge direction of the current pixel. ij As the center, take the current pixel P ij The horizontal and vertical gradient values ​​of the pixels in the 5x5 area are calculated respectively to determine the edge direction of the current pixel;

[0082] The calculation formula of the horizontal gradient value is as follows:

[0083]

[0084] The vertical gradient value calculation formula is as follows:

[0085]

[0086] In the above two equations, m is the horizontal offset relative to the current pixel coordinate position, n is the vertical offset relative to the current pixel coordinate position, Δ h is the calculated horizontal gradient sum, Δ v is the calculated sum of the vertical gradients.

[0087] Determine the current pixel P according to the following relationship i,j The edge direction:

[0088]

[0089] In the above formula, sc is a constant with a range of

[12] , Edge is the edge direction, 0 means that the pixel is located in the non-texture area, 1 means that the pixel is located at the horizontal edge, and 2 means that the pixel is located at the vertical edge.

[0090] Step S103, calculate the green G pixel value according to the results of step S101 and step S102, and if the current pixel is a green pixel, output the original value; G i,j =P i,j .

[0091] If the current pixel is not green, Cfa ij <4, then interpolate using neighboring pixels according to the edge direction;

[0092]

[0093] Step S104, calculate the red R and blue B pixel values ​​according to the results of step S101 and step S103, and combine them with the green G pixel value in step S103 to output a color visible light image (RGB color image, the specific formula is not described here, see Figure 3 );

[0094] Step S105, calculate the infrared IR pixel value according to the results of step S101 and step S103, and output a black and white near infrared image (IR image, see the specific calculation formula Figure 4 ).

[0095] Step S200, generating corresponding initial disparity images and confidence maps according to the color visible light image and the black-and-white near-infrared image respectively;

[0096] In one embodiment, a stereo matching method is used to generate corresponding initial disparity images and confidence maps, including the steps of:

[0097] Step S201, calculate the sum of the differences between the pixel block in the 5x5 area of ​​the color visible light image centered on the current pixel of the left view and the right view; the formula is as follows:

[0098]

[0099]

[0100]

[0101] SRGB i,j,p =SR i,j,p +SG i,j,p +SB i,j,p

[0102] In the above formula, SR i,j,p is the disparity value of the red component at coordinates i, j and disparity p, SG i,j,p is the green component disparity value, SB i,j,p is the blue component disparity value, SRGB i,j,p is the disparity value of visible light color, RL is the left view of the red component, RR is the right view of the red component, GL is the left view of the green component, GR is the right view of the green component, BL is the left view of the blue component, BR is the right view of the blue component, m is the horizontal offset relative to the current pixel coordinate position, and n is the vertical offset relative to the current pixel coordinate position;

[0103] Step S202, finding the disparity position of the color visible light image with the minimum disparity value, which is the disparity of the current pixel, to generate an initial disparity image of the color visible light image; as shown in the following formula:

[0104] Index

[0105] In the above formula, Pmax is the disparity search range, and the minimum disparity value is found within the disparity range [0pmax];

[0106] Step S203, calculating a confidence map of the initial disparity image of the color visible light image, includes:

[0107] Get the minimum disparity value;

[0108]

[0109] Calculate the average difference ERGB of the current pixel in the left view of the color visible light image i,j ;

[0110]

[0111] In the above formula, is the 5x5 area average of the red component of the current pixel, is the mean of the green component, blue component mean;

[0112] The confidence CRGB of the current pixel of the color visible light image is obtained according to the minimum disparity value and the average difference i,j ;

[0113]

[0114] In the above formula, CRGB i,j The larger the value, the more reliable the disparity value of the color visible light image;

[0115] Step S204, calculating the difference between the pixel block in the 5x5 area of ​​the black and white near-infrared image centered on the current pixel of the left view and the pixel block in the right view, the formula is as follows:

[0116]

[0117] In the above formula, SIR i,j,p is the disparity value of the black-and-white near-infrared image with coordinates i, j and disparity p, I RL is the left view of the black-and-white near-infrared image, and I RR is the right view of the black-and-white near-infrared image;

[0118] Step S205, finding the parallax position of the black-and-white near-infrared image with the minimum parallax value, which is the parallax of the current pixel, to generate an initial parallax image of the black-and-white near-infrared image; as shown in the following formula:

[0119] Index

[0120] In the above formula, pmax is the disparity search range, and the minimum disparity value is found within the disparity range [0pmax];

[0121] Step S206, calculating the confidence map of the initial disparity image of the black and white near-infrared image, including:

[0122] (1) Obtain the minimum disparity value SIR min i,j :

[0123]

[0124] (6) Calculate the mean difference EIR of the current pixel in the left view of the black and white near-infrared image i,j :

[0125]

[0126] In the above formula, The mean value of the 5x5 area of ​​the near-infrared image of the current pixel in black and white;

[0127] (7) The confidence CIR of the current pixel of the black-and-white near-infrared image is obtained based on the minimum disparity value and the average difference i,j :

[0128]

[0129] Among them, CIR i,j The larger the value, the more reliable the parallax value of the black and white near-infrared image.

[0130] Step S300, fusing the generated disparity image and the confidence map into a final disparity image;

[0131] Specifically, they include:

[0132] Step S301, filter the current pixel of the confidence map of the color visible light image and the black and white near infrared image respectively, the formula is as follows:

[0133]

[0134]

[0135] In the above formula, flt is a 5x5 constant filter:

[0136] 1 / 32 1 / 32 1 / 32 1 / 32 1 / 32

[0137] 1 / 32 1 / 32 1 / 16 1 / 32 1 / 32

[0138] 1 / 32 1 / 16 1 / 8 1 / 16 1 / 32

[0139] 1 / 32 1 / 32 1 / 16 1 / 32 1 / 32

[0140] 1 / 32 1 / 32 1 / 32 1 / 32 1 / 32

[0141] Step S302: compare the confidence map of the colored visible light image and the filtered black and white near infrared image pixel by pixel, and select the parallax value with the larger confidence as the final parallax image Pout after fusion. i,j , the formula is as follows:

[0142]

[0143] Step S400: converting the final disparity image into a depth map.

[0144] The final disparity image is converted into a depth map by the following formula:

[0145] D i,j =fx*dx / Pout i,j

[0146] In the above formula, fx is the focal length of the lens, and dx is the distance between the optical centers of the left and right cameras.

[0147] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A depth map extraction method based on RGBIR binocular camera, characterized in that: Includes steps: Step S100, obtaining two left and right original viewpoint images of the same scene, and interpolating the two left and right original viewpoint images into one color visible light image and the other black and white near infrared image; Step S200, generating corresponding initial disparity images and confidence maps according to the color visible light image and the black-and-white near-infrared image respectively; Step S300, fusing the generated disparity image and the confidence map into a final disparity image; Step S400, converting the final disparity image into a depth map; In step S200, a corresponding confidence map is generated by using a stereo matching method, including the steps of: Step S201, calculate the sum of the differences between the pixel block in the 5x5 area of ​​the color visible light image centered on the current pixel of the left view and the right view; the formula is as follows: SRGB i,j,p =SR i,j,p +SG i,j,p +SB i,j,p In the above formula, SR i,j,p is the disparity value of the red component at coordinates i, j and disparity p, SG i,j,p is the green component disparity value, SB i,j,p is the blue component disparity value, SRGB i,j,p is the disparity value of visible light color, RL is the left view of the red component, RR is the right view of the red component, GL is the left view of the green component, GR is the right view of the green component, BL is the left view of the blue component, BR is the right view of the blue component, m is the horizontal offset relative to the current pixel coordinate position, and n is the vertical offset relative to the current pixel coordinate position; Step S203, calculating a confidence map of the initial disparity image of the color visible light image, includes: Get the minimum disparity value; Calculate the average difference ERGB of the current pixel in the left view of the color visible light image i,j ; In the above formula, is the 5x5 area mean of the red component of the current pixel, is the mean of the green component, blue component mean; The confidence CRGB of the current pixel of the color visible light image is obtained according to the minimum disparity value and the average difference i,j ; In the above formula, CRGB i,j The larger the value, the more reliable the disparity value of the color visible light image; Step S204, calculating the difference between the pixel block in the 5x5 area of ​​the black and white near-infrared image centered on the current pixel of the left view and the pixel block in the right view, the formula is as follows: In the above formula, SIR i,j,p is the disparity value of the black-and-white near-infrared image with coordinates i, j and disparity p, I RL is the left view of the black-and-white near-infrared image, and IRR is the right view of the black-and-white near-infrared image; Step S206, calculating the confidence map of the initial disparity image of the black and white near-infrared image, including: (1) Obtain the minimum disparity value SIRmin i,j : (2) Calculate the mean difference EIR of the current pixel in the left view of the black and white near-infrared image i,j : In the above formula, The mean value of the 5x5 area of ​​the near-infrared image of the current pixel in black and white; (3) The confidence CIR of the current pixel of the black-and-white near-infrared image is obtained based on the minimum disparity value and the average difference i,j : Among them, CIR i,j The larger the value, the more reliable the parallax value of the black and white near-infrared image.

2. The depth map extraction method based on RGBIR binocular camera according to claim 1, characterized in that: In the step S200, the corresponding initial disparity image is generated by using a stereo matching method, including the following steps: Step S202, finding the disparity position of the color visible light image with the smallest disparity value, which is the disparity of the current pixel, to generate the initial disparity image of the color visible light image; as shown in the following formula: In the above formula, Pmax is the disparity search range, and the minimum disparity value is found within the disparity range [0pmax]; Step S205, finding the parallax position of the black-and-white near-infrared image with the minimum parallax value, which is the parallax of the current pixel, to generate an initial parallax image of the black-and-white near-infrared image; as shown in the following formula: In the above formula, pmax is the disparity search range, and the minimum disparity value is found within the disparity range [0pmax].

3. The depth map extraction method based on RGBIR binocular camera according to claim 1, characterized in that: In the step S100, the two original viewpoint images on the left and right are interpolated into a color visible light image and a black and white near infrared image, using an RGBIR image demosaicing algorithm, including the steps of: Step S101, judging the color of the current pixel at the coordinate position according to the coordinate position of the current pixel; the color of the current pixel includes red R, green G, blue B, infrared IR, setting the RGBIR4X4 color array, and the first pixel in the upper left corner is blue B; According to the horizontal and vertical coordinate positions i and j of the current pixel, the color is marked as follows: In the formula, Cfa ij is the color mark of coordinate position i,j; Step S102: taking the current pixel P ij As the center, take the current pixel P ij The horizontal and vertical gradient values ​​of the pixels in the 5x5 area are calculated respectively to determine the edge direction of the current pixel; Step S103, calculating the green G pixel value according to the results of step S101 and step S102, and if the current pixel is a green pixel, outputting the original value; if the current pixel is not green, interpolating the value using adjacent pixels according to the edge direction; Step S104, calculating the red R and blue B pixel values ​​according to the results of step S101 and step S103, and combining them with the green G pixel value in step S103, outputting a color visible light image; Step S105 , calculating the infrared IR pixel value according to the results of step S101 and step S103 , and outputting a black and white near infrared image.

4. The depth map extraction method based on RGBIR binocular camera according to claim 1, characterized in that: In step S300, the generated disparity image and the confidence map are fused into a final disparity image, including: Step S301, filter the current pixel of the confidence map of the color visible light image and the black and white near infrared image respectively, the formula is as follows: In the above formula, flt is a 5x5 constant filter; Step S302: compare the confidence map of the colored visible light image and the filtered black and white near infrared image pixel by pixel, and select the parallax value with the larger confidence as the final parallax image Pout after fusion. i,j , the formula is as follows:

5. The depth map extraction method based on RGBIR binocular camera according to claim 1, characterized in that: In step S400, the final disparity image is converted into a depth map by the following formula: Di,j=fx*dx / Pouti,j In the above formula, fx is the focal length of the lens, and dx is the distance between the optical centers of the left and right cameras.

Images