Image processing method and device and electronic equipment
By switching between RGB and IR images under different brightness conditions, the problem of overexposure of IR cameras in bright scenes is solved, achieving higher accuracy in depth information acquisition, especially in preserving more detailed features under high brightness conditions.
Patent Information
- Application Number
- CN202411639300.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2026-05-15
AI Technical Summary
In high-brightness shooting scenarios, IR cameras are prone to overexposure in object depth information acquisition schemes based on binocular stereo vision, resulting in the loss of object details in the image and affecting the accuracy of depth information acquisition.
The system uses a first camera and a second camera to capture RGB and IR images under different brightness conditions, and calculates depth information for each. RGB images are used when the ambient brightness is greater than or equal to a threshold, while IR images are used when the brightness is less than the threshold. This leverages the high dynamic range of RGB images to retain more detailed features.
It improves the accuracy of acquiring object depth information under different brightness conditions. In particular, when the brightness is high, it retains more detailed features through RGB images, thereby improving the accuracy of depth information acquisition.
Smart Images
Figure CN122053808A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus and electronic device. Background Technology
[0002] Binocular stereo vision is an important form of machine vision and a crucial technique for acquiring three-dimensional geometric information of objects. Depth information refers to the distances between various points on the object and the image acquisition device, representing the object's proximity to the image acquisition device and constituting important three-dimensional information about the object.
[0003] Current methods for acquiring object depth information based on binocular stereo vision typically involve using two infrared (IR) cameras to capture two IR images of the object being measured. The depth information of the object is then obtained by calculating the positional deviation between corresponding points in the two IR images.
[0004] However, IR cameras may suffer from overexposure in bright shooting scenarios. Overexposure results in a significant loss of detail in the captured image of the object being measured, making it impossible to obtain depth information in many detailed areas of the object, thus leading to lower accuracy in obtaining the object's depth information. Summary of the Invention
[0005] To overcome the problems existing in related technologies, this application provides an image processing method, apparatus, and electronic device.
[0006] According to a first aspect of the embodiments of this application, an image processing method is provided, applied to an image processing system, the image processing system including a first camera and a second camera, both the first camera and the second camera being used to capture RGB images and IR images; the method includes:
[0007] Obtain the ambient brightness of the shooting scene;
[0008] When the ambient brightness is greater than or equal to a brightness threshold, acquire a first RGB image captured by the first camera and a second RGB image captured by the second camera.
[0009] Based on the first RGB image and the second RGB image, calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene;
[0010] When the ambient brightness is less than the brightness threshold, acquire a first IR image captured by the first camera and a second IR image captured by the second camera.
[0011] Based on the first IR image and the second IR image, calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene.
[0012] According to a second aspect of the present application, an image processing apparatus is provided, applied to an image processing system, the image processing system including a first camera and a second camera, both the first camera and the second camera being used to capture RGB images and IR images; the apparatus includes:
[0013] The acquisition module is used to acquire the ambient brightness of the shooting scene;
[0014] The acquisition module is further configured to acquire a first RGB image captured by the first camera and a second RGB image captured by the second camera when the ambient brightness is greater than or equal to a brightness threshold.
[0015] The calculation module is used to calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first RGB image and the second RGB image;
[0016] The acquisition module is further configured to acquire a first IR image captured by the first camera and a second IR image captured by the second camera when the ambient brightness is less than the brightness threshold.
[0017] The calculation module is further configured to calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first IR image and the second IR image.
[0018] According to a third aspect of the embodiments of this application, an electronic device is provided, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the image processing method as described in the first aspect.
[0019] According to a fourth aspect of the embodiments of this application, an embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image processing method as described in the first aspect.
[0020] According to a fifth aspect of the embodiments of this application, an embodiment of this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the image processing method as described in the first aspect.
[0021] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:
[0022] By acquiring the ambient brightness of the shooting scene, when the ambient brightness is less than a brightness threshold, depth information of the overlapping shooting area of the first and second cameras in the shooting scene is calculated based on the first IR image captured by the first camera and the second IR image captured by the second camera. Furthermore, when the ambient brightness is greater than or equal to the brightness threshold, depth information of the overlapping shooting area of the first and second cameras in the shooting scene is calculated based on the first RGB image captured by the first camera and the second RGB image captured by the second camera. In this technical solution, since the RGB image is a High Dynamic Range Imaging (HDR) image, it has a larger exposure dynamic range. Therefore, when the ambient brightness is greater than or equal to the brightness threshold, the RGB images captured by the first and second cameras can retain more detailed features of the shooting scene. Therefore, using the RGB images captured by the first and second cameras instead of the IR image for depth information calculation can effectively extract depth information from more detailed locations in the shooting scene, improving the accuracy of depth information acquisition.
[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.
[0025] Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application;
[0026] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application;
[0027] Figure 3 This is a schematic diagram illustrating the calculation principle of depth information provided in an embodiment of this application;
[0028] Figure 4 This is a schematic diagram illustrating the principle of an image processing method provided in an embodiment of this application;
[0029] Figure 5 This is a block diagram of an image processing apparatus provided in an embodiment of this application;
[0030] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0032] Please refer to Figure 1 This diagram illustrates a block diagram of an image processing system provided in an embodiment of this application. The image processing method provided in this embodiment can be applied to this image processing system. Figure 1 As shown, the image processing system 10 may include a first camera 101 and a second camera 102. Both the first camera 101 and the second camera 102 are used to capture RGB images and IR images. The first camera 101 and the second camera 102 are arranged opposite to each other.
[0033] Optionally, both the first camera 101 and the second camera 102 can be RGB_IR cameras. The RGB_IR cameras can selectively output RGB images, or selectively output IR images, or simultaneously output RGB and IR images. For example, the RGB_IR cameras can simultaneously output RGB and IR images in ABAB mode. The first camera can be referred to as the master RGB_IR camera (RGB_IR_main); the second camera can be a slave RGB_IR camera (RGB_IR_sub). Specifically, for example, the RGB_IR cameras can simultaneously output m RGB and IR images per second in ABAB mode. The value of m can be 30, 35, or 40, etc.
[0034] Optionally, the image processing system 10 may also include a display device 103. The display device 103 can be used to display images captured by the first camera 101 and the second camera 102. Optionally, the display device 103 can be a light-emitting diode (LED) display. For example, the display device 103 can be an IR_LED display, which can be used to display IR images.
[0035] In addition, the image processing system 10 may also include a processing device. The processing device can be used to execute the image processing method provided in the embodiments of this application. Optionally, the processing device may also be connected to the first camera 101 and the second camera 102 to acquire RGB image data and IR image data captured by the first camera 101 and the second camera 102, generate an RGB image based on the RGB image data, and generate an IR image based on the IR image data.
[0036] It should be noted that in some embodiments, the image processing system 10 can be applied to an electronic device. The electronic device can be a mobile phone, tablet computer, in-vehicle device, wearable device, virtual reality (VR) terminal device, augmented reality (AR) terminal device, etc., and the embodiments of this application are not limited to this.
[0037] Please refer to Figure 2 The diagram illustrates a flowchart of an image processing method provided in an embodiment of this application. The image processing method can be applied to the image processing system provided in this embodiment. Optionally, the image processing method can be applied to... Figure 1 The image processing system shown is illustrated. Furthermore, the explanation will take the example of an image processing method being executed by a processing device. Figure 2 As shown, the image processing methods include:
[0038] Step 201: Obtain the ambient brightness of the shooting scene.
[0039] The ambient brightness of the shooting scene refers to the brightness of the ambient light in the shooting scene. The ambient brightness of the shooting scene affects the exposure levels of the first camera and the second camera. In an optional implementation, the image processing system may further include an ambient light sensor. The ambient light sensor is used to detect the ambient brightness of the shooting scene. Based on this, the process by which the processing device acquires the ambient brightness of the shooting scene can be: the processing device acquires the detected ambient brightness of the shooting scene through the ambient light sensor.
[0040] In another alternative implementation, the processing device may also utilize a target camera to acquire the ambient brightness of the shooting scene. The target camera may be a first camera or a second camera. The process by which the processing device acquires the ambient brightness of the shooting scene may include steps 2011 to 2013.
[0041] In step 2011, the target camera is controlled to capture a target RGB image of the scene.
[0042] Optionally, the processing device can output a shooting command to the target camera, so that the target camera responds to the shooting command, shoots the shooting scene, and obtains a target RGB image of the shooting scene.
[0043] In step 2012, the image brightness of the target RGB image, as well as the brightness gain value and exposure time when the target camera captures the target RGB image, are obtained.
[0044] Optionally, the processing device can control the shooting parameters of the target camera. Shooting parameters include brightness gain, exposure time, aperture, etc. The processing device can directly obtain the brightness gain and exposure time used by the target camera when capturing the target RGB image. Further optionally, the processing device can use a standard brightness calculation formula to calculate the image brightness of the target RGB image based on the pixel values of the RGB image, thereby obtaining the image brightness of the target RGB image.
[0045] In step 2013, the ambient brightness is calculated based on the image brightness, brightness gain value, and exposure time.
[0046] In some embodiments of this application, the processing device may employ a first formula to calculate the ambient brightness of the shooting scene based on the image brightness of the target RGB image, the brightness gain value and exposure time when the target camera captures the target RGB image. The target formula is: Len = Lavg / t / g. In this formula, Len represents the ambient brightness of the shooting scene; Lavg represents the image brightness of the target RGB image; t represents the exposure time; and g represents the brightness gain.
[0047] In this embodiment, after acquiring the ambient brightness of the shooting scene, the processing device can determine the relationship between the ambient brightness and a brightness threshold to determine whether the current shooting scene is prone to overexposure. If the ambient brightness is greater than or equal to the brightness threshold, indicating that the current shooting scene is prone to overexposure, the processing device can execute step 202. If the ambient brightness is less than the brightness threshold, indicating that the current shooting scene is not prone to overexposure, the processing device can execute step 204.
[0048] Optionally, the processing device may store a brightness threshold (Lena) within its own storage space to read the brightness threshold and determine the relationship between the ambient brightness and the brightness threshold. The brightness threshold is also known as the light-dark boundary value. Alternatively, the brightness threshold may be stored on a third-party device, from which the processing device can obtain the brightness threshold.
[0049] Step 202: When the ambient brightness is greater than or equal to the brightness threshold, acquire the first RGB image captured by the first camera and the second RGB image captured by the second camera.
[0050] In this embodiment, when the ambient brightness is greater than or equal to a brightness threshold, it indicates that the current shooting scene is prone to overexposure. The processing device can acquire a first RGB image captured by a first camera and a second RGB image captured by a second camera.
[0051] Step 203: Calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first RGB image and the second RGB image.
[0052] The depth information may include the depth values of each point within the overlapping shooting area of the first and second cameras in the shooting scene. Alternatively, when the image processing method provided in this embodiment is applied to read the depth information of a target object in the shooting scene, the depth information may also include the depth values of each point of the target object within the overlapping shooting area of the first and second cameras in the shooting scene.
[0053] In one alternative implementation, the process by which the processing device calculates the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first RGB image and the second RGB image may include the following steps 2031 to 2033.
[0054] In step 2031, stereo matching processing is performed on the first RGB image and the second RGB image to obtain multiple matching pixel pairs between the first RGB image and the second RGB image.
[0055] The binocular stereo matching process is used to determine corresponding points between the first RGB image and the second RGB image, serving as matching pixel pairs. These corresponding points refer to the projection points (i.e., speed-limiting points) of the same point in the three-dimensional space of the shooting scene onto the imaging surfaces of the first and second cameras. Optionally, the binocular stereo matching process can determine multiple matching pixel pairs between the first and second RGB images based on image features. These image features may include at least one of the following: edge features, texture features, brightness variation features, etc.
[0056] In one optional implementation, the process by which the processing device performs binocular stereo matching on the first RGB image and the second RGB image to obtain multiple matching pixel pairs between the first RGB image and the second RGB image may include:
[0057] The processing device extracts image feature regions from the first RGB image and the second RGB image, respectively. These image feature regions may include at least one of the following: edge regions, texture regions, corner points, etc. The processing device employs a feature matching algorithm to establish a correspondence between the image feature regions of the first RGB image and the second RGB image, determining the pixel correspondence at corresponding positions within the two image feature regions in the first and second RGB images, thus obtaining multiple matching pixel pairs. Optionally, the feature matching algorithm may be a Scale-Invariant Feature Transform (SIFT) algorithm or an ORB algorithm, etc. ORB stands for Oriented Fast and Rotated BRIEF, which is a fast feature point extraction and description algorithm.
[0058] Optionally, the processing device can further perform secondary filtering on multiple matching pixel pairs in the first RGB image and the second RGB image using a matching optimization algorithm to eliminate erroneous matching pixel pairs. The matching optimization algorithm can be epipolar constraint, least squares fitting algorithm, or random sample consensus algorithm, etc. In this way, by performing secondary filtering on multiple matching pixel pairs in the first RGB image and the second RGB image, the accuracy of multiple matching pixel pairs is further ensured, thereby effectively guaranteeing the accuracy of subsequent depth information generation.
[0059] In step 2032, the focal lengths of the first camera and the second camera, as well as the baseline distance between the first camera and the second camera, are obtained.
[0060] The baseline distance refers to the distance between the projection centers of the first camera and the second camera. The processing device can store the baseline distances of the first and second cameras for later retrieval. Furthermore, the focal lengths of the first and second cameras are typically identical. The processing device can obtain the focal length of the first camera when capturing the first RGB image, or the focal length of the second camera when capturing the second RGB image.
[0061] In step 2033, for each matching pixel pair, the disparity of the matching pixel pair is calculated based on the pixel position of each pixel in the matching pixel pair. Based on the disparity, focal length, and baseline distance, the depth corresponding to the matching pixel pair is calculated to obtain the depth corresponding to each matching pixel pair, and the depth information of the shooting area where the first camera and the second camera overlap in the shooting scene is generated.
[0062] Parallax refers to the pixel distance between the projection points of the same 3D point on the imaging planes of the left and right cameras. The parallax of a matching pixel pair can be understood as the pixel distance between the projection points of the same point on the imaging planes of the first and second cameras in the 3D space of the shooting scene.
[0063] Please refer to Figure 3 This illustrates a schematic diagram of the principle of binocular parallax provided in an embodiment of this application. For example... Figure 3 As shown, the first and second cameras are set at... Figure 3 Take the left and right sides as an example.
[0064] The projection center O of the first camera L The projection center O of the second camera R The connection distance between them is the baseline distance b. Any point P in the three-dimensional space of the shooting scene is the imaging point (i.e., pixel) of the first camera. L The imaging point of the second camera is P. R According to the principle of rectilinear propagation of light, point P in three-dimensional space is the projection center O of the first camera. L With imaging point P L The intersection of the connecting lines is also the projection center O of the second camera. R With imaging point P R The intersection of the lines connecting point P. Furthermore, the length of the imaging planes of the first and second cameras is L. The image point P of the first camera at point P. L The distance to the left side of its imaging plane is X L , that is, X L For imaging point P L The x-coordinate in the image coordinate system, X L It is usually a positive value. The image point P of the second camera at point P. R The distance to the left side of its imaging plane is X R , that is, X R For imaging point P R The x-coordinate in the image coordinate system, X R Typically, the value is negative. Thus, according to the definition of parallax of point P in the first and second cameras, the imaging point P... L and imaging point P R The disparity d satisfies: d = |X L -X R |
[0065] And the imaging point P L and imaging point P R Distance P between L P R Satisfy: P L P R =b-(X)L -L / 2)-(L / 2-X R Based on this, we can derive the second formula:
[0066] In the second formula, Z represents the distance from point P to the projection planes of the first and second cameras, which is the depth value of point P. f represents the focal length of the first and second cameras. b represents the baseline distance. X L This represents the image point P of the first camera at point P. L The distance to the left edge of its imaging plane. X R This represents the image point P of the second camera at point P. R The distance to the left side of its imaging plane.
[0067] In some embodiments of this application, the pixel position refers to the position coordinates of the pixel in the image coordinate system of its image. For each matching pixel pair, the processing device can determine, based on the pixel position of each pixel in the matching pixel pair, a first distance from the pixel in the first RGB image to the left edge of its imaging plane, and a second distance from the pixel in the second RGB image to the left edge of its imaging plane. The absolute value of the difference between the first distance and the second distance is calculated to obtain the disparity of the matching pixel pair. Here, the first distance is the abscissa of the pixel in the first RGB image, and the second distance is the abscissa of the pixel in the second RGB image.
[0068] The processing device uses a third formula to calculate the depth corresponding to the matched pixel pair based on the parallax, focal length, and baseline distance. This yields the depth information for the overlapping shooting area of the first and second cameras in the shooting scene. The third formula satisfies: Z = b × f ÷ d. In this formula, Z represents the depth corresponding to the matched pixel pair. b represents the baseline distance. f represents the focal length of the first and second cameras. d represents the parallax of the matched pixel pair. The depth information includes the depth value corresponding to each matched pixel pair.
[0069] In one alternative approach, the process by which the processing device calculates the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first RGB image and the second RGB image may further include:
[0070] Before performing stereo matching processing on the first RGB image and the second RGB image in step 2031, the processing device can also perform distortion correction processing on the first RGB image and the second RGB image to obtain the first RGB image and the second RGB image after distortion correction.
[0071] Optionally, the processing device can acquire calibration parameters of the first camera and the second camera. Based on the calibration parameters of the first camera, the distortion coefficients of the first camera are determined, and based on the calibration parameters of the second camera, the distortion coefficients of the second camera are determined. The distortion coefficients include radial distortion coefficients (e.g., k1, k2, k3) and tangential distortion coefficients (e.g., p1, p2). These distortion coefficients are used to describe the degree and direction of camera distortion.
[0072] For the first RGB image, the processing device uses the fourth formula to calculate the distorted coordinates of each pixel after displacement due to distortion, based on the distortion coefficients of the first RGB image and the original coordinates of each pixel. The fourth formula satisfies the following:
[0073] x _distort =x(1+k1r) 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 );
[0074] y _distort =y(1+k1r) 2 +k2r 4 +k3r 6 )+p1(r 2 +2y 2 )+2p2xy.
[0075] Where, x _distort This represents the x-coordinate in the distorted coordinates after pixel offset, i.e., the x-coordinate of the pixel after correction in the first RGB image. _distort represents the ordinate in the distorted coordinates after pixel offset, i.e., the ordinate of the pixel after correction in the first RGB image. k1, k2, and k3 represent the radial distortion coefficients. p1 and p2 represent the tangential distortion coefficients. x represents the x-coordinate in the original pixel coordinates. y represents the ordinate in the original pixel coordinates. r satisfies: r = SQRT(x 2 +y 2 ).
[0076] The processing device performs image remapping processing based on the distortion coordinates of each pixel in the first RGB image to obtain a distortion-corrected first RGB image and a second RGB image. The image remapping processing may include bilinear interpolation or bicubic interpolation, etc.
[0077] In this way, by performing distortion correction processing on the first and second RGB images, image distortion caused by factors such as the camera manufacturing and installation process can be eliminated. The distortion-corrected image can more accurately reflect the shape and position of objects in three-dimensional space, thereby improving the accuracy of depth information calculation.
[0078] Further optionally, the process by which the processing device calculates the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first RGB image and the second RGB image may further include:
[0079] Before performing stereo matching processing on the first RGB image and the second RGB image in step 2031, the processing device can also perform color correction processing on the first RGB image and the second RGB image to obtain the first RGB image and the second RGB image after distortion correction.
[0080] Optionally, color correction processing can be achieved using histogram equalization algorithms, histogram specification algorithms, gamma correction algorithms, white balance adjustment algorithms, or color correction algorithms based on color matrices. This is because the first RGB image and the second RGB image may differ in color due to lighting conditions, camera parameter differences, etc. This color inconsistency can affect subsequent parallax calculations, thus impacting the accuracy of depth information. Therefore, by performing color correction processing on the first IR image and the second IR image, the color differences between them can be eliminated, effectively ensuring the quality of the input data required for subsequent depth information calculations and improving the accuracy of subsequent depth information calculations.
[0081] Step 204: When the ambient brightness is less than the brightness threshold, acquire the first IR image captured by the first camera and the second IR image captured by the second camera.
[0082] In this embodiment, when the ambient brightness is less than a brightness threshold, it indicates that the current shooting scene is unlikely to trigger overexposure. The processing device can acquire a first IR image captured by the first camera and a second IR image captured by the second camera.
[0083] Step 205: Calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first IR image and the second IR image.
[0084] In one alternative implementation, the process by which the processing device calculates the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first IR image and the second IR image may include the following steps 2051 to 2053.
[0085] In step 2051, stereo matching processing is performed on the first IR image and the second IR image to obtain multiple matching pixel pairs between the first IR image and the second IR image.
[0086] The binocular stereo matching process is used to determine corresponding points between the first IR image and the second IR image, as matching pixel pairs. Optionally, the binocular stereo matching process can determine multiple matching pixel pairs between the first IR image and the second IR image based on image features in the first IR image and the second IR image. These image features may include at least one of the following: edge features, texture features, brightness variation features, etc.
[0087] In one optional implementation, the process by which the processing device performs binocular stereo matching on the first IR image and the second IR image to obtain multiple matching pixel pairs of the first IR image and the second IR image may include:
[0088] The processing device extracts image feature regions from the first IR image and the second IR image, respectively. These image feature regions may include at least one of the following: edge regions, texture regions, corner points, etc. The processing device employs a feature matching algorithm to establish a correspondence between the image feature regions of the first IR image and the second IR image, determining the pixel correspondence at corresponding positions within the two corresponding image feature regions in the first and second IR images, thus obtaining multiple matching pixel pairs. Optionally, the feature matching algorithm may be the SIFT algorithm or the ORB algorithm, etc.
[0089] Optionally, the processing device can further perform secondary filtering on multiple matching pixel pairs in the first IR image and the second IR image using a matching optimization algorithm to eliminate erroneous matching pixel pairs. The matching optimization algorithm can be epipolar constraint, least squares fitting algorithm, or random sample consensus algorithm, etc. In this way, by performing secondary filtering on multiple matching pixel pairs in the first IR image and the second IR image, the accuracy of multiple matching pixel pairs is further ensured, thereby effectively guaranteeing the accuracy of subsequent depth information generation.
[0090] In step 2052, the focal lengths of the first camera and the second camera, as well as the baseline distance between the first camera and the second camera, are obtained.
[0091] The processing device can store the baseline distance between the first camera and the second camera for retrieval. Furthermore, the processing device can obtain the focal length of the first camera when capturing the first IR image, or the focal length of the second camera when capturing the second IR image.
[0092] In step 2053, for each matching pixel pair, the disparity of the matching pixel pair is calculated based on the pixel position of each pixel in the matching pixel pair. Based on the disparity, focal length, and baseline distance, the depth corresponding to the matching pixel pair is calculated to obtain the depth corresponding to each matching pixel pair, and the depth information of the shooting area where the first camera and the second camera overlap in the shooting scene is generated.
[0093] In some embodiments of this application, the pixel position refers to the position coordinates of the pixel in the image coordinate system of its image. For each matching pixel pair, the processing device can determine, based on the pixel position of each pixel in the matching pixel pair, a first distance from the pixel in the first IR image to the left edge of its imaging plane, and a second distance from the pixel in the second IR image to the left edge of its imaging plane. The absolute value of the difference between the first distance and the second distance is calculated to obtain the disparity of the matching pixel pair. Here, the first distance is the abscissa of the pixel in the first IR image, and the second distance is the abscissa of the pixel in the second IR image.
[0094] The processing device uses a third formula to calculate the depth corresponding to the matched pixel pair based on the parallax, focal length, and baseline distance. This yields the depth information for the overlapping shooting area of the first and second cameras in the shooting scene. The third formula satisfies: Z = b × f ÷ d. In this formula, Z represents the depth corresponding to the matched pixel pair. b represents the baseline distance. f represents the focal length of the first and second cameras. d represents the parallax of the matched pixel pair. The depth information includes the depth value corresponding to each matched pixel pair.
[0095] In one alternative approach, the process by which the processing device calculates depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first IR image and the second IR image may further include:
[0096] Before performing stereo matching processing on the first IR image and the second IR image in step 2051, the processing device can also perform distortion correction processing on the first IR image and the second IR image to obtain the first IR image and the second IR image after distortion correction.
[0097] Optionally, the processing device can acquire the calibration parameters of the first camera and the second camera. Based on the calibration parameters of the first camera, the distortion coefficients of the first camera are determined, and based on the calibration parameters of the second camera, the distortion coefficients of the second camera are determined. The distortion coefficients include radial distortion coefficients (e.g., k1, k2, k3) and tangential distortion coefficients (e.g., p1, p2). These distortion coefficients describe the degree and direction of camera distortion. For the first IR image, the processing device uses a fourth formula to calculate the distortion coordinates of each pixel after offset due to distortion, based on the distortion coefficients of the first IR image and the original coordinates of each pixel. Based on the distortion coordinates of each pixel in the first IR image, the processing device performs image remapping processing to obtain the distortion-corrected first IR image and the second IR image. The image remapping processing may include bilinear interpolation or bicubic interpolation, etc.
[0098] In this way, by performing distortion correction processing on the first and second IR images, image distortion caused by factors such as the camera manufacturing and installation process can be eliminated. The distortion-corrected image can more accurately reflect the shape and position of objects in three-dimensional space, thereby improving the accuracy of depth information calculation.
[0099] In some embodiments of this application, the processing device may also directly output the RGB image captured by the target camera for display on a display device. Optionally, the image processing method further includes: in response to an image capture command, controlling the target camera to capture an image and outputting the RGB image captured by the target camera. Correspondingly, the process of acquiring the ambient brightness of the shooting scene may include: in response to a depth information acquisition command, acquiring the ambient brightness of the shooting scene.
[0100] The image capture command is used by the target camera to directly output the RGB image it captures. Optionally, the image capture command can be triggered by user input to the capture control. This input can be in the form of a click, long press, swipe, or voice input to the capture control. For example, if a user wants to capture an image, they can click the "camera" control, causing the processing device to generate an image capture command in response to the click. In response to the image capture command, the target camera is controlled to capture an image, outputting the RGB image captured by the target camera for image preview, image capture, etc.
[0101] The depth information acquisition command is used by the processing device to generate depth information of the overlapping shooting area of the first and second cameras in the shooting scene. This depth information can be used in various scenarios such as user gesture recognition and facial unlocking. For example, suppose the image processing system is applied to a mobile phone. When a user picks up the phone and faces the screen, the phone can generate and respond to the depth information acquisition command to acquire the depth information of a face in the shooting scene, and then perform facial recognition based on the depth information.
[0102] In this embodiment, by acquiring the ambient brightness of the shooting scene, when the ambient brightness is less than a brightness threshold, depth information of the overlapping shooting area of the first and second cameras in the shooting scene is calculated based on the first IR image captured by the first camera and the second IR image captured by the second camera. Furthermore, when the ambient brightness is greater than or equal to the brightness threshold, depth information of the overlapping shooting area of the first and second cameras in the shooting scene is calculated based on the first RGB image captured by the first camera and the second RGB image captured by the second camera. In this technical solution, since the RGB image is an HDR image, it has a wider dynamic range. Therefore, when the ambient brightness is greater than or equal to the brightness threshold, the RGB images captured by the first and second cameras can retain more detailed features of the shooting scene. Thus, using the RGB images captured by the first and second cameras instead of the IR images for depth information calculation can effectively extract depth information from more detailed locations in the shooting scene, improving the accuracy of depth information acquisition.
[0103] For ease of understanding, the following will be used as... Figure 4 The image processing method provided in this application will be further described using the example shown. Assume the first camera is a primary RGB_IR camera and the second camera is a secondary RGB_IR camera. For example... Figure 4 As shown, the first and second cameras can output RGB and IR images in an ABAB format. Here, A represents the RGB image and B represents the IR image. The image processing device has a normal shooting mode, a night and low-light mode, and a high-light mode. In normal shooting mode, the main RGB image captured by the first camera can be directly acquired to output a wide dynamic range RGB image for image preview, video recording, and photo capture. In high-light mode, when the ambient brightness of the shooting scene is greater than or equal to a brightness threshold, depth information can be output based on the main RGB image captured by the first camera and the secondary RGB image captured by the second camera, for use in 3D applications such as gesture recognition. In night and low-light mode, when the ambient brightness of the shooting scene is less than a brightness threshold, depth information can be output based on the main IR image captured by the first camera and the secondary IR image captured by the second camera, for use in 3D applications such as gesture recognition. This depth information is also called 3D depth information.
[0104] In summary, the image processing method provided in this application obtains the ambient brightness of the shooting scene. When the ambient brightness is less than a brightness threshold, it calculates the depth information of the overlapping shooting area of the first and second cameras in the shooting scene based on the first IR image captured by the first camera and the second IR image captured by the second camera. Furthermore, when the ambient brightness is greater than or equal to the brightness threshold, it calculates the depth information of the overlapping shooting area of the first and second cameras in the shooting scene based on the first RGB image captured by the first camera and the second RGB image captured by the second camera. In this technical solution, since the RGB image is an HDR image, it has a wider dynamic range. Therefore, when the ambient brightness is greater than or equal to the brightness threshold, the RGB images captured by the first and second cameras can retain more detailed features of the shooting scene. Thus, using the RGB images captured by the first and second cameras instead of the IR images for depth information calculation can effectively extract depth information from more detailed locations in the shooting scene, improving the accuracy of depth information acquisition.
[0105] Please refer to Figure 5 This diagram illustrates a block diagram of an image processing apparatus provided in an embodiment of this application. The image processing apparatus is applied to an image processing system. Optionally, the image processing apparatus can be applied to... Figure 1 The image processing system shown. (As shown) Figure 5 As shown, the image processing device 500 includes an acquisition module 501 and a calculation module 502.
[0106] The acquisition module 501 is used to acquire the ambient brightness of the shooting scene;
[0107] The acquisition module 501 is further configured to acquire a first RGB image captured by the first camera and a second RGB image captured by the second camera when the ambient brightness is greater than or equal to a brightness threshold.
[0108] The calculation module 502 is used to calculate the depth information of the shooting area where the first camera and the second camera overlap in the shooting scene based on the first RGB image and the second RGB image;
[0109] The acquisition module 501 is further configured to acquire a first IR image captured by the first camera and a second IR image captured by the second camera when the ambient brightness is less than a brightness threshold.
[0110] The calculation module 502 is also used to calculate the depth information of the shooting area where the first camera and the second camera overlap in the shooting scene based on the first IR image and the second IR image.
[0111] Optionally, the computing module 502 is also used for:
[0112] Perform stereo matching processing on the first RGB image and the second RGB image to obtain multiple matching pixel pairs between the first RGB image and the second RGB image;
[0113] Obtain the focal lengths of the first and second cameras, as well as the baseline distance between the first and second cameras;
[0114] For each matching pixel pair, calculate the disparity of the matching pixel pair based on the pixel position of each pixel in the matching pixel pair.
[0115] Based on parallax, focal length, and baseline distance, the depth corresponding to the matching pixel pair is calculated to obtain the depth corresponding to each matching pixel pair, thereby generating depth information of the shooting area where the first camera and the second camera overlap in the shooting scene.
[0116] Optionally, the calculation module 502 is further configured to: perform distortion correction processing on the first RGB image and the second RGB image to obtain the first RGB image and the second RGB image after distortion correction.
[0117] Optionally, the computing module 502 is also used for:
[0118] Color correction processing is performed on the first RGB image and the second RGB image to obtain the first RGB image and the second RGB image after distortion correction.
[0119] Optionally, the computing module 502 is also used for:
[0120] Binocular stereo matching processing is performed on the first IR image and the second IR image to obtain multiple matching pixel pairs between the first IR image and the second IR image;
[0121] Obtain the focal lengths of the first and second cameras, as well as the baseline distance between the first and second cameras;
[0122] For each matching pixel pair, calculate the disparity of the matching pixel pair based on the pixel position of each pixel in the matching pixel pair.
[0123] Based on parallax, focal length, and baseline distance, the depth corresponding to the matching pixel pair is calculated to obtain the depth corresponding to each matching pixel pair, thereby generating depth information of the shooting area where the first camera and the second camera overlap in the shooting scene.
[0124] Optionally, the acquisition module 501 is also used for:
[0125] Control the target camera to capture a target RGB image of the shooting scene. The target camera can be a first camera or a second camera.
[0126] Acquire the image brightness of the target RGB image, as well as the brightness gain value and exposure time when the target camera captures the target RGB image;
[0127] Calculate the ambient brightness based on the image brightness, brightness gain value, and exposure time.
[0128] Optionally, the image processing device 500 further includes: a control module, configured to control the target camera to capture an image in response to an image capture command, and output an RGB image captured by the target camera, wherein the target camera is a first camera or a second camera;
[0129] The acquisition module 501 is also used to: acquire the ambient brightness of the shooting scene in response to the depth information acquisition command.
[0130] In summary, the image processing apparatus provided in this application acquires the ambient brightness of the shooting scene. When the ambient brightness is less than a brightness threshold, it calculates the depth information of the overlapping shooting area of the first and second cameras in the shooting scene based on the first IR image captured by the first camera and the second IR image captured by the second camera. Furthermore, when the ambient brightness is greater than or equal to the brightness threshold, it calculates the depth information of the overlapping shooting area of the first and second cameras in the shooting scene based on the first RGB image captured by the first camera and the second RGB image captured by the second camera. In this technical solution, since the RGB image is an HDR image, it has a wider dynamic range. Therefore, when the ambient brightness is greater than or equal to the brightness threshold, the RGB images captured by the first and second cameras can retain more detailed features of the shooting scene. Thus, using the RGB images captured by the first and second cameras instead of the IR images for depth information calculation can effectively extract depth information from more detailed locations in the shooting scene, improving the accuracy of depth information acquisition.
[0131] The image processing apparatus provided in this application embodiment can execute the image processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the image processing method.
[0132] This application also provides an electronic device. For example... Figure 6 As shown, the electronic device 600 includes a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the computer program is executed by the processor 601, it implements the various processes of the above-described image processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.
[0133] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described image processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0134] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. An image processing method, characterized in that, The method is applied to an image processing system, which includes a first camera and a second camera, both of which are used to capture RGB and IR images; the method includes: Obtain the ambient brightness of the shooting scene; When the ambient brightness is greater than or equal to a brightness threshold, acquire a first RGB image captured by the first camera and a second RGB image captured by the second camera. Based on the first RGB image and the second RGB image, calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene; When the ambient brightness is less than the brightness threshold, acquire a first IR image captured by the first camera and a second IR image captured by the second camera. Based on the first IR image and the second IR image, calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene.
2. The method according to claim 1, characterized in that, The step of calculating the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first RGB image and the second RGB image includes: Perform stereo matching processing on the first RGB image and the second RGB image to obtain multiple matching pixel pairs between the first RGB image and the second RGB image; Obtain the focal lengths of the first camera and the second camera, as well as the baseline distance between the first camera and the second camera; For each matching pixel pair, the disparity of the matching pixel pair is calculated based on the pixel position of each pixel in the matching pixel pair. Based on the parallax, the focal length, and the baseline distance, the depth corresponding to the matching pixel pair is calculated to obtain the depth corresponding to each matching pixel pair, and the depth information of the shooting area where the first camera and the second camera overlap in the shooting scene is generated.
3. The method according to claim 2, characterized in that, Before performing stereo matching processing on the first RGB image and the second RGB image, the step of calculating the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first RGB image and the second RGB image further includes: Distortion correction processing is performed on the first RGB image and the second RGB image to obtain the first RGB image and the second RGB image after distortion correction.
4. The method according to claim 1, characterized in that, The step of calculating the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first IR image and the second IR image includes: Perform binocular stereo matching processing on the first IR image and the second IR image to obtain multiple matching pixel pairs between the first IR image and the second IR image; Obtain the focal lengths of the first camera and the second camera, as well as the baseline distance between the first camera and the second camera; For each matching pixel pair, the disparity of the matching pixel pair is calculated based on the pixel position of each pixel in the matching pixel pair. Based on the parallax, the focal length, and the baseline distance, the depth corresponding to the matching pixel pair is calculated to obtain the depth corresponding to each matching pixel pair, and the depth information of the shooting area where the first camera and the second camera overlap in the shooting scene is generated.
5. The method according to claim 1, characterized in that, The step of obtaining the ambient brightness of the shooting scene includes: Control the target camera to capture a target RGB image of the shooting scene, wherein the target camera is either the first camera or the second camera; The image brightness of the target RGB image is obtained, as well as the brightness gain value and exposure time when the target camera captures the target RGB image; The ambient brightness is calculated based on the image brightness, the brightness gain value, and the exposure time.
6. The method according to claim 1, characterized in that, The method further includes: In response to an image capture command, the target camera is controlled to capture an image and outputs an RGB image captured by the target camera, wherein the target camera is either the first camera or the second camera; The step of acquiring the ambient brightness of the shooting scene includes: acquiring the ambient brightness of the shooting scene in response to a depth information acquisition command.
7. An image processing apparatus, characterized in that, The device is applied to an image processing system, which includes a first camera and a second camera, both of which are used to capture RGB and IR images; the device includes: The acquisition module is used to acquire the ambient brightness of the shooting scene; The acquisition module is further configured to acquire a first RGB image captured by the first camera and a second RGB image captured by the second camera when the ambient brightness is greater than or equal to a brightness threshold. The calculation module is used to calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first RGB image and the second RGB image; The acquisition module is further configured to acquire a first IR image captured by the first camera and a second IR image captured by the second camera when the ambient brightness is less than the brightness threshold. The calculation module is further configured to calculate the depth information of the overlapping shooting area of the first camera and the second camera in the shooting scene based on the first IR image and the second IR image.
8. The apparatus according to claim 7, characterized in that, The computing module is also used for: Perform stereo matching processing on the first RGB image and the second RGB image to obtain multiple matching pixel pairs between the first RGB image and the second RGB image; Obtain the focal lengths of the first camera and the second camera, as well as the baseline distance between the first camera and the second camera; For each matching pixel pair, the disparity of the matching pixel pair is calculated based on the pixel position of each pixel in the matching pixel pair. Based on the parallax, the focal length, and the baseline distance, the depth corresponding to the matching pixel pair is calculated to obtain the depth corresponding to each matching pixel pair, and the depth information of the shooting area where the first camera and the second camera overlap in the shooting scene is generated.
9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the image processing method as described in any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the image processing method as described in any one of claims 1 to 6.