A real-time binocular RGB-D stitching method based on 3D geometric properties

Abstract

The present invention provides a real-time binocular RGB-D camera image stitching method based on three-dimensional geometric properties. By accurately estimating the spatial relative relationship between cameras, efficient depth map stitching is achieved. First, a binocular RGB-D camera is used to capture the target plane to obtain multiple sets of data. Then, the target plane in the point cloud is extracted, and the rotation matrix and translation vector between the cameras are calculated to obtain the spatial transformation matrix. Next, a mapping is calculated to map the right camera depth map to the left camera plane to achieve unified depth values. Finally, through feature extraction and geometric correspondence establishment, the stitching fusion parameters are optimized to complete the stitching of the RGB-D images. This method does not require a calibration plate, is adaptable to different lighting conditions, and has broad application prospects.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and digital image processing, and in particular to a real-time binocular RGB-D stitching method based on three-dimensional geometric properties. Background Art

[0002] With the continuous development of digital image processing and computer vision technologies, deep image stitching has become an important research field. Traditional image stitching techniques rely primarily on two-dimensional image data, generating panoramic images through feature point matching and image fusion. However, this approach is prone to distortion and unnatural seams when dealing with complex scenes, occluded objects, and insufficient depth information.

[0003] Recent advances in depth perception technology have provided new solutions for image stitching. Depth cameras (such as LIDAR, structured light, and time-of-flight cameras) can acquire scene depth information in real time, providing more comprehensive data support for stitching algorithms. This depth information not only improves stitching accuracy but also enhances the three-dimensionality of the composite image, making the final result more realistic and natural.

[0004] Accurate depth image stitching is crucial for applications such as virtual reality (VR), augmented reality (AR), robotic navigation, and autonomous driving. These fields place increasing demands on scene understanding and spatial information, prompting researchers to explore more efficient and robust stitching algorithms.

[0005] Currently, most of the existing image stitching methods are RGB image stitching, while there are fewer depth image stitching methods, and there are still some problems, such as the need for calibration plates and the ability to adapt to different lighting conditions.

[0006] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0007] The main purpose of the present invention is to overcome the defects in the above-mentioned background technology and provide a real-time binocular RGB-D stitching method based on three-dimensional geometric properties.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A real-time binocular RGB-D stitching method based on three-dimensional geometric properties includes the following steps:

[0010] S1. Use a fixed binocular RGB-D camera to capture the same target plane in different poses, obtaining multiple sets of RGB images, depth images, and point cloud data.

[0011] S2. Extract the target plane from the two point clouds and calculate the rotation matrix between the cameras based on the unit normal vector of the target plane;

[0012] S3. Based on the rotation matrix, a translation vector is calculated by calculating multiple sets of point cloud data of the target plane, thereby obtaining a spatial transformation matrix between the camera coordinate systems of the two cameras;

[0013] S4. Calculate a mapping using the spatial transformation matrix, wherein the mapping is used to map all pixels on the depth map of the right camera imaging from a plane relative to the right camera to a plane relative to the left camera, to obtain a transformed depth map of the right camera;

[0014] S5. Extract features from images captured in actual application scenarios, establish geometric correspondences and reprojection fusion parameters between the images, and obtain splicing fusion parameters for the two camera images;

[0015] S6. Use the mapping to transform the right camera depth map obtained in real time to obtain a transformed right camera depth map; according to the obtained stitching and fusion parameters, stitch and fuse the RGB images of the left and right cameras, the original depth map of the left camera, and the transformed depth map of the right camera to obtain an RGB-D image with a larger field of view.

[0016] Furthermore, step S5 specifically includes:

[0017] The feature point extraction method is used to extract features from images taken in actual application scenarios, and the geometric relationship between images is determined using feature point matching technology to obtain a rough RGB camera spatial relationship;

[0018] Through the optimization algorithm, the geometric correspondence between accurate images and the reprojection fusion parameters are established;

[0019] Perform seam search to obtain the best seam for seam fusion of two camera images;

[0020] Get the parameters of the two-camera stitching and fusion.

[0021] Furthermore, the feature point extraction method is a SIFT (Scale-Invariant Feature Transform) method.

[0022] Furthermore, the feature point matching technology includes homography matching.

[0023] Furthermore, the optimization algorithm is a BA (Bundle Adjustment) optimization algorithm.

[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the real-time binocular RGB-D stitching method.

[0025] A computer program product includes a computer program, which, when executed by a processor, implements the real-time binocular RGB-D stitching method.

[0026] The present invention has the following beneficial effects:

[0027] The present invention's binocular RGB-D camera image stitching method, based on three-dimensional geometric properties, is a novel and efficient depth image stitching technology. By processing the geometric information of the three-dimensional point cloud acquired by the cameras, the precise spatial relative relationship between the cameras is estimated. The depth values ​​of the depth maps from the two cameras are then unified relative to the same camera plane. This allows for the stitching of depth maps using established RGB image stitching algorithms, resulting in a simple and precise fusion of corresponding RGB-D camera images. This method has significant practical value and broad application prospects, with potential applications in a variety of fields, including virtual reality, augmented reality, robotic navigation, and autonomous driving.

[0028] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a simplified diagram of the overall process of the real-time binocular RGB-D camera image stitching method based on three-dimensional geometric properties according to an embodiment of the present invention.

[0030] Figure 2 This is a detailed flowchart of the offline deep preprocessing stage of an embodiment of the present invention.

[0031] Figure 3 Detailed flowchart of the real-time RGB-D stitching stage of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0033] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0034] The present invention proposes a real-time binocular RGB-D camera image stitching method based on three-dimensional geometric properties, which mainly includes the following steps: Step 1: Use a fixed binocular RGB-D camera to shoot the same wall or other plane in different postures to obtain several sets of RGB images, depth images and point clouds respectively; Step 2: Extract the main planes in the two point clouds, and calculate the rotation matrix between the cameras by using the plane unit normal vector. Matrix); Step 3: Based on the rotation matrix, a more accurate translation vector (TranslationVector) is calculated through multiple sets of data, thereby obtaining the spatial transformation matrix (TransformationMatrix) between the camera coordinate systems of the two cameras; Step 4: Calculate a mapping through the obtained spatial transformation matrix, and map all pixel points on the depth map of the right camera imaging from the value relative to the right camera plane to the value relative to the left camera plane to obtain the transformed right camera depth map; Step 5: Feature extraction is performed on the images obtained by general scene shooting, and the geometric correspondence relationship and reprojection fusion parameters between the images are established; Step 6: Through the geometric correspondence relationship and reprojection fusion parameters between the images of the left and right cameras obtained by the above process, the RGB images of the left and right cameras, the original depth map of the left camera and the transformed depth map of the right camera are reprojected and fused respectively to obtain an RGB image and depth image with a larger field of view (FOV).

[0035] The above-mentioned binocular RGB-D camera image stitching method based on 3D geometric properties estimates the precise spatial relative relationship between cameras by processing the geometric information of the 3D point cloud acquired by the cameras. It then unifies the depth values ​​of the depth maps of the two cameras relative to the same camera plane. The depth maps can then be stitched together using a mature RGB image stitching algorithm, thus achieving a simple and precise corresponding stitching and fusion of RGB-D camera images.

[0036] See Figures 1 to 3 The embodiment of the present invention provides a real-time binocular RGB-D stitching method based on three-dimensional geometric properties, comprising the following steps:

[0037] S1. Use a fixed binocular RGB-D camera to capture the same target plane in different poses, obtaining multiple sets of RGB images, depth images, and point cloud data.

[0038] S2. Extract the target plane from the two point clouds and calculate the rotation matrix between the cameras based on the unit normal vector of the target plane;

[0039] S3. Based on the rotation matrix, a translation vector is calculated by calculating multiple sets of point cloud data of the target plane, thereby obtaining a spatial transformation matrix between the camera coordinate systems of the two cameras;

[0040] S4. Calculate a mapping using the spatial transformation matrix, wherein the mapping is used to map all pixels on the depth map of the right camera imaging from a plane relative to the right camera to a plane relative to the left camera, to obtain a transformed depth map of the right camera;

[0041] S5. Extract features from images captured in actual application scenarios, establish geometric correspondences and reprojection fusion parameters between the images, and obtain splicing fusion parameters for the two camera images;

[0042] S6. Use the mapping to transform the right camera depth map obtained in real time to obtain a transformed right camera depth map; according to the obtained stitching and fusion parameters, stitch and fuse the RGB images of the left and right cameras, the original depth map of the left camera, and the transformed depth map of the right camera to obtain an RGB-D image with a larger field of view.

[0043] In a preferred embodiment, step S5 specifically includes: extracting features from images captured in actual application scenarios using a feature point extraction method, and determining the geometric relationship between images using feature point matching technology to obtain a rough RGB camera spatial relationship; establishing the geometric correspondence between precise images and reprojection fusion parameters through an optimization algorithm; performing seam search to obtain the optimal seam for seam fusion of the two camera images; and obtaining parameters for the two-camera stitching fusion.

[0044] In a preferred embodiment, the feature point extraction method is the SIFT (Scale-Invariant Feature Transform) method. The feature point matching technology includes homography matching. The optimization algorithm is the BA (Bundle Adjustment) optimization algorithm.

[0045] The present invention proposes an efficient real-time binocular RGB-D stitching method, which makes full use of three-dimensional geometric properties to accurately estimate the spatial relative relationship between cameras, thereby solving some key problems existing in traditional depth image stitching. By processing the three-dimensional point cloud data obtained by the binocular RGB-D camera, the present invention can directly calculate the rotation matrix and translation vector between the cameras without relying on the calibration plate, and then obtain the spatial transformation matrix. Using this spatial transformation matrix, the present invention can map the depth map of the right camera to the coordinate system of the left camera, achieve the unification of the depth value, and thus simplify the stitching process of the depth map. In addition, the present invention also optimizes the image stitching and fusion parameters by feature extraction and the establishment of geometric correspondence between images, further improving the quality of the stitched image. This method not only improves the accuracy and efficiency of stitching, but also enhances the adaptability to different lighting conditions, so that the stitched RGB-D image has a larger field of view and a wider application prospect, and is suitable for multiple fields such as virtual reality, augmented reality, robot navigation and autonomous driving.

[0046] Specific embodiments of the present invention are further described below.

[0047] A real-time binocular RGB-D camera image stitching method is proposed, which mainly includes a preprocessing stage and a stitching stage. The preprocessing stage includes depth preprocessing and stitching preprocessing, and its purpose is to obtain a unified mapping and stitching fusion parameters of the depth map between cameras offline. The stitching stage is to achieve online real-time RGB-D stitching based on the results of the preprocessing stage. The specific implementation process of the method includes the following:

[0048] Offline preprocessing:

[0049] 1) Deep preprocessing.

[0050] 1.1 Use a fixed binocular RGB-D camera to shoot the same wall or other surface in different postures to obtain several sets of RGB images, depth images and point clouds;

[0051] 1.2 As Figure 2 As shown, the main planes in the two point clouds are extracted, and the rotation matrix between the cameras is calculated by the plane unit normal vector. The PCD is the point cloud;

[0052] Plane extraction:

[0053] a,b,c,d=RANSAC(PCD(x,y,z),δ)

[0054] Plane normal vector calculation:

[0055]

[0056] Rotation matrix calculation:

[0057]

[0058] 1.3 As Figure 2 As shown in the figure, based on the rotation matrix, a more accurate translation vector is calculated through multiple sets of data, thereby obtaining the spatial transformation matrix between the two camera coordinate systems;

[0059] Translation vector calculation, fixed R:

[0060] t=ICP(PCD1,PCD2,R)

[0061] Space transformation matrix:

[0062]

[0063] 1.4 Calculate a mapping f using the obtained spatial transformation matrix T , all pixel points on the depth map of the right camera imaging are mapped from the value relative to the right camera plane to the value relative to the left camera plane, and the transformed right camera depth map is obtained.

[0064] Transformation of points in the two camera coordinate systems:

[0065]

[0066] And the relationship between the depth value D and the camera coordinate system can be obtained from the intrinsic parameter matrix K:

[0067]

[0068] Combined to get:

[0069] D1=f T (D2)

[0070] 2) Splicing preprocessing.

[0071] 2.1 Use SIFT and other feature point extraction methods to extract features from images captured in general scenes, and perform homography matching to obtain a rough RGB camera spatial relationship;

[0072] 2.2 Through BA optimization, establish more accurate geometric correspondence between images and reprojection fusion parameters;

[0073] 2.3 Perform seam search to obtain the best seam for fusion of the two camera images;

[0074] 2.4 Get the parameter S of the two-camera stitching fusion and save it.

[0075] Online real-time RGB-D stitching:

[0076] 1) Transform the right camera depth map using fT obtained by depth preprocessing;

[0077] 2) The RGB image and the transformed depth image are processed according to the splicing fusion parameter S respectively to obtain the final spliced ​​RGB-D image.

[0078] In summary, the present invention provides a real-time binocular RGB-D camera image stitching method based on three-dimensional geometric properties, which realizes efficient stitching of depth maps by accurately estimating the spatial relative relationship between cameras. First, a binocular RGB-D camera is used to shoot the target plane to obtain multiple sets of data; then, the target plane in the point cloud is extracted, and the rotation matrix and translation vector between the cameras are calculated to obtain the spatial transformation matrix; then, the right camera depth map is mapped to the left camera plane by calculation to achieve depth value unification; finally, through feature extraction and geometric correspondence establishment, the stitching fusion parameters are optimized to complete the stitching of RGB-D images. This method does not require a calibration plate, is adaptable to different lighting conditions, and has broad application prospects.

[0079] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0080] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0081] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0082] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0083] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0084] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0086] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0087] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0088] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0089] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0090] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0091] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A real-time binocular RGB-D stitching method based on three-dimensional geometric properties, characterized in that: The following steps are involved: S1. Use a fixed binocular RGB-D camera to capture the same target plane in different poses, generating multiple sets of RGB images, depth images, and point cloud data. S2. Extract the target plane from the two point clouds and calculate the rotation matrix between the cameras based on the unit normal vector of the target plane. S3. Based on the rotation matrix, a translation vector is calculated by calculating the multiple point cloud data of the target plane, thereby obtaining a spatial transformation matrix between the camera coordinate systems of the two cameras; S4. Calculate the mapping using the spatial transformation matrix, the mapping is used to map all pixels on the depth map of the right camera imaging from the plane relative to the right camera to the plane relative to the left camera value, to obtain the transformed right camera depth map; S5. Perform feature extraction on images captured in the actual application scenario, establish geometric correspondences between the images and reprojection fusion parameters, and obtain the stitching and fusion parameters for the two-camera images. This includes: extracting features from the images captured in the actual application scenario using a feature point extraction method, and using feature point matching techniques to determine the geometric relationship between the images, thereby obtaining a rough RGB camera spatial relationship. Through the optimization algorithm, the geometric correspondence between the precise images and the reprojection fusion parameters are established; the seam search is performed to obtain the optimal seam for the fusion of the two camera images; and the parameters for the two-camera stitching fusion are obtained; S6. Use the mapping to transform the right camera depth map acquired in real time to obtain a transformed right camera depth map; and stitch and fuse the RGB images of the left and right cameras, the original depth map of the left camera, and the transformed depth map of the right camera based on the obtained stitching and fusion parameters to obtain an RGB-D image with a larger field of view.

2. The real-time binocular RGB-D stitching method based on three-dimensional geometric properties according to claim 1, characterized in that: The feature point extraction method is the SIFT method.

3. The real-time binocular RGB-D stitching method based on three-dimensional geometric properties according to claim 1, characterized in that: The feature point matching technology includes homography matching.

4. The real-time binocular RGB-D stitching method based on three-dimensional geometric properties according to claim 1, characterized in that: The optimization algorithm is the Bundle Adjustment optimization algorithm.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the real-time binocular RGB-D stitching method according to any one of claims 1 to 4 is implemented.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the real-time binocular RGB-D stitching method according to any one of claims 1 to 4 is implemented.

Images