Real-time binocular RGB-D splicing method based on three-dimensional geometric property

Abstract

The invention provides a real-time binocular RGB-D camera image splicing method based on three-dimensional geometric properties, and efficient splicing of depth images is realized by accurately estimating a spatial relative relationship between cameras. Firstly, a binocular RGB-D camera is used for shooting a target plane to obtain multiple groups of data; then, extracting a target plane in the point cloud, and calculating a rotation matrix and a translation vector between cameras to obtain a spatial transformation matrix; thirdly, mapping a right camera depth map to a left camera plane through calculation mapping to realize depth value unification; and finally, through feature extraction and geometric correspondence establishment, optimizing splicing fusion parameters, and completing RGB-D image splicing. The method does not need a calibration plate, adapts to different illumination conditions, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and digital image processing, and in particular to a real-time binocular RGB-D splicing method based on three-dimensional geometric properties. Background Art

[0002] With the continuous development of digital image processing and computer vision technology, deep image stitching has become an important research field. Traditional image stitching technology mainly relies on the data of two-dimensional images to generate panoramas through feature point matching and image fusion. However, this method is prone to distortion or unnatural seams when dealing with complex scenes, occluded objects and insufficient depth information.

[0003] In recent years, advances in depth perception technology have provided new solutions for image stitching. Depth cameras (such as LIDAR, structured light, and time-of-flight cameras) can obtain depth information of the scene in real time, thus providing more comprehensive data support for the stitching algorithm. This depth information can not only improve the accuracy of stitching, but also enhance the three-dimensional sense of the synthesized image, making the final result more realistic and natural.

[0004] Accurate depth image stitching is particularly important in applications such as virtual reality (VR), augmented reality (AR), robot navigation and autonomous driving. The increasing requirements for scene understanding and spatial information in these fields have prompted researchers to continuously explore more efficient and robust stitching algorithms.

[0005] Currently, most of the existing image stitching methods are RGB image stitching, and there are fewer depth image stitching methods. In addition, there are still some problems, such as the need for calibration plates and the ability to adapt to different lighting conditions.

[0006] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0007] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a real-time binocular RGB-D stitching method based on three-dimensional geometric properties.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A real-time binocular RGB-D stitching method based on three-dimensional geometric properties comprises the following steps:

[0010] S1. Use a fixed binocular RGB-D camera to shoot the same target plane in different postures to obtain multiple sets of RGB images, depth images and point cloud data;

[0011] S2. extract the target planes from the two point clouds, and calculate the rotation matrix between the cameras based on the unit normal vector of the target planes;

[0012] S3. Based on the rotation matrix, a translation vector is calculated by using multiple sets of point cloud data of the target plane, thereby obtaining a spatial transformation matrix between the camera coordinate systems of the two cameras;

[0013] S4. Calculate a mapping using the spatial transformation matrix, wherein the mapping is used to map all pixel points on the depth map of the right camera imaging from values ​​relative to the right camera plane to values ​​relative to the left camera plane, thereby obtaining a transformed right camera depth map;

[0014] S5. Extract features from images taken in actual application scenarios, establish geometric correspondences and reprojection fusion parameters between images, and obtain splicing fusion parameters of the two camera images;

[0015] S6. Use the mapping to transform the right camera depth map obtained in real time to obtain a transformed right camera depth map; according to the obtained stitching and fusion parameters, stitch and fuse the RGB images of the left and right cameras, the original depth map of the left camera, and the transformed depth map of the right camera to obtain an RGB-D image with a larger field of view.

[0016] Furthermore, step S5 specifically includes:

[0017] The feature point extraction method is used to extract features from images taken in actual application scenarios, and the geometric relationship between images is determined using feature point matching technology to obtain a rough RGB camera spatial relationship;

[0018] Through the optimization algorithm, the geometric correspondence between accurate images and the reprojection fusion parameters are established;

[0019] Perform seam search to obtain the best seam for seam fusion of two camera images;

[0020] Get the parameters of the two-camera stitching and fusion.

[0021] Furthermore, the feature point extraction method is a SIFT (Scale-Invariant Feature Transform) method.

[0022] Furthermore, the feature point matching technology includes homography matching.

[0023] Furthermore, the optimization algorithm is a BA (Bundle Adjustment) optimization algorithm.

[0024] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the real-time binocular RGB-D stitching method is implemented.

[0025] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the real-time binocular RGB-D stitching method is implemented.

[0026] The present invention has the following beneficial effects:

[0027] The binocular RGB-D camera image stitching method based on three-dimensional geometric properties of the present invention is a novel and efficient depth image stitching technology. It estimates the precise spatial relative relationship between cameras by processing the geometric information of the three-dimensional point cloud acquired by the camera, and then unifies the depth values ​​of the depth maps of the two cameras to the same camera plane. The depth map stitching can be realized according to the mature RGB image stitching algorithm, so that the corresponding stitching fusion of RGB-D camera images is realized concisely and accurately. The method of the present invention has important practical value and broad application prospects, and is suitable for various application fields such as virtual reality, augmented reality, robot navigation and autonomous driving.

[0028] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall process of a real-time binocular RGB-D camera image stitching method based on three-dimensional geometric properties according to an embodiment of the present invention.

[0030] Figure 2 It is a detailed flow chart of the offline deep preprocessing stage of an embodiment of the present invention.

[0031] Figure 3 Detailed flowchart of the real-time RGB-D stitching stage of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0033] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0034] The present invention proposes a real-time binocular RGB-D camera image stitching method based on three-dimensional geometric properties, which mainly includes the following steps: the first step: using a fixed binocular RGB-D camera to shoot the same wall or other plane in different postures, and obtaining several groups of RGB images, depth images and point clouds respectively; the second step: extracting the main planes in the two point clouds, and calculating the rotation matrix (Rotation matrix) between the cameras through the plane unit normal vector Matrix); Step 3: Based on the rotation matrix, a more accurate translation vector (TranslationVector) is calculated through multiple sets of data, thereby obtaining the spatial transformation matrix (TransformationMatrix) between the camera coordinate systems of the two cameras; Step 4: A mapping is calculated through the obtained spatial transformation matrix to map all pixel points on the depth map of the right camera from the plane relative to the right camera to the plane relative to the left camera, and the transformed depth map of the right camera is obtained; Step 5: Feature extraction is performed on the images obtained by general scene shooting, and the geometric correspondence between the images and the reprojection fusion parameters are established; Step 6: The RGB images of the left and right cameras, the original depth map of the left camera, and the transformed depth map of the right camera are reprojected and fused respectively through the geometric correspondence between the images and the reprojection fusion parameters obtained by the above process, so as to obtain an RGB image and a depth image with a larger field of view (FOV).

[0035] The above-mentioned binocular RGB-D camera image stitching method based on three-dimensional geometric properties estimates the precise spatial relative relationship between cameras by processing the geometric information of the three-dimensional point cloud obtained by the camera, and then unifies the depth values ​​of the depth maps of the two cameras to the same camera plane. The depth map stitching can be realized according to the mature RGB image stitching algorithm, thus realizing the corresponding stitching and fusion of RGB-D camera images in a simple and accurate manner.

[0036] See also Figures 1 to 3 The embodiment of the present invention provides a real-time binocular RGB-D stitching method based on three-dimensional geometric properties, comprising the following steps:

[0037] S1. Use a fixed binocular RGB-D camera to shoot the same target plane in different postures to obtain multiple sets of RGB images, depth images and point cloud data;

[0038] S2. extract the target planes from the two point clouds, and calculate the rotation matrix between the cameras based on the unit normal vector of the target planes;

[0039] S3. Based on the rotation matrix, a translation vector is calculated by using multiple sets of point cloud data of the target plane, thereby obtaining a spatial transformation matrix between the camera coordinate systems of the two cameras;

[0040] S4. Calculate a mapping using the spatial transformation matrix, wherein the mapping is used to map all pixel points on the depth map of the right camera imaging from values ​​relative to the right camera plane to values ​​relative to the left camera plane, thereby obtaining a transformed right camera depth map;

[0041] S5. Extract features from images taken in actual application scenarios, establish geometric correspondences and reprojection fusion parameters between images, and obtain splicing fusion parameters of the two camera images;

[0042] S6. Use the mapping to transform the right camera depth map obtained in real time to obtain a transformed right camera depth map; according to the obtained stitching and fusion parameters, stitch and fuse the RGB images of the left and right cameras, the original depth map of the left camera, and the transformed depth map of the right camera to obtain an RGB-D image with a larger field of view.

[0043] In a preferred embodiment, step S5 specifically includes: extracting features from images captured in actual application scenarios using a feature point extraction method, and determining the geometric relationship between images using a feature point matching technique to obtain a rough RGB camera spatial relationship; establishing a geometric correspondence relationship and reprojection fusion parameters between precise images through an optimization algorithm; performing seam search to obtain the best seam for seam fusion of two camera images; and obtaining parameters for stitching and fusion of the two cameras.

[0044] In a preferred embodiment, the feature point extraction method is a SIFT (Scale-Invariant Feature Transform) method. The feature point matching technology includes homography matching. The optimization algorithm is a BA (Bundle Adjustment) optimization algorithm.

[0045] The present invention proposes an efficient real-time binocular RGB-D stitching method, which makes full use of the three-dimensional geometric properties to accurately estimate the spatial relative relationship between cameras, thereby solving some key problems existing in traditional depth image stitching. By processing the three-dimensional point cloud data obtained by the binocular RGB-D camera, the present invention can directly calculate the rotation matrix and translation vector between the cameras without relying on the calibration plate, and then obtain the spatial transformation matrix. Using this spatial transformation matrix, the present invention can map the depth map of the right camera to the coordinate system of the left camera, realize the unification of the depth value, and thus simplify the stitching process of the depth map. In addition, the present invention also optimizes the image stitching and fusion parameters by feature extraction and the establishment of geometric correspondence between images, and further improves the quality of the stitched image. This method not only improves the accuracy and efficiency of stitching, but also enhances the adaptability to different lighting conditions, so that the stitched RGB-D image has a larger field of view and a wider application prospect, and is suitable for multiple fields such as virtual reality, augmented reality, robot navigation and autonomous driving.

[0046] The specific embodiments of the present invention are further described below.

[0047] A real-time binocular RGB-D camera image stitching method mainly includes a preprocessing stage and a stitching stage. The preprocessing stage includes depth preprocessing and stitching preprocessing, the purpose of which is to obtain the unified mapping and stitching fusion parameters of the depth map between cameras offline; the stitching stage is to realize online real-time RGB-D stitching based on the results of the preprocessing stage. The specific method includes the following implementation process:

[0048] Offline preprocessing:

[0049] 1) Deep preprocessing.

[0050] 1.1 Use a fixed binocular RGB-D camera to shoot the same wall or other plane in different postures to obtain several sets of RGB images, depth images and point clouds;

[0051] 1.2 As Figure 2 As shown, the main planes in the two point clouds are extracted, and the rotation matrix between the cameras is calculated by the plane unit normal vector. The PCD is the point cloud;

[0052] Plane extraction:

[0053] a,b,c,d=RANSAC(PCD(x,y,z),δ)

[0054] Plane normal vector calculation:

[0055]

[0056] Rotation matrix calculation:

[0057]

[0058] 1.3 As Figure 2 As shown in the figure, based on the rotation matrix, a more accurate translation vector is obtained by calculating multiple sets of data, thereby obtaining the spatial transformation matrix between the two camera coordinate systems;

[0059] Translation vector calculation, fixed R:

[0060] t=ICP(PCD1,PCD2,R)

[0061] Space transformation matrix:

[0062]

[0063] 1.4 Calculate a mapping f using the obtained spatial transformation matrix T , all pixel points on the depth map of the right camera imaging are mapped from values ​​relative to the right camera plane to values ​​relative to the left camera plane, and the transformed right camera depth map is obtained.

[0064] Transformation of points in the two camera coordinate systems:

[0065]

[0066] And the relationship between the depth value D and the camera coordinate system can be obtained from the intrinsic parameter matrix K:

[0067]

[0068] Combined to get:

[0069] D1=f T (D2)

[0070] 2) Splicing preprocessing.

[0071] 2.1 Use SIFT and other feature point extraction methods to extract features from images taken in general scenes, and perform homography matching to obtain a rough RGB camera spatial relationship;

[0072] 2.2 Through BA optimization, establish more accurate geometric correspondence between images and reprojection fusion parameters;

[0073] 2.3 Search for the seam and obtain the best seam for the fusion of the two camera images;

[0074] 2.4 Get the parameter S of the two-camera stitching fusion and save it.

[0075] Online real-time RGB-D stitching:

[0076] 1) Transform the right camera depth map using fT obtained by depth preprocessing;

[0077] 2) The RGB image and the transformed depth image are processed according to the splicing fusion parameter S respectively to obtain the final spliced ​​RGB-D image.

[0078] In summary, the present invention provides a real-time binocular RGB-D camera image stitching method based on three-dimensional geometric properties, which realizes efficient stitching of depth maps by accurately estimating the spatial relative relationship between cameras. First, use a binocular RGB-D camera to shoot the target plane to obtain multiple sets of data; then, extract the target plane in the point cloud, calculate the rotation matrix and translation vector between the cameras, and obtain the spatial transformation matrix; then, calculate the mapping to map the right camera depth map to the left camera plane to achieve the unification of depth values; finally, through feature extraction and geometric correspondence establishment, the stitching fusion parameters are optimized to complete the stitching of RGB-D images. This method does not require a calibration plate, is adaptable to different lighting conditions, and has broad application prospects.

[0079] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0080] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0081] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0082] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0083] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0084] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0085] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0086] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0087] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0088] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0089] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0090] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0091] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A real-time binocular RGB-D stitching method based on three-dimensional geometric properties, characterized in that: The following steps are involved: S1. Use a fixed binocular RGB-D camera to shoot the same target plane in different postures to obtain multiple sets of RGB images, depth images and point cloud data; S2. extract the target planes from the two point clouds, and calculate the rotation matrix between the cameras based on the unit normal vector of the target planes; S3. Based on the rotation matrix, a translation vector is calculated by using multiple sets of point cloud data of the target plane, thereby obtaining a spatial transformation matrix between the camera coordinate systems of the two cameras; S4. Calculate a mapping using the spatial transformation matrix, wherein the mapping is used to map all pixel points on the depth map of the right camera imaging from values ​​relative to the right camera plane to values ​​relative to the left camera plane, thereby obtaining a transformed right camera depth map; S5. Extract features from images captured in actual application scenarios, establish geometric correspondences and reprojection fusion parameters between images, and obtain splicing fusion parameters of the two camera images; S6. Use the mapping to transform the right camera depth map obtained in real time to obtain a transformed right camera depth map; according to the obtained stitching and fusion parameters, stitch and fuse the RGB images of the left and right cameras, the original depth map of the left camera, and the transformed depth map of the right camera to obtain an RGB-D image with a larger field of view.

2. The real-time binocular RGB-D stitching method based on three-dimensional geometric properties as claimed in claim 1, characterized in that: Step S5 specifically includes: The feature point extraction method is used to extract features from images taken in actual application scenarios, and the geometric relationship between images is determined using feature point matching technology to obtain a rough RGB camera spatial relationship; Through the optimization algorithm, the geometric correspondence between accurate images and the reprojection fusion parameters are established; Perform seam search to obtain the best seam for seam fusion of two camera images; Get the parameters of the two-camera stitching and fusion.

3. The real-time binocular RGB-D stitching method based on three-dimensional geometric properties as claimed in claim 2, characterized in that: The feature point extraction method is a SIFT (Scale-Invariant Feature Transform) method.

4. The real-time binocular RGB-D stitching method based on three-dimensional geometric properties as claimed in claim 2, characterized in that: The feature point matching technology includes homography matching.

5. The real-time binocular RGB-D stitching method based on three-dimensional geometric properties as claimed in claim 2, characterized in that: The optimization algorithm is a BA (Bundle Adjustment) optimization algorithm.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the real-time binocular RGB-D stitching method according to any one of claims 1 to 5 is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the real-time binocular RGB-D stitching method according to any one of claims 1 to 8 is implemented.

Images