Data fusion method, structured light camera device, electronic device, and storage medium

Abstract

The embodiment of the present application relates to the field of image processing, and discloses a data fusion method, a structured light camera device, an electronic device and a storage medium, wherein the data fusion method comprises: acquiring depth maps collected when a structured light camera rotates to a plurality of preset angles around a specified central axis; wherein during the rotation of the structured light camera, the optical center of an infrared lens of the structured light camera and the projection point of a speckle projector are always located at a fixed position of the central axis and synchronously rotate around the central axis; the depth maps collected at the plurality of preset angles are converted into cylindrical projection maps from planar projection maps, and adjacent cylindrical projection maps after the conversion are spliced to obtain a depth panorama. The data fusion method can generate a depth panorama with a real scene scale, thereby bringing a user a relatively real visual viewing effect.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing, and in particular to a data fusion method, a structured light camera device, an electronic device, and a storage medium. Background Technology

[0002] When renting or selling a house, you can usually see panoramic views of the house displayed by the real estate agent or the seller. Buyers can view the house information in 360° by swiping the screen.

[0003] However, to reduce image costs, businesses commonly create panoramic images by stretching the aspect ratio of captured 2D images. Because the stretched image alters the true proportions of the image content in the visual field, the image content does not match the actual scene, resulting in a poor visual effect. Summary of the Invention

[0004] The purpose of this invention is to provide a data fusion method, a structured light camera device, an electronic device, and a storage medium that can generate a depth panoramic image with a realistic scene scale, thereby providing users with a more realistic visual viewing experience.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a data fusion method, comprising: acquiring depth maps obtained by a structured light camera rotating around a specified central axis to multiple preset angles; wherein, during the rotation of the structured light camera, the optical center of the infrared lens of the structured light camera and the projection point of the speckle projector are always located at fixed positions on the central axis and rotate synchronously around the central axis; converting the depth maps obtained at the multiple preset angles from planar projection maps to cylindrical projection maps, and stitching together adjacent cylindrical projection maps after conversion to obtain a depth panoramic image.

[0006] An embodiment of the present invention also provides a structured light camera device for performing the data fusion method described above, comprising: a structured light camera and a stepper motor located at the bottom of the structured light camera; the structured light camera is equipped with an infrared lens, a color lens, and a speckle projector; the stepper motor is used to drive the structured light camera to rotate around a specified central axis to multiple preset angles; the infrared lens structured light camera is used to acquire depth maps and color maps when rotating around the specified central axis to multiple preset angles, convert the depth maps acquired at the multiple preset angles from planar projection maps to cylindrical projection maps, and stitch together adjacent cylindrical projection maps after conversion to obtain a depth panoramic image; the structured light camera is also used to perform error correction on the depth map using the color image; wherein, during the rotation of the structured light camera, the optical center of the infrared lens, the optical center of the color lens, and the projection point of the speckle projector are always located at a fixed position on the central axis and rotate synchronously around the central axis.

[0007] Embodiments of the present invention also provide an electronic device, comprising: at least one processor; a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data fusion method described above.

[0008] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data fusion method.

[0009] The data fusion method of this invention, during the rotation of the structured light camera, ensures that the optical center of the infrared lens and the projection point of the speckle projector remain fixed at the central axis and rotate synchronously around it. Therefore, one rotation of the structured light camera acquires depth maps obtained when it rotates to multiple preset angles around the designated central axis, thus obtaining all depth information of the environment in which the structured light camera is located. By converting the depth maps acquired at multiple preset angles from planar projection to cylindrical projection, each depth map can be transformed from a planar projection captured by the camera into a cylindrical projection that conforms to human visual perception. Adjacent cylindrical projections are then stitched together to create a panoramic image, resulting in a depth panorama. Since this depth panorama reflects the image proportions of the real scene, it provides users with a more realistic visual experience.

[0010] In addition, a stepper motor is installed at the bottom of the structured light camera to acquire depth maps obtained when the structured light camera rotates around a specified central axis to multiple preset angles. This includes controlling the structured light camera to rotate around the central axis to multiple preset angles via the stepper motor, and controlling the acquisition of depth maps at preset angle positions. In this application, because a stepper motor is installed at the bottom of the structured light camera, the stepper motor can control the structured light camera to rotate at the same angle each time, thereby controlling the structured light camera to rotate around the central axis to multiple preset angles, and controlling the acquisition of depth maps at preset angle positions, thus obtaining the planar projection depth map required to generate the depth panorama.

[0011] In addition, the structured light camera also includes a color lens. During the rotation of the structured light camera, the optical center of the color lens is always located at a fixed position on the central axis and rotates synchronously around the central axis. Before converting the depth maps acquired at multiple preset angles from planar projection to cylindrical projection and stitching the adjacent cylindrical projections after conversion to obtain a depth panoramic image, the method further includes: acquiring color images acquired when the structured light camera rotates around a specified central axis to multiple preset angles; performing error correction on the depth maps based on the color images; converting the depth maps acquired at multiple preset angles from planar projection to cylindrical projection and stitching the adjacent cylindrical projections after conversion to obtain a depth panoramic image, including: converting the error-corrected depth maps acquired at multiple preset angles from planar projection to cylindrical projection and stitching the adjacent cylindrical projections after conversion to obtain a depth panoramic image. In this application, since the depth map can only reflect the depth information of the photographed object, it is impossible to distinguish between two different objects with the same depth value. Therefore, there may be some errors in the depth map. Based on the color map, different objects can be further distinguished through color information. Therefore, a color lens is set in the structured light camera. The optical center of the color lens is always located at a fixed position on the central axis and rotates synchronously around the central axis. The color map is acquired when the structured light camera rotates around the specified central axis to multiple preset angles. Based on the color map, the depth map is corrected for errors, thereby obtaining a more accurate depth map, and thus a more accurate depth panoramic image.

[0012] Furthermore, based on the color image, error correction is performed on the depth map, including: obtaining the translation matrix of the structured light camera based on the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth map; wherein, adjacent images are captured at adjacent preset angles during the rotation of the structured light camera, and the translation matrix represents the translation relationship between the pose of each image in the color image and the depth map and the reference pose, the reference pose being selected from multiple poses of the structured light camera, and multiple poses corresponding to multiple preset angles; and optical center alignment processing is performed on the depth map based on the translation matrix. In this application, since adjacent images are captured at adjacent preset angles during the rotation of the structured light camera, there are corresponding point pairs between two adjacent images. Based on the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth map, and based on the positional relationship between the corresponding point pairs, the translation matrix of the structured light camera representing the translation relationship between the pose of each image in the color image and the depth map and the reference pose can be obtained, and optical center alignment processing is performed on the depth map based on the translation matrix, thereby correcting the error of the depth map.

[0013] In addition, error correction of the depth map also includes: interpolating and filling blank areas in the depth map after optical center alignment; where blank areas are image regions without captured content caused by optical center movement during optical center alignment. In this application, blank areas are image regions without captured content caused by optical center movement during optical center alignment. Therefore, interpolating and filling blank areas in the depth map after optical center alignment can obtain more complete image content, thereby avoiding blank areas that affect the image aesthetics after optical center movement, and thus improving the user experience. Attached Figure Description

[0014] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0015] Figure 1 This is a schematic flowchart of a data fusion method according to an embodiment of the present invention. Figure 1 ;

[0016] Figure 2 This is a schematic flowchart of a data fusion method according to an embodiment of the present invention. Figure 2 ;

[0017] Figure 3 This is a schematic diagram of the preset step angle used in a structured light camera according to an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of a structured light camera device according to an embodiment of the present invention;

[0019] Figure 5 This is a schematic diagram of an electronic device structure provided in one embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the various embodiments of the present invention to facilitate a better understanding of this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0021] This invention relates to a data fusion method. The data fusion method of this invention is executed by an image processing module, which can be built into a structured light camera device or installed in other electronic devices. That is, the executing entity of the data fusion method of this invention can be a structured light camera device or other electronic devices. The specific process is as follows: Figure 1 As shown.

[0022] Step 101: Obtain depth maps acquired by the structured light camera when it rotates around a specified central axis to multiple preset angles; wherein, during the rotation of the structured light camera, the optical center of the infrared lens of the structured light camera and the projection point of the speckle projector are always located at a fixed position on the central axis and rotate synchronously around the central axis.

[0023] Step 102: Convert the depth maps collected at multiple preset angles from planar projection maps to cylindrical projection maps, and stitch the adjacent cylindrical projection maps after conversion to obtain a panoramic depth map.

[0024] The data fusion method of this embodiment is used to generate depth panoramic images, which can be used in systems such as house display and site display to ensure that the images can restore the real information of the scene. Taking a house display system as an example, the house display system can include a shooting module, an image processing module, and a display module to provide users with online house viewing services. The shooting module is placed in a structured light camera device. The structured light camera device needs to be placed in the house to be displayed in advance or in real time to capture a depth map. The image processing module generates a depth panoramic image, which is then displayed to the user by the display module. The image processing module can be assembled in the structured light camera device or assembled separately in an image processing server. This application describes the case where the image processing module can be assembled in the structured light camera device as an example. When the structured light camera captures the depth map, it can be transmitted to the image processing module to generate the depth panoramic image. The image processing module can automatically transmit the depth panoramic image to the display module, or the staff can manually transmit the depth panoramic image of the house to the display module, or the depth panoramic image can be stored in a database, and the display module can download the house display panoramic image from the database in response to user requests. Online home viewing users possess electronic devices equipped with display modules, through which they view and select properties.

[0025] However, some unscrupulous vendors, in order to reduce image costs, commonly generate panoramic images by stretching the aspect ratio of captured 2D images. Because this stretching alters the true proportional relationship of the image content in visual perception, the image content does not match the real scene, resulting in poor visual quality. Therefore, there is an urgent need for a method that can accurately reflect the proportional relationship of the real scene content, providing users with a more realistic visual experience.

[0026] The data fusion method of this application, because during the rotation of the structured light camera, the optical center of the infrared lens and the projection point of the speckle projector are always located at a fixed position on the central axis and rotate synchronously around the central axis, can acquire depth maps when the structured light camera rotates to multiple preset angles around the specified central axis in one revolution. That is, it can acquire all the depth information of the environment in which the structured light camera is located. By converting the depth maps acquired at multiple preset angles from planar projection maps to cylindrical projection maps, the planar projection maps captured by the camera can be converted into cylindrical projection maps that conform to the visual effect of the human eye. The adjacent cylindrical projection maps after conversion are stitched together to combine the cylindrical projection maps into a panoramic image, thereby obtaining a depth panoramic image. Since the depth panoramic image reflects the details of the captured object in three-dimensional space, each pixel in the depth panoramic image carries real spatial depth information, which can accurately reflect the proportional relationship of the real scene content, thus bringing users a more realistic visual viewing effect.

[0027] The following is a detailed description of the implementation details of the data fusion method in this embodiment. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0028] In step 101, the structured light camera acquires depth maps obtained by rotating around a specified central axis to multiple preset angles. These preset angles are specific angles that can be preset by engineers according to different usage scenarios and are controlled by the rotation mechanism within the structured light camera. For example, they can be set to capture images of the same area in adjacent images, or to return to their original position after one full rotation. The structured light camera controls itself to rotate by the preset angle each time it takes a rotational shot. During the rotation of the structured light camera, the optical center of the infrared lens and the projection point of the speckle projector remain at a fixed position on the central axis and rotate synchronously around it.

[0029] In one example, a stepper motor is installed at the bottom of the structured light camera. The structured light camera device controls its rotation by a preset angle each time through the stepper motor. Obtaining depth maps acquired when the structured light camera rotates around a specified central axis to multiple preset angles can be achieved as follows: The structured light camera is controlled to rotate around the central axis to multiple preset angles via a stepper motor, and at each preset angle position, the depth map acquired by the structured light camera is controlled.

[0030] In this embodiment, since a stepper motor is provided at the bottom of the structured light camera, the structured light camera can be controlled to rotate at the same angle each time, thereby controlling the structured light camera to rotate around the central axis to multiple preset angles, and at the preset angle position, the depth map acquired by the structured light camera can be obtained, thus obtaining the planar projection depth map required to generate the depth panorama.

[0031] In step 102, the structured light camera device converts the depth maps acquired at multiple preset angles from planar projection to cylindrical projection, and then stitches together adjacent cylindrical projections to obtain a depth panorama. Specifically, the structured light camera device can set the preset angles to precisely connect the captured images, thus directly transforming the depth maps from planar projections to cylindrical projections, and then stitching together adjacent cylindrical projections to obtain the depth panorama.

[0032] In one example, the structured light camera also includes a color lens, wherein the optical center of the color lens remains fixed at the central axis and rotates synchronously around the central axis during the rotation of the structured light camera. The optical centers of the color lens, the infrared lens, and the projection point of the speckle projector can be arranged vertically, and the distance between them should be as close as possible to minimize the difference between the color image and the depth image. Before converting the depth images acquired at multiple preset angles from planar projection to cylindrical projection and stitching together adjacent cylindrical projections to obtain the depth panorama, the structured light camera also acquires color images acquired when rotating around a specified central axis to multiple preset angles. Based on the color images, error correction is performed on the depth image. For example, when there is a deviation between the actual rotation angle of the camera and the preset rotation angle, this method can be used for correction.

[0033] The depth maps acquired from multiple preset angles are converted from planar projection to cylindrical projection, and the adjacent cylindrical projections after conversion are stitched together to obtain a panoramic depth map. This is achieved by converting the error-corrected depth maps acquired from multiple preset angles from planar projection to cylindrical projection, and then stitching the adjacent cylindrical projections after conversion together to obtain a panoramic depth map.

[0034] In this embodiment, since the depth map can only reflect the depth information of the object being photographed, it is impossible to distinguish between two different objects with the same depth value. Therefore, there may be some errors in the depth map. Based on the color map, different objects can be further distinguished through color information. Therefore, a color lens is set in the structured light camera. The optical center of the color lens is always located at a fixed position on the central axis and rotates synchronously around the central axis. The color map is acquired when the structured light camera rotates around the specified central axis to multiple preset angles. Based on the color map, the depth map is corrected for errors, thereby obtaining a more accurate depth map, and thus a more accurate depth panoramic image.

[0035] In one example, error correction of the depth map based on the color image can be achieved by: obtaining the translation matrix of the structured light camera based on the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth map; wherein, the adjacent images are captured at adjacent preset angles during the rotation of the structured light camera, and the translation matrix represents the translation relationship between the pose of each image in the color image and the depth map and the reference pose. The reference pose is selected from multiple poses of the structured light camera, and multiple poses correspond to multiple preset angles. Based on the translation matrix, the depth map is subjected to optical center alignment processing.

[0036] In this embodiment, since adjacent images are captured at adjacent preset angles during the rotation of the structured light camera, there are corresponding point pairs between two adjacent images. Based on the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth image, and based on the positional relationship between the corresponding point pairs, the translation matrix of the structured light camera representing the translation relationship between the pose of each image in the color image and the reference pose and the position of the reference pose can be obtained. Based on the translation matrix, the optical center alignment processing of the depth image is performed, thereby correcting the error of the depth image.

[0037] Furthermore, after optical center alignment, the blank areas at the boundaries caused by the alignment need to be filled. This can be done by filling the blank areas at the boundaries with content from adjacent images.

[0038] In one example, error correction of the depth map also includes the following steps: interpolating and filling blank areas in the depth map after optical center alignment, where blank areas are image areas without captured content caused by optical center movement during optical center alignment.

[0039] Analysis: Blank areas are image regions without captured content caused by the movement of the optical center during the optical center alignment process of the depth map. By interpolating and filling blank areas in the depth map after optical center alignment, a more complete image content can be obtained, thus avoiding blank areas that affect the aesthetics of the image after optical center movement, thereby improving the user experience.

[0040] In one example, when the structured light camera performs error correction on the depth map based on the color image, it also obtains the rotation matrix of the structured light camera according to the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth map. The rotation matrix represents the rotational relationship between the pose of each image in the color image and the depth map and the reference pose. When stitching together the transformed adjacent cylindrical projection images to obtain the depth panorama, the rotation matrix is ​​also used to stitch together the transformed adjacent cylindrical projection images to obtain the depth panorama.

[0041] Furthermore, based on the rotation matrix, the adjacent cylindrical projection images after transformation are stitched together to obtain a depth panorama. This can be achieved in the following way: Based on the rotation matrix and the corresponding point pairs of adjacent images in the depth image, the region to be fused in the depth image is obtained. Within the region to be fused, the point with the smallest gray-level difference is searched through dynamic programming. Based on the point with the smallest gray-level difference, a stitching line is formed. Based on the stitching line, the depth images after cylindrical projection are stitched together.

[0042] In this embodiment, the region to be fused in the depth map is obtained by using the rotation matrix and the corresponding point pairs of adjacent images in the depth map. Within the region to be fused, the point with the smallest gray-level difference is searched by dynamic programming. Based on the point with the smallest gray-level difference, a stitching line is formed. Then, the depth map after cylindrical projection can be stitched quickly according to the stitching line.

[0043] In one example, the translation and rotation matrices are obtained as follows: The intrinsic and extrinsic parameter matrices of the infrared lens, color lens, and speckle projector are acquired. Based on the intrinsic parameter matrix, corresponding point pairs in adjacent images in the color image and depth image are converted from image coordinates to camera coordinates. Based on the conversion result, the essential matrix is ​​calculated. Then, the differential rotation and differential translation matrices between each pose are calculated based on the essential matrix. The propagation error of the extrinsic parameter matrix is ​​optimized. Finally, the rotation and translation matrices are obtained based on the differential rotation and differential translation matrices. The intrinsic and extrinsic parameters of the color lens, infrared lens, and speckle projector are all part of the camera parameters, which are preset by the engineer based on the model of the infrared lens and speckle projector selected during the design of the structured light camera.

[0044] In one example, the conversion from image coordinates to camera coordinates is achieved using the following expression:

[0045]

[0046] Where Z ik Let X be the depth value of the k-th pair of corresponding points in the depth map under the i-th pose. ik Y ikLet u represent the X and Y coordinates of the k-th pair of corresponding points in the depth map under the i-th pose, respectively, after transformation in the camera coordinate system. ik v ik Let cx, cy, fx, and fy represent the coordinates of the k-th pair of speckle pairs with the same name in the depth map under the i-th pose on the u and v axes of the image coordinate system, respectively. cx, cy, fx, and fy are obtained from the internal parameter matrix. cx and cy are the translation dimensions of the origin of the coordinate system during the coordinate transformation from the image coordinate system to the camera coordinate system. fx and fy are the quotients of the image distance and the physical length of the pixel during the coordinate transformation from the image coordinate system to the camera coordinate system.

[0047] In one example, optical center alignment is achieved using the following expression:

[0048]

[0049] Where (u', v') are the coordinates of the point (u, v) in the color image after the translation matrix transformation, and (X', Y', Z') are the coordinates of the point corresponding to (u', v') after the depth map is transformed from the image coordinate system to the camera coordinate system. Tx, Ty, and Tz are the coordinate changes of the X, Y, and Z axes during the translation matrix transformation, respectively.

[0050] In one example, the data fusion method of this application is implemented as follows: Figure 2 As shown, it includes:

[0051] 1. Based on the size of the structured light camera, select the appropriate lenses to ensure that the optical centers of the infrared and color lenses are aligned with the central axis of the stepper motor shaft.

[0052] 2. Obtain camera parameters. Camera parameters typically refer to the extrinsic and intrinsic parameters of the infrared and color lenses in a structured light camera, as well as the extrinsic parameters of the infrared lens and the speckle projector. The extrinsic parameters refer to the rotation component matrix R and the translation vector T between the two components.

[0053] 3. Determine the rotation angle of the motor for a single shot. Infrared and color lenses typically output pixel-aligned images, where pixels at the same position represent the same object. Based on the aligned image, the horizontal field of view α1 of the lens can be obtained, thus determining the rotation angle α2 of the motor for a single shot. Typically, α2 is less than or equal to half of α1.

[0054] 4. Collect panoramic data. Rotate the stepper motor; collect data once for every α2 rotations. The data includes a color image (I). colori and depth map I depthi Where i represents the i-th data collection, and assume that a total of N data collections are needed to complete the collection of content covering 360° horizontally, such as... Figure 3 As shown.

[0055] 5. Obtain corresponding point pairs for adjacent color image pairs. Obtain corresponding point pairs for adjacent color image pairs through SIFT feature matching.

[0056] 6. Obtain the camera rotation matrix R under each pose. i Translation matrix T i Although the camera's optical center was designed to be positioned on the rotating axis during selection, assembly and lens manufacturing errors prevent the optical center from perfectly aligning with the axis. This causes displacement of the optical center during rotation around the axis. While the stepper motor rotates according to a specific angle α2, system errors mean the actual camera rotation angle is not equal to α2. Therefore, it is necessary to reconstruct the camera rotation matrix R using corresponding points. ij Translation matrix T ij .

[0057] (1) Convert the coordinates of the corresponding point pairs to coordinates in the camera coordinate system. The calculation method is shown in Equation 1, where Z... ik Let X be the depth value at the position corresponding to the k-th pair of corresponding points in the depth map Idepthi acquired in the i-th pose. ik Y ik Let u represent the coordinates on the X and Y axes of the camera coordinate system after the transformation of the k-th pair of corresponding points in the i-th pose, respectively. ik v ik Let x, y, fx, and fy represent the coordinates on the u and v axes of the k-th pair of corresponding image coordinate systems in the i-th pose, respectively. cx, cy, fx, and fy represent the camera's intrinsic parameters, which are obtained through calibration.

[0058]

[0059] (2) Calculate the essential matrix using the five-point method, and further recover the rotation matrix R from the essential matrix. ij Translation matrix T ij The propagation error of the extrinsic parameter matrix is ​​optimized using the bundle adjustment method, and the camera rotation matrix R for each attitude is obtained based on the first attitude. i Translation matrix T i .

[0060] 7. Optical Center Alignment. Since the optical center is not on the rotation axis, the optical centers of the camera coordinate systems in different poses are not at the same point. Therefore, alignment is necessary to effectively prevent data misalignment during panoramic image generation. Here, optical center alignment is performed using the recovered translation matrix Ti.

[0061] (1) Color Image Conversion. A new color image Icolori' is generated by direct sampling using a translation matrix, as shown in Equation 2. In the equation, u' and v' are the image coordinates (u, v) of the color image Icolori after the translation matrix transformation on the image Icolori'. (X', Y', Z') are the coordinates of the corresponding point (u', v') in the camera coordinate system of Icolori'. These coordinates are obtained from the coordinates (X, Y, Z) in the camera coordinate system of Icolori, which can be obtained using Equation 1. The three-channel grayscale values ​​at the (u, v) coordinates are assigned to the pixels at the (u', v') coordinates of Icolori' to obtain the converted color image Icolori'.

[0062]

[0063] (2) Depth map conversion. The depth map is converted in the same way as the color map, resulting in a converted depth map Idepthi'. The point cloud set Vcloudi' is generated by storing (X',Y',Z',r,g,b), where r, g, and b refer to the grayscale values ​​of the three color channels at the corresponding positions in the color map.

[0064] (3) Pixel Interpolation. Equation 3 shows the indirect sampling form of image conversion, where the variables have the same meaning as in Equation 2. From Equation 3, it can be seen that Δu and Δv affect the values ​​of u and v. Under normal circumstances, the focal lengths of cameras fx and fy are usually 900 pixels, cx and cy are usually 500 pixels, and Z is usually 1000-3000 mm indoors. Considering lens selection, the distance of optical center movement is usually less than 2 mm, so Δu and Δv are no greater than 1 pixel. Therefore, the number of blank pixels in the converted image obtained by direct sampling is very small, and there must be effective pixels around the blank pixels. Therefore, the blank pixels in Icolori' and Idepthi' can be assigned values ​​by region interpolation.

[0065]

[0066] 8. Data Fusion. After the optical centers coincide, the data at each attitude only have rotational bias, so a cylindrical model is used for panoramic data fusion.

[0067] (1) Based on the rotation matrix R i Complete the conversion of the planar projection image Icolori' to the cylindrical projection image Icolor" for each posture.

[0068] (2) Take the maximum and minimum x-axis values ​​umin and umax of each pair of feature points to form a fusion region. Within the fusion region, search for pixel points pr row by row using dynamic programming, and string the points of each row together to form a stitching line li, stitching adjacent images together. According to the principle of dynamic programming, the difference in grayscale values ​​between adjacent images on the stitching line li is minimized. Therefore, it can be guaranteed that there is no misalignment in the stitched panoramic image.

[0069] (3) Since the depth map and the color map are pixel aligned, the depth map is stitched according to the stitching line. Since the depth map and the point cloud set are in one-to-one correspondence, the X, Y, and Z components in the point cloud set are converted into coordinates in the reference coordinate system according to the form P' = RP. Then, redundant point clouds are removed according to the stitching line to complete the point cloud fusion.

[0070] 9. Data Display. After data fusion is completed, two channels are provided for users to choose from: a two-dimensional panoramic image channel, which can display a color panoramic image under the cylindrical projection model; and a full-view cloud image channel, which can display a full-view cloud model composed of point clouds and containing grayscale information, i.e., a depth panoramic image.

[0071] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0072] Embodiments of the present invention also relate to a structured light camera device, such as... Figure 4 As shown, it includes:

[0073] The structured light camera 400 and the stepper motor 401 located at the bottom of the structured light camera 400 are provided with an infrared lens 402, a color lens 403 and a speckle projector 404.

[0074] Stepper motor 401 is used to drive structured light camera 400 to rotate around a specified central axis to multiple preset angles;

[0075] Infrared lens 402 and structured light camera 400 are used to acquire depth maps and color maps when rotating around a specified central axis to multiple preset angles. The depth maps acquired at multiple preset angles are converted from planar projection maps to cylindrical projection maps, and adjacent cylindrical projection maps after conversion are stitched together to obtain a depth panoramic map.

[0076] The structured light camera 400 is also used to correct errors in depth maps using color images;

[0077] During the rotation of the structured light camera 400, the optical center of the infrared lens 402, the optical center of the color lens 403, and the projection point of the speckle projector 404 of the structured light camera 400 are always located at a fixed position on the central axis and rotate synchronously around the central axis.

[0078] The structured light camera device in this embodiment is used to execute any of the above data fusion methods. Therefore, this embodiment corresponds to the above embodiments and can be implemented in conjunction with them. The relevant technical details mentioned in the above embodiments remain valid in this embodiment, and will not be repeated here to reduce repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0079] Embodiments of the present invention also relate to an electronic device, such as... Figure 5 As shown, it includes: at least one processor 501; and a memory 502 communicatively connected to the at least one processor; wherein the memory 502 stores instructions executable by the at least one processor 501, the instructions being executed by the at least one processor 501 to perform the method of any of the above embodiments.

[0080] The memory 502 and processor 501 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 501 and memory 502 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Information processed by processor 501 is transmitted over a wireless medium via an antenna, which further receives information and transmits it to processor 501.

[0081] Processor 501 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 502 can be used to store information used by the processor during operation.

[0082] Embodiments of the present invention relate to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the above-described method embodiments.

[0083] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A data fusion method, characterized in that, include: The depth map and color map are acquired when the structured light camera rotates around a specified central axis to multiple preset angles; wherein, during the rotation of the structured light camera, the optical center of the infrared lens, the optical center of the color lens, and the projection point of the speckle projector of the structured light camera are always located at a fixed position on the central axis and rotate synchronously around the central axis. Based on the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth image, the translation matrix of the structured light camera is obtained; wherein, the adjacent images are captured at adjacent preset angles during the rotation of the structured light camera, and the translation matrix represents the translation relationship between the pose of each image in the color image and the depth image and the reference pose, the reference pose is selected from multiple poses of the structured light camera, and the multiple poses correspond to the multiple preset angles; based on the translation matrix, the depth image is subjected to optical center alignment processing to align the optical centers of the camera coordinate system under each pose; The depth map after optical center alignment is filled with blank areas by interpolation; wherein, the blank areas are the areas of the depth map without captured content caused by the movement of the optical center during the optical center alignment process. Based on the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth image, the rotation matrix of the structured light camera is obtained; the rotation matrix represents the rotational relationship between the pose of each image in the color image and the depth image and the reference pose; the depth images acquired at the multiple preset angles and filled with interpolated blank areas are converted from planar projection images to cylindrical projection images, and according to the rotation matrix, the adjacent cylindrical projection images after conversion are stitched together to obtain a depth panoramic image; wherein, The translation matrix and the rotation matrix are obtained in the following way: Obtain the internal parameter matrix and external parameter matrix of the infrared lens, the color lens, and the speckle projector; Based on the internal parameter matrix, the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth image are converted from image coordinates to camera coordinates; Based on the transformation results of the camera coordinates, the essential matrix is ​​calculated using the five-point method, and the difference rotation matrix and difference translation matrix between adjacent poses are recovered based on the essential matrix. The propagation error of the external parameter matrix is ​​optimized by the bundle adjustment method. Based on the first attitude, the rotation matrix and translation matrix of each attitude are obtained.

2. The data fusion method according to claim 1, characterized in that, The structured light camera is equipped with a stepper motor at its bottom. The process of acquiring depth maps obtained when the structured light camera rotates around a specified central axis to multiple preset angles includes: The stepper motor controls the structured light camera to rotate around the central axis to the multiple preset angles, and at the preset angle positions, the structured light camera is controlled to acquire the depth map.

3. The data fusion method according to claim 2, characterized in that, The process of converting the depth maps acquired at the multiple preset angles from planar projection to cylindrical projection, and then stitching together adjacent cylindrical projections to obtain a panoramic depth image, includes: The depth maps obtained from the multiple preset angles and after error correction are converted from planar projection maps to cylindrical projection maps, and the adjacent cylindrical projection maps after conversion are stitched together to obtain a panoramic depth map.

4. The data fusion method according to claim 1, characterized in that, The step of stitching together adjacent cylindrical projection images after transformation according to the rotation matrix to obtain a depth panoramic image includes: Based on the rotation matrix and the corresponding point pairs of adjacent images in the depth map, the region to be fused in the depth map is obtained; Within the area to be merged, the point with the smallest grayscale difference is searched using dynamic programming; A splicing line is formed based on the point with the smallest grayscale difference; The depth map projected onto the cylindrical surface is stitched together according to the stitching line.

5. The data fusion method according to claim 1, characterized in that, The conversion from image coordinates to camera coordinates is achieved through the following expression: Where Z ik Let X be the depth value of the k-th pair of corresponding points in the depth map under the i-th pose. ik Y ik Let u represent the X and Y coordinates of the k-th pair of corresponding points in the depth map under the i-th pose, respectively, after transformation in the camera coordinate system. ik v ik Let cx, cy, fx, and fy represent the coordinates of the k-th pair of speckle pairs with the same name in the depth map under the i-th pose on the u and v axes of the image coordinate system, respectively. cx, cy, fx, and fy are obtained from the internal parameter matrix. cx and cy are the translation dimensions of the origin of the coordinate system during the transformation from the image coordinate system to the camera coordinate system. fx and fy are the quotients of the image distance and the physical length of the pixel during the transformation from the image coordinate system to the camera coordinate system.

6. The data fusion method according to claim 5, characterized in that, The optical center alignment process is achieved through the following expression: Where (u', v') are the coordinates of the point (u, v) in the color image after the translation matrix transformation, and (X', Y', Z') are the coordinates of the point corresponding to (u', v') in the depth image after the image coordinate system is transformed into the camera coordinate system. Tx, Ty, and Tz are the coordinate changes of the X, Y, and Z axes during the translation matrix transformation, respectively.

7. A structured light camera device for performing the data fusion method as described in any one of claims 1-6, characterized in that, include: A structured light camera and a stepper motor located at the bottom of the structured light camera; the structured light camera is equipped with an infrared lens, a color lens and a speckle projector. The stepper motor is used to drive the structured light camera to rotate around a specified central axis to multiple preset angles; The structured light camera is used to acquire depth maps and color maps when rotating around a specified central axis to multiple preset angles. The depth maps acquired at the multiple preset angles and filled with the blank area are converted from planar projection maps to cylindrical projection maps. According to the rotation matrix, the adjacent cylindrical projection maps after conversion are stitched together to obtain a depth panoramic map. The structured light camera is also used to perform error correction on the depth map using the color image; The structured light camera is further configured to obtain a translation matrix of the structured light camera based on corresponding point pairs of adjacent images in the color image and corresponding point pairs of adjacent images in the depth image; wherein, the adjacent images are captured at adjacent preset angles during the rotation of the structured light camera, and the translation matrix represents the translation relationship between the pose of each image in the color image and the depth image and a reference pose, wherein the reference pose is selected from multiple poses of the structured light camera, and the multiple poses correspond to the multiple preset angles; based on the translation matrix, the depth image is subjected to optical center alignment processing to align the optical centers of the camera coordinate systems under each pose, and blank areas are interpolated and filled in the optical center aligned depth image; wherein, the blank areas are areas of the depth image without captured content caused by the movement of the optical center during the optical center alignment processing; The structured light camera is further specifically used to obtain the rotation matrix of the structured light camera based on the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth image; the rotation matrix represents the rotation relationship between the pose of each image in the color image and the depth image and the reference pose; During the rotation of the structured light camera, the optical center of the infrared lens, the optical center of the color lens, and the projection point of the speckle projector are always located at a fixed position on the central axis and rotate synchronously around the central axis. The translation matrix and the rotation matrix are obtained in the following ways: Obtain the internal parameter matrix and external parameter matrix of the infrared lens, the color lens, and the speckle projector; Based on the internal parameter matrix, the corresponding point pairs of adjacent images in the color image and the corresponding point pairs of adjacent images in the depth image are converted from image coordinates to camera coordinates; Based on the transformation results of the camera coordinates, the essential matrix is ​​calculated using the five-point method, and the difference rotation matrix and difference translation matrix between adjacent poses are recovered based on the essential matrix. The propagation error of the external parameter matrix is ​​optimized by the bundle adjustment method. Based on the first attitude, the rotation matrix and translation matrix of each attitude are obtained.

8. An electronic device, characterized in that, include: At least one processor; A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the data fusion method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data fusion method as described in any one of claims 1 to 6.

Images