Three-dimensional map reconstruction method and system in virtual reality
By reconstructing the point cloud data set into images and fusing the color features obtained by shooting, the problem of inconspicuous point cloud data characteristics and lighting effects in three-dimensional map reconstruction in virtual reality is solved, and feature recognition accuracy and point cloud data are improved Accuracy of position recognition.
Patent Information
- Application Number
- CN202411545686.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In virtual reality, the three-dimensional map reconstruction has poor effect in scenarios where point cloud data characteristics are not obvious, and the unstable lighting conditions affect the accuracy of feature recognition, resulting in poor feature alignment effects in scenarios such as mining goaf modeling.
The point cloud data set is reconstructed into an image, and fuses with the color features of the image to be built as a photographed, and processes it through the image reconstruction model and feature classification model to construct a monitoring time point conversion matrix to realize the registration and registration of point cloud data. Fusion to improve feature recognition accuracy.
The problem of poor feature alignment effect of point cloud data set is avoided, the impact of light instability on feature recognition is reduced, the accuracy of feature recognition in the same area is improved, and the accuracy of point cloud data location recognition is ensured.
Smart Images

Figure CN119048697A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of map reconstruction, and in particular to a method and system for reconstructing a three-dimensional map in virtual reality. Background Art
[0002] In the development of virtual reality technology, 3D map reconstruction is one of the key technologies. Its core lies in extracting feature points from multi-view images or point cloud data sets, and aligning point clouds through these feature points to build a high-precision 3D model. However, in some scenarios where the features of point cloud data are not obvious, such as modeling of mine goafs, the effect of aligning point clouds through features in point cloud data sets is poor; and identifying features by taking pictures will be affected by unstable lighting conditions, resulting in low feature recognition accuracy. Summary of the invention
[0003] The present invention reconstructs the point cloud dataset into an image, which can provide information such as the size and texture of the corresponding area, and then merges it with the color and other features of the image to be built obtained by shooting. This can avoid the problem of poor effect of directly aligning the point cloud through the features of the point cloud dataset, and at the same time avoid the influence of unstable lighting on the feature recognition of the image to be built during the shooting process, thereby improving the accuracy of feature recognition for the same area and ensuring the accuracy of corresponding position recognition of subsequent point cloud data.
[0004] A method for reconstructing a three-dimensional map in virtual reality, comprising: At the monitoring time point, several groups of images to be built and point cloud data sets to be built are obtained through high-definition cameras and lidar respectively; The position of the laser radar at the first monitoring time point is recorded as the origin of the map coordinate system, and the monitoring time points are arranged in chronological order. The actual posture information corresponding to the latter monitoring time point is constructed through several groups of to-be-built images and to-be-built point cloud data sets corresponding to two adjacent monitoring time points; All point cloud data sets corresponding to each monitoring time point are registered using the actual posture information corresponding to the monitoring time point, that is, the registered point cloud data set; All registered point cloud data sets corresponding to all monitoring time points are fused to construct the point cloud data set to be constructed; Perform 3D modeling operations based on the point cloud dataset to be constructed and map it to the VR model to obtain a virtual reality 3D map; The actual posture information corresponding to the latter monitoring time point is constructed by using a plurality of groups of to-be-built images and to-be-built point cloud data sets corresponding to two adjacent monitoring time points, including the following steps: For each set of pictures to be constructed and point cloud data sets to be constructed, the following operations are performed: the point cloud data sets to be constructed are sent to the image reconstruction model for processing to construct a reconstructed picture; the reconstructed picture and the picture to be constructed are sent to the image feature classification model for processing to obtain an enhanced image, image feature labels and the rectangular frame in which they are located. The enhanced image is obtained by processing the reconstructed picture and the picture to be constructed through the self-attention mechanism; Construct a monitoring time point conversion matrix based on the enhanced images, image feature labels and rectangular frames corresponding to two adjacent monitoring time points; Based on the monitoring time point transformation matrix, the actual posture information corresponding to the previous monitoring time point is transformed into coordinates to obtain the actual posture information corresponding to the subsequent monitoring time point, that is, the actual posture information corresponding to the latter monitoring time point; and when the position of the laser radar at the first monitoring time point is used as the actual posture information corresponding to the previous monitoring time point, the actual posture information corresponding to the previous monitoring time point is the origin coordinate.
[0005] As a preferred aspect, a monitoring time point conversion matrix is constructed based on the enhanced images, image feature labels and rectangular frames corresponding to two adjacent monitoring time points, specifically including the following steps: The two adjacent monitoring time points are recorded as the front monitoring time point and the back monitoring time point in chronological order, and the enhanced image, image feature label and rectangular frame corresponding to each group of to-be-built pictures and to-be-built point cloud data sets in the front monitoring time point are recorded as the first enhanced image FP i , the first image feature label FT i and the first rectangular frame FH i , i=1, 2, 3, ..., I, I is the total number of groups corresponding to the pictures to be built and the point cloud data set to be built; the enhanced image, image feature label and rectangular box corresponding to each group of pictures to be built and the point cloud data set to be built at the post-monitoring time point are respectively recorded as the second enhanced image BP i , the second image feature label BT i and the second rectangular frame BH i ; Traverse the first image feature label FT i , for each selected first image feature label FT i , perform the following operations, from all the second image feature labels BT i Select the second image feature label BT in turn i With the first image feature label FT selected i Matching, if the selected second image feature label BT i With the first image feature label FT selected i If the first image feature label FT is selected, i The corresponding first rectangular frame FH iMapped to the first enhanced image FP corresponding to the previous monitoring time point i In the example, the first feature image is segmented and the selected second image feature label BT is i The corresponding second rectangular frame BH i Mapped to the second enhanced image BP corresponding to the later monitoring time point i In the example, the second feature image is segmented, the smaller one of the first feature image and the second feature image is selected as the feature image to be processed, and the larger one of the first feature image and the second feature image is recorded as the reference feature image. The feature image to be processed is enlarged by the bicubic interpolation method to obtain the processed feature image, and then the reference feature image and the processed feature image are similarity calculated to obtain the feature similarity, and it is determined whether the feature similarity is greater than the similarity threshold. If the feature similarity is greater than the similarity threshold, the first feature image is back-projected to the selected first image feature label FT i In the corresponding point cloud dataset to be built, the first point cloud dataset is constructed, and the second feature image is back-projected to the selected second image feature label BT i In the corresponding point cloud dataset to be built, a second point cloud dataset is constructed, and point cloud registration operation is performed based on the first point cloud dataset and the second point cloud dataset to construct a local monitoring time point transformation matrix; if the feature similarity is greater than the similarity threshold, continue to obtain the feature labels BT from all the second images. i Select the second image feature label BT i With the first image feature label FT selected i Matching; if the selected second image feature label BT i With the first image feature label FT selected i Consistent, continue directly from all second image feature labels BT i Select the second image feature label BT i With the first image feature label FT selected i Make a match; All local monitoring time point conversion matrices are obtained, and all local monitoring time point conversion matrices are added and averaged to obtain the monitoring time point conversion matrix.
[0006] As a preferred aspect, the image reconstruction model includes a point cloud feature extraction layer and an image reconstruction layer, wherein the point cloud feature extraction layer is established based on the PointNet model, and is used to extract point cloud features in the point cloud data set to be constructed to construct a point cloud feature vector; the image reconstruction layer is established based on the decoder in the U-net, and is used to reconstruct the image based on the point cloud feature vector to construct a reconstructed picture.
[0007] As a preferred aspect, training the image reconstruction model specifically includes the following steps: A number of image reconstruction training samples are obtained, and the image reconstruction training samples include a point cloud data set and its corresponding grayscale image; all the image reconstruction training samples are combined into an image reconstruction training set, and the image reconstruction training set is sent to a parameter-initialized image reconstruction model for training. During the training, the point cloud data set in the image reconstruction training samples is used as input, and the grayscale image in the image reconstruction training samples is used as the target output. The accuracy corresponding to the image reconstruction training samples is calculated, and it is determined whether the accuracy corresponding to the image reconstruction training samples is greater than a preset threshold. If the accuracy corresponding to the image reconstruction training samples is greater than the preset threshold, the trained image reconstruction model is output; otherwise, the image reconstruction model is continuously trained through the image reconstruction training set.
[0008] As a preferred aspect, the image feature classification model includes a self-attention mechanism layer and an image feature classification layer, wherein the self-attention mechanism layer is used to perform self-attention mechanism processing on the reconstructed image and the image to be constructed to construct an enhanced image; the image feature classification layer is established based on the Faster-RCNN model, and is used to perform image feature classification operations based on the enhanced image to construct image feature labels and rectangular boxes; When performing self-attention mechanism processing on the reconstructed image and the image to be constructed, the key matrix K and the value matrix V are constructed based on the image to be constructed, and the query matrix Q is constructed based on the reconstructed image.
[0009] As a preferred aspect, training the image feature classification model specifically includes the following steps: A number of image classification samples are obtained, and the image classification samples include a point cloud data set and its corresponding training pictures; the point cloud data set in the image classification samples is processed by an image reconstruction model to obtain a training reconstructed picture, and an image classification training sample is constructed, and the image classification training sample includes a training picture and its corresponding training reconstructed picture, and the training pictures in the image classification training samples are labeled by image feature labels and the rectangular frames in which they are located, and all labeled image classification training samples are formed into an image classification training set, and the image classification training set is sent to an image feature classification model with parameter initialization for training. During the training, a loss value is calculated to determine whether the loss value is within a preset range. If the loss value is within the preset range, the trained image feature classification model is output; otherwise, the image feature classification model is continuously trained through the image classification training set.
[0010] As a preferred aspect, when calculating the similarity between the reference feature graph and the processed feature graph, the similarity calculation method adopts one of the cosine similarity algorithm or the structural similarity measurement algorithm.
[0011] A three-dimensional map reconstruction system in virtual reality, comprising: The module for collecting pictures to be built is used to obtain pictures to be built through a high-definition camera; The point cloud data set acquisition module to be built is used to acquire the point cloud data set to be built through laser radar; The actual posture information calculation module is used to construct the actual posture information corresponding to the latter monitoring time point through a plurality of groups of to-be-built pictures and to-be-built point cloud data sets corresponding to two adjacent monitoring time points; The point cloud registration module is used to register all point cloud data sets corresponding to each monitoring time point through the actual posture information corresponding to the monitoring time point, that is, the registered point cloud data set; Point cloud fusion module, used to perform point cloud fusion on all registered point cloud data sets corresponding to all monitoring time points to construct the point cloud data set to be constructed; The virtual reality three-dimensional map construction module is used to perform three-dimensional modeling operations based on the point cloud data set to be constructed, and map it to the VR model to obtain a virtual reality three-dimensional map.
[0012] The present invention has the following advantages: The present invention reconstructs the point cloud dataset into an image, which can provide information such as the size and texture of the corresponding area, and then fuses it with the color and other features of the image to be built obtained by shooting. This can avoid the problem of poor effect of directly aligning the point cloud through the features of the point cloud dataset, and at the same time avoid the influence of unstable lighting on the feature recognition of the image to be built during the shooting process, thereby improving the accuracy of feature recognition for the same area and ensuring the accuracy of corresponding position recognition of subsequent point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a structural schematic diagram of a three-dimensional map reconstruction system in virtual reality adopted in an embodiment of the present invention. DETAILED DESCRIPTION
[0014] In order to enable persons skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0015] Embodiment 1, a method for reconstructing a three-dimensional map in virtual reality, comprising: At the monitoring time point, a number of groups of pictures to be built and point cloud data sets to be built are obtained through the high-definition camera and the laser radar respectively. It should be noted that in the process of reconstructing the three-dimensional map, the robot is equipped with a high-definition camera and a laser radar to collect data of the actual scene, and the high-definition camera and the laser radar are at the same position on the same platform of the robot, the optical axes are parallel, and the scanning angle of the laser radar is set according to the camera parameters of the high-definition camera, so that the scanning range of the laser radar is consistent with the shooting range of the high-definition camera; and in the process of the robot performing the data acquisition operation, the operator generally sets a preset path for the robot. In the process of driving along the preset path, the robot will stop at the monitoring time point and obtain the pictures to be built and the point cloud data sets to be built, and it is foreseeable that in the process of obtaining each group of pictures to be built and the point cloud data sets to be built, the high-definition camera and the laser radar will rotate several angles, so as to achieve the effect of complete collection of the same area, therefore, there will also be several groups of pictures to be built and point cloud data sets to be built; The position of the laser radar at the first monitoring time point is recorded as the origin of the map coordinate system, and the monitoring time points are arranged in chronological order. The actual posture information corresponding to the latter monitoring time point is constructed through several groups of to-be-built pictures and to-be-built point cloud data sets corresponding to two adjacent monitoring time points. It should be noted that the point cloud data obtained by the laser radar are all based on the coordinates of the laser radar with the origin. In order to construct a three-dimensional map, it is necessary to map all point cloud data to the same coordinate system, and the actual posture information corresponding to the monitoring time point is the bridge for mapping all point cloud data to the same coordinate system. The actual posture information corresponding to the monitoring time point here refers to the coordinates of the position of the laser radar at the monitoring time point on the map coordinate system. If the point cloud is aligned directly by the features of the point cloud data set, since the point cloud data set can only provide information such as size and texture, in cases such as mining area modeling, different features cannot be well distinguished by information such as size and texture, resulting in poor effect of aligning the point cloud directly by the features of the point cloud data set. Therefore, the present application integrates feature information such as the color of the taken photos. All point cloud data sets corresponding to each monitoring time point are registered by the actual posture information corresponding to the monitoring time point, that is, the registered point cloud data set; that is, the coordinate transformation matrix is calculated based on the actual posture information corresponding to the monitoring time point compared with the origin, and then each point cloud data in the point cloud data set is calculated with the transformation matrix to obtain the registered point cloud data in the registered point cloud data set; All registered point cloud data sets corresponding to all monitoring time points are point cloud fused to construct the point cloud data set to be constructed. The point cloud fusion adopts existing technology, such as iterative closest point algorithm, which will not be described in detail here. Perform 3D modeling operations based on the point cloud dataset to be constructed and map it to the VR model to obtain a virtual reality 3D map, thereby realizing the reconstruction of the 3D map in the virtual reality; The actual posture information corresponding to the latter monitoring time point is constructed by using a plurality of groups of to-be-built images and to-be-built point cloud data sets corresponding to two adjacent monitoring time points, including the following steps: For each group of pictures to be built and point cloud data sets to be built, perform the following operations: send the point cloud data sets to be built into the image reconstruction model for processing to construct a reconstructed picture; the point cloud data sets to be built can be mapped to a two-dimensional image through the image reconstruction model to provide a reinforcement effect for subsequent image feature extraction, and the reconstructed picture and the picture to be built are sent to the image feature classification model for processing to obtain an enhanced image, image feature labels and the rectangular frame in which they are located. The enhanced image is obtained by processing the reconstructed picture and the picture to be built through the self-attention mechanism. The image feature classification model can output the corresponding image feature labels in the picture to be built. In this embodiment, the three-dimensional map reconstruction in virtual reality is applied to mining area modeling, so the image feature labels can be For ore type, if it is applied to other fields, the developer can set the corresponding image feature label by himself. The rectangular box refers to the rectangular box corresponding to the position of the image feature label in the image to be built; the reconstructed image is reconstructed from the point cloud data set to be built, which can reflect the size and texture information of the corresponding area of the image to be built, while the image to be built is taken, which can reflect the color characteristics and other information of the corresponding area of the image to be built. Through the self-attention mechanism, the size and texture information corresponding to the reconstructed image can be fused with the color characteristics and other information to enhance the feature information of the image, avoid the influence of unstable lighting on the feature recognition of the image to be built during the shooting process, and improve the accuracy of the output of the image feature label; Construct a monitoring time point conversion matrix based on the enhanced images, image feature labels and rectangular frames corresponding to two adjacent monitoring time points; Based on the monitoring time point transformation matrix, the actual posture information corresponding to the previous monitoring time point is transformed into coordinates to obtain the actual posture information corresponding to the subsequent monitoring time point, that is, the actual posture information corresponding to the latter monitoring time point; and when the position of the laser radar at the first monitoring time point is used as the actual posture information corresponding to the previous monitoring time point, the actual posture information corresponding to the previous monitoring time point is the origin coordinate.
[0016] The monitoring time point conversion matrix is constructed based on the enhanced images, image feature labels and rectangular frames corresponding to two adjacent monitoring time points, specifically including the following steps: The two adjacent monitoring time points are recorded as the front monitoring time point and the back monitoring time point in chronological order, and the enhanced image, image feature label and rectangular frame corresponding to each group of to-be-built pictures and to-be-built point cloud data sets in the front monitoring time point are recorded as the first enhanced image FP i, the first image feature label FT i and the first rectangular frame FH i , i=1, 2, 3, ..., I, I is the total number of groups corresponding to the pictures to be built and the point cloud data set to be built; the enhanced image, image feature label and rectangular box corresponding to each group of pictures to be built and the point cloud data set to be built at the post-monitoring time point are respectively recorded as the second enhanced image BP i , the second image feature label BT i and the second rectangular frame BH i ; Traverse the first image feature label FT i , for each selected first image feature label FT i , perform the following operations, from all the second image feature labels BT i Select the second image feature label BT in turn i With the first image feature label FT selected i Matching, if the selected second image feature label BT i With the first image feature label FT selected i If the first image feature label FT is selected, i The corresponding first rectangular frame FH i Mapped to the first enhanced image FP corresponding to the previous monitoring time point i In the example, the first feature image is segmented and the selected second image feature label BT is i The corresponding second rectangular frame BH i Mapped to the second enhanced image BP corresponding to the later monitoring time point i In the example, the second feature image is segmented out, and the smaller one of the first feature image and the second feature image is selected as the feature image to be processed, and the larger one of the first feature image and the second feature image is recorded as the reference feature image. The feature image to be processed is enlarged by the bicubic interpolation method to obtain the processed feature image, and then the reference feature image and the processed feature image are similarly calculated. The similarity calculation method can be a cosine similarity algorithm or a structural similarity measurement algorithm to obtain feature similarity, and determine whether the feature similarity is greater than the similarity threshold. The similarity threshold is determined by the developer and is generally 0.85. If the feature similarity is greater than the similarity threshold, it means that the features corresponding to the first feature image and the second feature image are at the same position in the actual scene, and the corresponding point cloud data sets should also refer to the same position. The first feature image is back-projected to the selected first image feature label FT i In the corresponding point cloud dataset to be built, the first point cloud dataset is constructed, and the second feature image is back-projected to the selected second image feature label BT iIn the corresponding point cloud data set to be built, a second point cloud data set is constructed, and a point cloud registration operation is performed based on the first point cloud data set and the second point cloud data set to construct a local monitoring time point conversion matrix. The point cloud registration operation can be specifically to calculate the FPFH features corresponding to different point cloud data for the first point cloud data set and the second point cloud data set, and to register the point cloud data with consistent FPFH features in the first point cloud data set and the second point cloud data set based on the RANSAC algorithm to obtain the corresponding local monitoring time point conversion matrix; if the feature similarity is greater than the similarity threshold, it means that the features corresponding to the first feature image and the second feature image are not in the same position in the actual scene, and continue to obtain the feature labels BT from all the second images. i Select the second image feature label BT i With the first image feature label FT selected i Matching; if the selected second image feature label BT i With the first image feature label FT selected i Consistent, continue directly from all second image feature labels BT i Select the second image feature label BT i With the first image feature label FT selected i Make a match; All local monitoring time point conversion matrices are obtained, and all local monitoring time point conversion matrices are added and averaged to obtain the monitoring time point conversion matrix.
[0017] This application reconstructs the point cloud dataset into an image, which can provide information such as the size and texture of the corresponding area, and then integrates it with the color and other features of the image to be built obtained by shooting. It can avoid the problem of poor effect of directly aligning the point cloud through the features of the point cloud dataset, and at the same time avoid the influence of unstable lighting on the feature recognition of the image to be built during the shooting process, improve the accuracy of feature recognition for the same area, and ensure the accuracy of corresponding position recognition of subsequent point cloud data.
[0018] The image reconstruction model includes a point cloud feature extraction layer and an image reconstruction layer. The point cloud feature extraction layer is established based on the PointNet model and is used to extract point cloud features in the point cloud data set to be built to construct a point cloud feature vector. The image reconstruction layer is established based on the decoder in U-net and is used to reconstruct images based on point cloud feature vectors to construct a reconstructed picture.
[0019] Training the image reconstruction model includes the following steps: A number of image reconstruction training samples are obtained. The image reconstruction training samples include a point cloud dataset and its corresponding grayscale image. It should be noted that the point cloud dataset and its corresponding grayscale image are obtained by a high-definition camera and a laser radar set at the same position, and the scanning range of the laser radar is consistent with the shooting range of the high-definition camera, and the lighting environment is stable. All image reconstruction training samples are combined into an image reconstruction training set, and the image reconstruction training set is sent to a parameter-initialized image reconstruction model for training. During the training, the point cloud dataset in the image reconstruction training sample is used as input, and the grayscale image in the image reconstruction training sample is used as the target output. The accuracy corresponding to the image reconstruction training sample is calculated, and it is determined whether the accuracy corresponding to the image reconstruction training sample is greater than a preset threshold. The preset threshold is set by the developer according to the actual training situation. If the accuracy corresponding to the image reconstruction training sample is greater than the preset threshold, the trained image reconstruction model is output; otherwise, the image reconstruction model continues to be trained through the image reconstruction training set.
[0020] The image feature classification model includes a self-attention mechanism layer and an image feature classification layer. The self-attention mechanism layer is used to perform self-attention mechanism processing on the reconstructed image and the image to be constructed to construct an enhanced image. Through the self-attention mechanism processing, the size and texture information corresponding to the reconstructed image can be fused with information such as color features; the image feature classification layer is established based on the Faster-RCNN model and is used to perform image feature classification operations based on the enhanced image to construct image feature labels and rectangular boxes; When performing self-attention mechanism processing on the reconstructed image and the image to be reconstructed, the key matrix K and the value matrix V are constructed based on the image to be reconstructed, and the query matrix Q is constructed based on the reconstructed image. The self-attention mechanism processing refers to the Transformer model.
[0021] Training the image feature classification model includes the following steps: A number of image classification samples are obtained, and the image classification samples include a point cloud data set and its corresponding training images. It should be noted that the point cloud data set and its corresponding training images are obtained by a high-definition camera and a laser radar set at the same position, and the scanning range of the laser radar is consistent with the shooting range of the high-definition camera; the point cloud data set in the image classification sample is processed by an image reconstruction model to obtain a training reconstructed image, and an image classification training sample is constructed. The image classification training sample includes a training image and its corresponding training reconstructed image, and the training images in the image classification training sample are annotated by image feature labels and the rectangular frames in which they are located. All annotated image classification training samples are formed into an image classification training set, and the image classification training set is sent to the parameter-initialized image feature classification model for training. During the training, the loss value is calculated to determine whether the loss value is within a preset range. The preset range is determined by the developer. If the loss value is within the preset range, the trained image feature classification model is output; otherwise, the image feature classification model continues to be trained through the image classification training set.
[0022] Embodiment 2, a three-dimensional map reconstruction system in virtual reality, such as Figure 1 As shown, including: The module for collecting pictures to be built is used to obtain pictures to be built through a high-definition camera; The point cloud data set acquisition module to be built is used to acquire the point cloud data set to be built through laser radar; The actual posture information calculation module is used to construct the actual posture information corresponding to the latter monitoring time point through a plurality of groups of to-be-built pictures and to-be-built point cloud data sets corresponding to two adjacent monitoring time points; The point cloud registration module is used to register all point cloud data sets corresponding to each monitoring time point through the actual posture information corresponding to the monitoring time point, that is, to obtain the registered point cloud data set; that is, based on the actual posture information corresponding to the monitoring time point, the coordinate transformation matrix is calculated compared with the origin, and then each point cloud data in the point cloud data set is calculated with the transformation matrix to obtain the registered point cloud data in the registered point cloud data set; Point cloud fusion module, used to perform point cloud fusion on all registered point cloud data sets corresponding to all monitoring time points to construct the point cloud data set to be constructed; The virtual reality three-dimensional map construction module is used to perform three-dimensional modeling operations based on the point cloud data set to be constructed, and map it to the VR model to obtain a virtual reality three-dimensional map, thereby realizing the reconstruction of the three-dimensional map in virtual reality.
[0023] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention. Parts not described in detail in this specification belong to the prior art known to those skilled in the art.
Claims
1. A method for reconstructing a three-dimensional map in virtual reality, characterized in that: include: At the monitoring time point, several groups of images to be built and point cloud data sets to be built are obtained through high-definition cameras and lidar respectively; The position of the laser radar at the first monitoring time point is recorded as the origin of the map coordinate system, and the monitoring time points are arranged in chronological order. The actual posture information corresponding to the latter monitoring time point is constructed through several groups of to-be-built images and to-be-built point cloud data sets corresponding to two adjacent monitoring time points; All point cloud data sets corresponding to each monitoring time point are registered using the actual posture information corresponding to the monitoring time point, that is, the registered point cloud data set; All registered point cloud data sets corresponding to all monitoring time points are fused to construct the point cloud data set to be constructed; Perform 3D modeling operations based on the point cloud dataset to be constructed and map it to the VR model to obtain a virtual reality 3D map; The actual posture information corresponding to the latter monitoring time point is constructed by using a plurality of groups of to-be-built images and to-be-built point cloud data sets corresponding to two adjacent monitoring time points, including the following steps: For each set of pictures to be constructed and point cloud data sets to be constructed, the following operations are performed: the point cloud data sets to be constructed are sent to the image reconstruction model for processing to construct a reconstructed picture; the reconstructed picture and the picture to be constructed are sent to the image feature classification model for processing to obtain an enhanced image, image feature labels and the rectangular frame in which they are located. The enhanced image is obtained by processing the reconstructed picture and the picture to be constructed through the self-attention mechanism; Construct a monitoring time point conversion matrix based on the enhanced images, image feature labels and rectangular frames corresponding to two adjacent monitoring time points; Based on the monitoring time point transformation matrix, the actual posture information corresponding to the previous monitoring time point is subjected to coordinate transformation operation to obtain the actual posture information corresponding to the subsequent monitoring time point, that is, the actual posture information corresponding to the latter monitoring time point; and when the position of the laser radar at the first monitoring time point is used as the actual posture information corresponding to the previous monitoring time point, the actual posture information corresponding to the previous monitoring time point is the origin coordinate.
2. A method for reconstructing a three-dimensional map in virtual reality according to claim 1, characterized in that: The monitoring time point conversion matrix is constructed based on the enhanced images, image feature labels and rectangular frames corresponding to two adjacent monitoring time points, specifically including the following steps: The two adjacent monitoring time points are recorded as the front monitoring time point and the back monitoring time point in chronological order, and the enhanced image, image feature label and rectangular frame corresponding to each group of to-be-built pictures and to-be-built point cloud data sets in the front monitoring time point are recorded as the first enhanced image FP i , the first image feature label FT i and the first rectangular frame FH i , i=1, 2, 3, …, I, I is the total number of groups corresponding to the images to be built and the point cloud data sets to be built; The enhanced image, image feature label and rectangular frame corresponding to each group of to-be-built pictures and to-be-built point cloud data sets at the post-monitoring time point are respectively recorded as the second enhanced image BP i , the second image feature label BT i and the second rectangular frame BH i ; Traverse the first image feature label FT i , for each selected first image feature label FT i , perform the following operations, from all the second image feature labels BT i Select the second image feature label BT in turn i With the first image feature label FT selected i Matching, if the selected second image feature label BT i With the first image feature label FT selected i If the first image feature label FT is selected i The corresponding first rectangular frame FH i Mapped to the first enhanced image FP corresponding to the previous monitoring time point i In the example, the first feature image is segmented and the selected second image feature label BT is i The corresponding second rectangular frame BH i Mapped to the second enhanced image BP corresponding to the later monitoring time point i In the example, the second feature image is segmented, the smaller one of the first feature image and the second feature image is selected as the feature image to be processed, and the larger one of the first feature image and the second feature image is recorded as the reference feature image. The feature image to be processed is enlarged by the bicubic interpolation method to obtain the processed feature image, and then the reference feature image and the processed feature image are similarity calculated to obtain the feature similarity, and it is determined whether the feature similarity is greater than the similarity threshold. If the feature similarity is greater than the similarity threshold, the first feature image is back-projected to the selected first image feature label FT i In the corresponding point cloud dataset to be built, the first point cloud dataset is constructed, and the second feature image is back-projected to the selected second image feature label BT i In the corresponding point cloud dataset to be built, a second point cloud dataset is constructed, and point cloud registration operation is performed based on the first point cloud dataset and the second point cloud dataset to construct a local monitoring time point transformation matrix; if the feature similarity is greater than the similarity threshold, continue to obtain the feature labels BT from all the second images. i Select the second image feature label BT i With the first image feature label FT selected i Matching; if the second image feature label BT is selected i With the first image feature label FT selected i Consistent, continue directly from all second image feature labels BT i Select the second image feature label BT i With the first image feature label FT selected i Make a match; All local monitoring time point conversion matrices are obtained, and all local monitoring time point conversion matrices are added and averaged to obtain the monitoring time point conversion matrix.
3. A method for reconstructing a three-dimensional map in virtual reality according to claim 2, characterized in that: The image reconstruction model includes a point cloud feature extraction layer and an image reconstruction layer. The point cloud feature extraction layer is established based on the PointNet model and is used to extract point cloud features in the point cloud data set to be built to construct a point cloud feature vector. The image reconstruction layer is established based on the decoder in U-net and is used to reconstruct images based on point cloud feature vectors to construct a reconstructed picture.
4. A method for reconstructing a three-dimensional map in virtual reality according to claim 3, characterized in that: Training the image reconstruction model includes the following steps: A number of image reconstruction training samples are obtained, and the image reconstruction training samples include a point cloud data set and its corresponding grayscale image; all the image reconstruction training samples are combined into an image reconstruction training set, and the image reconstruction training set is sent to a parameter-initialized image reconstruction model for training. During the training, the point cloud data set in the image reconstruction training samples is used as input, and the grayscale image in the image reconstruction training samples is used as the target output. The accuracy corresponding to the image reconstruction training samples is calculated, and it is determined whether the accuracy corresponding to the image reconstruction training samples is greater than a preset threshold. If the accuracy corresponding to the image reconstruction training samples is greater than the preset threshold, the trained image reconstruction model is output; otherwise, the image reconstruction model is continuously trained through the image reconstruction training set.
5. A method for reconstructing a three-dimensional map in virtual reality according to claim 4, characterized in that: The image feature classification model includes a self-attention mechanism layer and an image feature classification layer, wherein the self-attention mechanism layer is used to perform self-attention mechanism processing on the reconstructed image and the image to be constructed to construct an enhanced image; The image feature classification layer is built based on the Faster-RCNN model and is used to perform image feature classification operations based on enhanced images to construct image feature labels and their rectangular boxes. When performing self-attention mechanism processing on the reconstructed image and the image to be constructed, the key matrix K and the value matrix V are constructed based on the image to be constructed, and the query matrix Q is constructed based on the reconstructed image.
6. A method for reconstructing a three-dimensional map in virtual reality according to claim 5, characterized in that: Training the image feature classification model includes the following steps: Acquire several image classification samples, the image classification samples include a point cloud data set and its corresponding training pictures; process the point cloud data set in the image classification samples through an image reconstruction model to obtain a training reconstructed picture, and construct an image classification training sample, the image classification training sample includes a training picture and its corresponding training reconstructed picture, annotate the training pictures in the image classification training samples through image feature labels and the rectangular frames in which they are located, form an image classification training set with all annotated image classification training samples, and send the image classification training set to a parameter-initialized image feature classification model for training, during the training, calculate the loss value, and determine whether the loss value is within a preset range, if the loss value is within the preset range, output the trained image feature classification model; Otherwise, continue to train the image feature classification model using the image classification training set.
7. A method for reconstructing a three-dimensional map in virtual reality according to claim 2, characterized in that: When calculating the similarity between the reference feature map and the processed feature map, the similarity calculation method adopts one of the cosine similarity algorithm or the structural similarity measurement algorithm.
8. A three-dimensional map reconstruction system in virtual reality, characterized in that: The system applies a method for reconstructing a three-dimensional map in virtual reality according to any one of claims 1 to 7, comprising: The module for collecting pictures to be built is used to obtain pictures to be built through a high-definition camera; The point cloud data set acquisition module to be built is used to acquire the point cloud data set to be built through laser radar; The actual posture information calculation module is used to construct the actual posture information corresponding to the latter monitoring time point through a plurality of groups of to-be-built pictures and to-be-built point cloud data sets corresponding to two adjacent monitoring time points; The point cloud registration module is used to register all point cloud data sets corresponding to each monitoring time point through the actual posture information corresponding to the monitoring time point, that is, the registered point cloud data set; Point cloud fusion module, used to perform point cloud fusion on all registered point cloud data sets corresponding to all monitoring time points to construct the point cloud data set to be constructed; The virtual reality three-dimensional map construction module is used to perform three-dimensional modeling operations based on the point cloud data set to be constructed, and map it to the VR model to obtain a virtual reality three-dimensional map.
Citation Information
Patent Citations
Three-dimensional point cloud reconstruction device and method based on multi-fusion sensor
CN110415342A
Three-dimensional point cloud scene segmentation method and system fusing image features
CN114255238A
Track three-dimensional reconstruction method fusing laser radar and image data
CN114332348A
Single image three-dimensional reconstruction method based on RealPoin3D
CN115601498A
Template engineering frame body intelligent identification system based on deep learning
CN117611797A
Cited By
MR equipment virtual-real alignment method and system based on point cloud map
CN120014137A
