Three-dimensional positioning mapping method and device, equipment, storage medium and program product

Abstract

The invention discloses a three-dimensional positioning mapping method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining a first image sequence; the first image sequence comprises multiple frames of images shot by a camera at different continuous moments; performing optical flow decoding, depth decoding and back projection on adjacent frames of images in the first image sequence to obtain a first point cloud image sequence corresponding to the first image sequence; performing point cloud splicing on a plurality of point cloud pictures in the first point cloud picture sequence to obtain a first global point cloud map corresponding to the first point cloud picture sequence; and performing semantic segmentation and target detection based on the first image sequence and the first global point cloud map to obtain a target global semantic point cloud map.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a three-dimensional positioning mapping method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] The environment perception and positioning mapping of related autonomous vehicles mainly rely on multiple sensors to realize three-dimensional reconstruction of the surrounding environment of the vehicle and self pose estimation through multi-source heterogeneous data fusion. However, this method cannot meet the performance requirements in harsh environments, and the time and space synchronization between multiple sensors is difficult, leading to complex data fusion and joint optimization, making it difficult to construct a high-precision dense map, and thus difficult to meet the requirements of large-scale application of autonomous driving for low cost, high precision and high robustness. SUMMARY

[0003] To solve the above technical problems, the embodiments of the present application provide a three-dimensional positioning mapping method, device, equipment, storage medium and program product.

[0004] The three-dimensional positioning mapping method provided by the embodiments of the present application comprises:

[0005] obtaining a first image sequence; the first image sequence comprises multiple frames of images obtained by a camera at continuous different time instants;

[0006] performing optical flow decoding, depth decoding and back projection on adjacent frame images in the first image sequence to obtain a first point cloud map sequence corresponding to the first image sequence;

[0007] performing point cloud stitching on multiple point cloud maps in the first point cloud map sequence to obtain a first global point cloud map corresponding to the first point cloud map sequence;

[0008] performing semantic segmentation and target detection based on the first image sequence and the first global point cloud map to obtain a target global semantic point cloud map.

[0009] The above method further comprises:

[0010] performing pose estimation on the camera based on the first global point cloud map and the first image sequence to obtain a first pose trajectory of the camera in the first global point cloud map;

[0011] determining a second image sequence newly added within a first time step based on the first pose trajectory;

[0012] performing optical flow decoding, depth decoding and back projection on adjacent frame images in the second image sequence to obtain a second point cloud map sequence corresponding to the second image sequence;

[0013] updating the first global point cloud map based on the second point cloud map sequence to obtain a second global point cloud map.

[0014] In the method, the decoding of the optical flow, the decoding of the depth, and the reverse projection of the adjacent frame images in the first image sequence to obtain the first point cloud map sequence corresponding to the first image sequence include:

[0015] For each pair of adjacent frame images in the first image sequence, feature extraction is performed on the each pair of adjacent frame images to obtain a first feature map and a second feature map corresponding to the each pair of adjacent frame images;

[0016] A matching cost volume is constructed based on the first feature map and the second feature map to obtain a matching cost between the first feature map and the second feature map;

[0017] Based on the matching cost, an optical flow field between the each pair of adjacent frame images is calculated, and the first feature map and the second feature map are transformed based on the optical flow field to obtain a third feature map and a fourth feature map corresponding to the each pair of adjacent frame images after image registration;

[0018] The third feature map and the fourth feature map are decoded in depth to obtain first depth information and second depth information corresponding to the each pair of adjacent frame images;

[0019] The first depth information and the second depth information corresponding to the each pair of adjacent frame images are reversely projected to a three-dimensional space to obtain the first point cloud map sequence.

[0020] The first point cloud map corresponding to a t-1 moment and the second point cloud map corresponding to a t moment of the each pair of adjacent frame images, and the first depth information and the second depth information corresponding to the each pair of adjacent frame images include a first confidence map of an image at the t-1 moment and a second confidence map of an image at the t moment, and t is greater than or equal to 1.

[0021] In the method, the point cloud stitching of the multiple point cloud maps in the first point cloud map sequence to obtain the first global point cloud map corresponding to the first point cloud map sequence includes:

[0022] The following steps are iteratively performed until t is equal to the length of the first image sequence to obtain the first global point cloud map:

[0023] The second point cloud map is transformed based on a first pose transformation matrix to obtain a third point cloud map; the first pose transformation matrix is a pose transformation matrix of the second point cloud map in a camera coordinate system at a t-1 moment;

[0024] perform weighted average on the first point cloud map and the third point cloud map based on the first confidence map and the second confidence map to obtain a fourth point cloud map;

[0025] perform point cloud registration on the first point cloud map and the third point cloud map to obtain a second pose transformation matrix; the second pose transformation matrix is the first pose transformation matrix after optimization;

[0026] update the fourth point cloud map based on the second pose transformation matrix, and increase t by 1.

[0027] In the method, the semantic segmentation and target detection based on the first image sequence and the first global point cloud map to obtain a target global semantic point cloud map comprises:

[0028] project the first global point cloud map onto each frame of image in the first image sequence to obtain a depth map corresponding to each frame of image;

[0029] splice each frame of image and the depth map corresponding to each frame of image in the channel dimension to obtain a plurality of combined images;

[0030] perform semantic segmentation on each combined image in the plurality of combined images to obtain a plurality of segmentation results;

[0031] perform target detection on each frame of image to obtain a plurality of first target detection results;

[0032] map each first target detection result in the plurality of first target detection results to a corresponding point cloud map respectively to obtain a plurality of second target detection results;

[0033] map the plurality of segmentation results and the plurality of second target detection results to the first global point cloud map to obtain the target global semantic point cloud map.

[0034] In the method, the updating the first global point cloud map based on the second point cloud sequence to obtain a second global point cloud map comprises:

[0035] set a time stamp for each three-dimensional point in the first global point cloud map, and set a forgetting time threshold for the time stamp;

[0036] for each three-dimensional point, if the time stamp of the three-dimensional point exceeds the forgetting time threshold, eliminate the three-dimensional point in the first global point cloud map, and superimpose the second point cloud sequence to the first global point cloud map to obtain the second global point cloud map.

[0037] The method further comprises:

[0038] extract a target object on the target global semantic point cloud map, and construct a network topology graph based on the target object;

[0039] perform path searching on the network topology graph based on a preset rule to obtain a global reference path;

[0040] slide a preset space-time window around the camera to obtain a point cloud map in a target region;

[0041] perform path planning based on the global reference path and the point cloud map in the target region to obtain a local target path;

[0042] execute the local target path in a second time step, and perform rolling optimization on the local target path based on the preset space-time window.

[0043] The processing device provided in the embodiments of the present application comprises a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any one of the three-dimensional positioning mapping methods.

[0044] The computer readable storage medium provided in the embodiments of the present application is used to store a computer program, and the computer program enables a computer to execute any one of the three-dimensional positioning mapping methods.

[0045] The computer program product provided in the embodiments of the present application comprises computer program instructions, and the computer program instructions enable a computer to execute any one of the three-dimensional positioning mapping methods.

[0046] In the technical solution of the embodiments of the present application, a first image sequence is acquired; the first image sequence comprises multiple frames of images captured by a camera at continuous different time instants; optical flow decoding, depth decoding and reverse projection are performed on adjacent frames of images in the first image sequence to obtain a first point cloud map sequence corresponding to the first image sequence; point cloud splicing is performed on multiple point cloud maps in the first point cloud map sequence to obtain a first global point cloud map corresponding to the first point cloud map sequence; semantic segmentation and target detection are performed based on the first image sequence and the first global point cloud map to obtain a target global semantic point cloud map. In this way, by proposing two branches of optical flow decoding and depth decoding, joint learning of motion estimation and depth estimation of image data can be achieved, which not only reduces the computational complexity, but also combines the complementary advantages of motion and depth data, improves the depth estimation accuracy and robustness, and further, by performing point cloud splicing on point cloud data, point cloud registration can be achieved, a large-scale dense point cloud map with centimeter-level precision can be obtained, and semantic segmentation and target detection further endow the point cloud map with pixel-level scene understanding, so that the point cloud map contains the category and location information of the target object, thereby finally constructing a three-dimensional map containing rich geometric and semantic information. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of a three-dimensional positioning mapping method provided by an embodiment of the present application;

[0048] Figure 2 is a general framework diagram of a monocular camera three-dimensional positioning mapping system based on deep learning provided by an embodiment of the present application;

[0049] Figure 3 is a flowchart of point cloud conversion of a monocular image provided by an embodiment of the present application;

[0050] Figure 4 is a flowchart of point cloud splicing of a point cloud map provided by an embodiment of the present application;

[0051] Figure 5 is a flowchart of semantic segmentation and target detection according to a global point cloud map and an image sequence provided by an embodiment of the present application;

[0052] Figure 6 is a structural diagram of a three-dimensional positioning mapping device provided by an embodiment of the present application;

[0053] Figure 7 is a structural diagram of a processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0055] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any way, and all of them belong to the protection scope of the embodiments of the present application.

[0056] The environment perception and positioning mapping of related autonomous vehicles mainly rely on multi-sensor fusion solutions, that is, combining multiple sensors such as lidar, GPS / IMU, monocular / binary / fish-eye cameras, etc., to realize three-dimensional reconstruction of the vehicle's surrounding environment and self pose estimation through multi-source heterogeneous data fusion. For example, the lidar point cloud positioning mapping method based on lidar point cloud includes: first, according to the global positioning system (GPS) / inertial measurement unit (IMU) pose initial value, the multi-frame laser point cloud is registered into a local map, and then the positioning in the local map is realized through inter-frame matching, and finally the local map is gradually spliced into a global map. For another example, the visual SLAM scheme based on vehicle-mounted multi-camera includes: obtaining the three-dimensional position of the feature points through binocular stereo matching, and then realizing camera pose estimation and sparse three-dimensional reconstruction through feature point tracking and triangulation.

[0057] However, the above method has the following disadvantages:

[0058] (1) The cost of lidar is high, and it is easily affected by bad weather, and its performance decreases significantly in rainy and snowy environments.

[0059] (2) The multi-camera calibration is complex, and it is easy to fail after long-term work, and the feature points are sparse, making it difficult to construct a high-precision dense map.

[0060] (3) It is difficult to synchronize different sensors in time and space, and the data fusion and joint optimization are complex.

[0061] (4) System integration is difficult, and scalability and economy are not good.

[0062] In summary, the above method is difficult to meet the requirements of low cost, high precision and high robustness for large-scale application of autonomous driving, and a new three-dimensional positioning mapping scheme that is simpler, more efficient and has higher cost performance is urgently needed.

[0063] To solve the above technical problems, the present application proposes a three-dimensional positioning mapping method, which realizes the joint learning of motion estimation and depth estimation of image data by proposing two branches of optical flow decoding and depth decoding, not only reduces the computational complexity, but also combines the complementary advantages of motion and depth data, improves the depth estimation accuracy and robustness, and through point cloud splicing of point cloud data, point cloud registration is realized, and a large-scale dense point cloud map with centimeter-level precision can be obtained. Semantic segmentation and target detection further endow the point cloud map with pixel-level scene understanding, so that the point cloud map contains the category and location information of the target object, so as to finally construct a three-dimensional map containing rich geometric and semantic information.

[0064] For the convenience of understanding the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, which all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0065] The embodiments of the present application provide a three-dimensional positioning mapping method, Figure 1 is a flowchart of the three-dimensional positioning mapping method provided by the embodiments of the present application, as shown in Figure 1 , the method comprises the following steps:

[0066] Step 101: obtaining a first image sequence.

[0067] Among them, the first image sequence includes multiple frames of images obtained by a camera at continuous different time points.

[0068] In the embodiments of the present application, the camera can be a monocular camera, which is installed at the back of the front windshield or the roof of the vehicle. During the driving of the vehicle, the camera continuously captures images at a fixed resolution (such as 1920x1080) and a fixed frame rate (such as 30Hz) to obtain a first image sequence including images at different time points.

[0069] Here, the vehicle can also be equipped with a GPS receiver and an IMU for obtaining GPS / IMU pose observation values of the vehicle at different time points.

[0070] Step 102: optical flow decoding, depth decoding and back projection are performed on adjacent frame images in the first image sequence to obtain a first point cloud map sequence corresponding to the first image sequence.

[0071] In the embodiments of the present application, after obtaining the first image sequence, point cloud conversion can be performed on each frame of image in the first image sequence to obtain a point cloud map corresponding to each frame of image. Specifically, optical flow decoding and depth decoding can be performed on each pair of adjacent frame images in the first image sequence to obtain a depth map and a confidence map corresponding to each pair of adjacent frame images, respectively, and the depth map corresponding to each pair of adjacent frame images is back projected to a three-dimensional space to obtain a point cloud map corresponding to each pair of adjacent frame images at two time points, respectively, thereby obtaining a first point cloud map sequence corresponding to the first image sequence.

[0072] In some embodiments, step 102 specifically comprises:

[0073] For each pair of adjacent frame images in the first image sequence, feature extraction is performed on each pair of adjacent frame images to obtain a first feature map and a second feature map corresponding to each pair of adjacent frame images;

[0074] construct a matching cost volume between the first feature map and the second feature map to obtain a matching cost between the first feature map and the second feature map;

[0075] calculate an optical flow field between each pair of adjacent frame images based on the matching cost, and transform the first feature map and the second feature map based on the optical flow field to obtain a third feature map and a fourth feature map corresponding to image registration of each pair of adjacent frame images;

[0076] perform depth decoding on the third feature map and the fourth feature map to obtain first depth information and second depth information corresponding to each pair of adjacent frame images;

[0077] back-project the first depth information and the second depth information corresponding to each pair of adjacent frame images into a three-dimensional space to obtain a first point cloud map sequence.

[0078] Here, for each pair of adjacent frame images in the first image sequence, a network structure shared by weights, such as a convolutional neural network (CNN), can be used to extract features of the two adjacent frame images to obtain a first feature map and a second feature map corresponding to the two adjacent frame images. Then, a matching cost volume between the first feature map and the second feature map corresponding to the two adjacent frame images is constructed to calculate the matching cost between the feature maps to estimate the correlation between the pixel points in the two adjacent frame images. Then, the optical flow field between the two adjacent frame images, i.e., the displacement vector of each pixel point in the two adjacent frame images, including the two-dimensional vector field of the motion trajectory and speed of each pixel point, is estimated according to the matching cost between the first feature map and the second feature map.

[0079] Then, according to the optical flow field between the two adjacent frame images, spatial transformation is performed on the first feature map and the second feature map to remap the previous frame image in the two adjacent frame images to obtain a new image that is as close as possible to the next frame image, and to obtain a third feature map and a fourth feature map corresponding to the two frame images after image registration. Then, depth decoding is performed on the third feature map and the fourth feature map to generate first depth information and second depth information corresponding to the two adjacent frame images, the first depth information including a depth map and a confidence map corresponding to the previous frame image, and the second depth information including a depth map and a confidence map corresponding to the next frame image. Finally, the depth maps corresponding to the two adjacent frame images are back-projected into a three-dimensional space to obtain point cloud maps corresponding to the two adjacent frame images at two time points, respectively, so that the point cloud maps corresponding to each pair of adjacent frame images can constitute a first point cloud map sequence.

[0080] Step 103: Perform point cloud stitching on the plurality of point cloud maps in the first point cloud map sequence to obtain a first global point cloud map corresponding to the first point cloud map sequence.

[0081] In the embodiments of the present application, after obtaining the first point cloud map sequence corresponding to the first image sequence, point cloud stitching is performed on each point cloud map in the first point cloud map sequence, a plurality of point cloud maps are superimposed by integrating each point cloud map into the same coordinate system, and a globally consistent three-dimensional dense point cloud map, i.e., a first global point cloud map, is formed.

[0082] In the first image sequence, each pair of adjacent frame images corresponds to a first point cloud map at a t-1 moment and a second point cloud map at a t moment, and the first depth information and the second depth information corresponding to each pair of adjacent frame images include a first confidence map of an image at the t-1 moment and a second confidence map of an image at the t moment, and t is greater than or equal to 1.

[0083] In some embodiments, step 103 specifically includes:

[0084] The following steps are iteratively performed until t is equal to the length of the first image sequence, and a first global point cloud map is obtained:

[0085] The second point cloud map is transformed based on a first pose transformation matrix to obtain a third point cloud map; the first pose transformation matrix is a pose transformation matrix of the second point cloud map in the camera coordinate system at the t-1 moment;

[0086] The first point cloud map and the third point cloud map are weighted averaged based on the first confidence map and the second confidence map to obtain a fourth point cloud map;

[0087] The first point cloud map and the third point cloud map are point cloud registered to obtain a second pose transformation matrix; the second pose transformation matrix is an optimized first pose transformation matrix;

[0088] The fourth point cloud map is updated in pose based on the second pose transformation matrix, and t is increased by 1.

[0089] Here, each point cloud map in the first point cloud map sequence can be frame by frame point cloud stitched to obtain the first global point cloud map. Specifically, for each pair of adjacent frame images at the t-1 moment and the t moment, the first point cloud map at the t-1 moment and the second point cloud map at the t moment correspond, and the first depth information corresponding to the image at the t-1 moment includes the first confidence map, and the second depth information corresponding to the image at the t moment includes the second confidence map. Starting from t equal to 1, the pose transformation matrix of the second point cloud map at the t moment in the camera coordinate system at the t-1 moment, i.e., the first pose transformation matrix, is used to transform the second point cloud map at the t moment to obtain a third point cloud map, and then the first confidence map and the second confidence map are used to weight average the first point cloud map at the t-1 moment and the third point cloud map, so as to perform point cloud stitching on the first point cloud map and the third point cloud map to obtain a fourth point cloud map. The fusion method can adaptively suppress depth noise with high uncertainty.

[0090] Then, the first point cloud map and the third point cloud map are registered by using an Iterative Closest Point (ICP) algorithm, rotation and translation parameters are iteratively calculated by minimizing the distance between corresponding points between the first point cloud map and the third point cloud map, so as to obtain an optimized first pose transformation matrix, i.e., a second pose transformation matrix; finally, the fourth point cloud map is updated in pose according to the second pose transformation matrix, and t is increased by 1, and the above process is repeatedly performed until t is equal to the length of the first image sequence, so as to obtain a globally consistent point cloud map, i.e., a first global point cloud map.

[0091] The first pose transformation matrix can be calculated by GPS / IMU integration to obtain an initial estimated value.

[0092] Step 104: performing semantic segmentation and target detection based on the first image sequence and the first global point cloud map to obtain a target global semantic point cloud map.

[0093] In the embodiments of the present application, after the first global point cloud map is obtained, the semantic segmentation of the depth map obtained by projecting the first global point cloud map to the first image sequence is performed to obtain a plurality of semantic segmentation results, and target detection is performed on each frame of image in the first image sequence to obtain a plurality of target detection results, and then the semantic content mapping and the detection content mapping of the first global point cloud map are performed according to the plurality of semantic segmentation results and the plurality of target detection results, so as to obtain a target global semantic point cloud map.

[0094] In some embodiments, step 104 specifically comprises:

[0095] projecting the first global point cloud map onto each frame of image in the first image sequence to obtain a depth map corresponding to each frame of image;

[0096] splicing each frame of image and the depth map corresponding to each frame of image in the channel dimension to obtain a plurality of combined images;

[0097] performing semantic segmentation on each combined image in the plurality of combined images to obtain a plurality of segmentation results;

[0098] performing target detection on each frame of image to obtain a plurality of first target detection results;

[0099] mapping each first target detection result in the plurality of first target detection results into a corresponding point cloud map respectively to obtain a plurality of second target detection results;

[0100] mapping the plurality of segmentation results and the plurality of second target detection results into the first global point cloud map to obtain a target global semantic point cloud map.

[0101] Here, first, the first global point cloud map is projected onto each frame of image in the first image sequence to obtain a depth map corresponding to each frame of image, and each frame of image is spliced with its corresponding depth map in the channel dimension to obtain a plurality of combined images, each combined image being an image including color and depth information; then a semantic segmentation network such as a pyramid scene parsing network (PSPNet) is used to perform semantic segmentation on each combined image to obtain a semantic segmentation result corresponding to each combined image; at the same time, a target detection algorithm such as a you only look once (YOLO) algorithm is used to perform target detection on each frame of image in the first image sequence to obtain a first target detection result corresponding to each frame of image, including a 2D bounding box of a target object contained in each frame of image, and then the first target detection result corresponding to each frame of image is mapped to the point cloud map corresponding thereto, and three-dimensional points within the 2D bounding box are extracted, so that a second target detection result corresponding to each frame of image is obtained, including a 3D bounding box of a target object contained in each frame of image.

[0102] Finally, the semantic segmentation result corresponding to each combined image is pixel by pixel mapped to the first global point cloud map, so that each 3D point in the first global point cloud map additionally contains a class information, and the second target detection result corresponding to each frame of image is mapped to the first global point cloud map, so that each target object in the first global point cloud map contains a 3D bounding box corresponding thereto, and each 3D bounding box includes a class label, a position, a size and an orientation parameter information of the corresponding target object; through the above process, a 3D map containing rich geometric and semantic information, i.e., a target global semantic point cloud map, is finally constructed.

[0103] In some embodiments, the above method can further include:

[0104] Performing pose estimation on the camera based on the first global point cloud map and the first image sequence to obtain a first pose trajectory of the camera in the first global point cloud map;

[0105] Based on the first pose trajectory, determining a second image sequence newly added in a first time step;

[0106] Performing optical flow decoding, depth decoding and reverse projection on adjacent frame images in the second image sequence to obtain a second point cloud map sequence corresponding to the second image sequence;

[0107] Updating the first global point cloud map based on the second point cloud map sequence to obtain a second global point cloud map.

[0108] Here, in the first global point cloud map, if the first global point cloud map includes the projections of the images in the first image sequence, the camera can be pose estimated according to the first global point cloud map and the first image sequence to obtain a first pose trajectory of the camera in the first global point cloud map, so that a first pose trajectory of the vehicle in which the camera is located in the first global point cloud map can be obtained. Specifically, for each frame of image in the first image sequence, 2D feature points are extracted in each frame of image, and the corresponding 3D point coordinates in the first global point cloud map are obtained according to the 2D feature points in each frame of image. Then, a pose positioning algorithm is used to calculate the camera pose corresponding to each frame of image according to the 2D-3D point pair and in combination with the camera intrinsic parameter, and a non-linear optimization method is further used to optimize the camera pose corresponding to each frame of image, so as to obtain the first pose trajectory of the camera in the first global point cloud map.

[0109] With the movement of the camera, new areas will be continuously exposed in the camera field of view. In order to maintain the real-time performance of the map, it is necessary to superimpose the newly collected point cloud data online. Specifically, a first time step can be set, and in the first pose trajectory, the image data collected by the camera within the first time step is determined, that is, a second image sequence newly added within the first time step is determined, and the adjacent frames of images in the second image sequence are subjected to optical flow decoding, depth decoding and back projection to obtain a second point cloud map sequence. Then, the first global point cloud map is updated according to the second point cloud map sequence to obtain a second global point cloud map.

[0110] In some embodiments, updating the first global point cloud map based on the second point cloud map sequence to obtain the second global point cloud map comprises:

[0111] A time stamp is set for each three-dimensional point in the first global point cloud map, and a forgetting time threshold is set for the time stamp.

[0112] For each three-dimensional point, if the time stamp of the three-dimensional point exceeds the forgetting time threshold, the three-dimensional point is removed from the first global point cloud map, and the second point cloud map sequence is superimposed to the first global point cloud map to obtain the second global point cloud map.

[0113] Here, a time stamp is set for each 3D point in the first global point cloud map to record the survival time of each 3D point data, and a forgetting time threshold is set for the time stamp of each 3D point. During the movement of the camera, it is inquired in real time whether the time stamp of each 3D point exceeds the forgetting time threshold. If there is a 3D point whose time stamp exceeds the forgetting time threshold, it is considered that the 3D point is expired, and the 3D point is removed from the first global point cloud map. Then, the newly added second point cloud map sequence is superimposed to the first global point cloud map to obtain the second global point cloud map.

[0114] It should be noted that, in order to avoid excessive redundancy of the map, the first global point cloud map and the second point cloud sequence can also be voxel down-sampled before the second point cloud sequence is superimposed on the first global point cloud map, so as to reduce the size of the point cloud data, reduce the storage and transmission cost, and the voxelization process can smooth the local noise points and improve the data quality.

[0115] In some embodiments, after obtaining the target global semantic point cloud map, the above method can further include:

[0116] extracting a target object on the target global semantic point cloud map, and constructing a network topology graph based on the target object;

[0117] performing path search on the network topology graph based on a preset rule to obtain a global reference path;

[0118] sliding a preset space-time window around the camera to obtain a point cloud map in a target region;

[0119] performing path planning based on the global reference path and the point cloud map in the target region to obtain a local target path;

[0120] executing the local target path in a second time step, and performing rolling optimization on the local target path based on the preset space-time window.

[0121] Here, on the basis of the target global semantic point cloud map, a target object can be extracted on the target global semantic point cloud map, and a network topology graph can be constructed according to the extracted target object. Then, according to a preset rule such as a traffic rule, path search is performed on the constructed network topology graph to obtain a global reference path. Then, taking the vehicle where the camera is located as the center, a preset space-time window is slid on the global reference path to obtain a point cloud map in a target region of the sliding, and the distribution of obstacles in the target region is analyzed. Vehicle kinematics and dynamics constraints are introduced to perform path planning in the target region near the global reference path to obtain a local optimal trajectory, i.e., a local target path. Finally, the local target path is executed for a preset number of time steps (second time steps), while the obstacle information is constantly updated, and the local optimal trajectory in the next region is rolled according to the preset space-time window, so as to realize global path and local trajectory optimization.

[0122] In the technical solution of the embodiment of the application, a first image sequence is acquired; the first image sequence includes multiple frames of images obtained by a camera at continuous different time instants; optical flow decoding, depth decoding and reverse projection are performed on adjacent frame images in the first image sequence to obtain a first point cloud map sequence corresponding to the first image sequence; multiple point cloud maps in the first point cloud map sequence are spliced to obtain a first global point cloud map corresponding to the first point cloud map sequence; semantic segmentation and target detection are performed based on the first image sequence and the first global point cloud map to obtain a target global semantic point cloud map. In this way, by proposing two branches of optical flow decoding and depth decoding, joint learning of motion estimation and depth estimation of image data can be achieved, which not only reduces the computational complexity, but also combines the complementary advantages of motion and depth data, improves the depth estimation accuracy and robustness, and by splicing point cloud data, point cloud registration can be achieved, a large-scale dense point cloud map with centimeter-level precision can be obtained, and semantic segmentation and target detection further endow the point cloud map with pixel-level scene understanding, so that the point cloud map contains the category and position information of the target object, thereby finally constructing a three-dimensional map containing rich geometric and semantic information.

[0123] The embodiment of the application further provides a monocular camera three-dimensional positioning mapping system based on deep learning, Figure 2 which is a general framework diagram of the monocular camera three-dimensional positioning mapping system based on deep learning provided by the embodiment of the application, as shown in the figure, the framework mainly includes a perception module, a mapping module, a positioning module and a planning module, and the specific content is as follows: Figure 2

[0124] (1) Perception module

[0125] The system adopts an industrial monocular camera with a forward or downward inclination of 45 degrees as the main perception device, which is installed at the rear of the front windshield or the roof of the vehicle. The camera adopts a global shutter, and the resolution is not less than 1920x1080, and the frame rate is not less than 30Hz. At the same time, the system is also equipped with a vehicle-mounted GPS receiver and a 6-axis IMU for obtaining the rough pose priori of the vehicle. In addition, the vehicle must be equipped with a wheel speed meter, an electronic compass and other conventional odometry sensors. The function of the perception module is to synchronously collect the video image sequence I t and the GPS / IMU pose observation value at the corresponding time instant t. The image sequence is sent to the subsequent perception algorithm pipeline for processing.

[0126] (2) Mapping module

[0127] ​This module employs the Dense Unsupervised Self-training for Scene Flow Reconstruction from Videos (DUSt3R) deep learning algorithm to convert monocular images into point clouds. For example... Figure 3 The diagram shown is a schematic representation of the process for point cloud conversion of a monocular image according to an embodiment of this application. The process includes the following steps:

[0128] Step 301: Input a pair of adjacent frame images I t and I {t+1} .

[0129] Step 302: Extract adjacent frame images I t and I {t+1} Feature map F t and F {t+1} .

[0130] Step 303: Calculate feature map F t and F {t+1} The matching cost between them.

[0131] Step 304: Calculate the image I of two adjacent frames based on the matching cost. t and I {t+1} The optical flow field f between them.

[0132] Step 305: Based on the optical flow field f, analyze the feature map F t and F {t+1} The transformation is performed to obtain the registered feature maps F′t and F′. {t+1} And the registered feature maps F′t and F′ {t+1} Perform deep decoding.

[0133] Step 306: Output two adjacent frames I t and I {t+1} The corresponding depth maps d t d {t+1} And confidence plot c t c {t+1} .

[0134] The network uses a pair of adjacent frame images I t and I {t+1} As input, the time interval between two frames is dt (typically 0.1–0.5 s). The network first uses a CNN backbone network with shared weights to extract the images I of two adjacent frames. t and I {t+1} Feature map F t and F {t+1} Then, a feature map F is constructed using a cost volume layer. tand F {t+1} The matching cost body between them is used to calculate the feature map F. t and F {t+1} The matching cost between them is then used to decode and generate adjacent two-frame images I through a flow block. t and I {t+1} The optical flow field f between them, and further, based on the optical flow field f, the feature map F t and F {t+1} Perform a warp transform to obtain the registered feature maps F′t and F′. {t+1} The two registered feature maps F′t and F′ {t+1} Then, each image is decoded through a depth block to generate two adjacent frames I. t and I {t+1} The corresponding depth maps d t d {t+1} And confidence plot c t c {t+1} The confidence map reflects the uncertainty of the depth estimate for each pixel and can be used for weighted averaging in the fusion stage.

[0135] Image I of two adjacent frames t and I {t+1} The corresponding depth maps d t d {t+1} By projecting the images backward into three-dimensional space, we obtain the point cloud images P of the two frames at two different times. t and P {t+1} The conversion formula is:

[0136] P t =d t ×K {-1} ×(u,v,1) T (1)

[0137] P {t+1} =d {t+1} ×K {-1} ×(u,v,1) T (2)

[0138] Where K is the camera intrinsic parameter matrix, and (u,v) are pixel coordinates.

[0139] The advantages of this network are:

[0140] 1) Two decoding branches, optical flow and depth, are proposed to decode the optical flow field and depth map from the matching cost volume, respectively, so as to realize the joint learning of motion estimation and depth estimation instead of separating the two.

[0141] 2) The optical flow field model establishes the pixel-level correspondence between two consecutive frames, and the F-axis can be transformed using a differentiable warp operation.t Registration to F {t+1} , which simplifies the computation of matching cost.

[0142] 3) A flow consistency loss is designed to encourage f t and f {t+1} to be inverse, explicitly modeling the temporal consistency constraint of optical flow.

[0143] 4) A self-discovery mask loss is proposed to mask outliers caused by dynamic objects, improving the accuracy of depth estimation.

[0144] 5) A multi-scale network structure is designed, which greatly reduces the computational overhead while ensuring the resolution of the depth map.

[0145] The above advantages enable the DUSt3R algorithm to output a higher quality depth map than traditional binocular matching, without being limited by calibration errors and baseline lengths, while the computational cost is comparable to monocular depth estimation. During network training, only the monocular video itself is needed, without the need for manually labeled depth ground truth, so the sample acquisition cost is very low.

[0146] Using a series of point cloud maps generated by the DUSt3R algorithm, the embodiment of the application proposes a three-dimensional reconstruction method combining point cloud (PointMap) splicing, perspective-n-point (PnP) positioning, and incremental mapping. As shown in Figure 4 The flowchart for splicing point clouds provided by the embodiment of the application includes the following steps:

[0147] Step 401: input point cloud maps P t and P {t+1} .

[0148] Step 402: input confidence maps c t and c {t+1} .

[0149] Step 403: input the initial pose T t from GPS / IMU.

[0150] Step 404: transform P t to the P t coordinate system using T {t+1} .

[0151] Step 405: fuse P t and P {t+1} by confidence weighted average.

[0152] Step 406: optimize T t to

[0153] Step 407: utilize updating the fused point cloud pose.

[0154] Step 408: output the global point cloud map P {global} .

[0155] Select some key frame images at fixed time intervals (e.g. 0.5s), and input the key frame images pair by pair into the PointMap reconstruction network to obtain the corresponding PointMap X n |n=1,…,N. Predict their PointMap using the PointMap reconstruction network. The result of this step is a set of PointMaps that express the scene geometry locally from different perspectives. Perform point cloud stitching frame by frame.

[0156] Global optimization: globally optimize all the PointMaps to align them as accurately as possible in the common three-dimensional space. When optimizing, the rigid body transformation (rotation and translation) and the scale factor of each PointMap need to be estimated. These parameters are obtained by minimizing the PointMap matching error between different perspectives. At this time, complex factors such as occlusion do not need to be considered, because PointMap matching is performed in three-dimensional space rather than image space. In the Pairwise graph above, by optimizing the pose transformation T n and the scale factor s n of each PointMap, global alignment is achieved:

[0157]

[0158] where χ * is the globally consistent point cloud map in the reference coordinate system, and ε is the edge set of the Pairwise graph.

[0159] Mesh reconstruction: reconstruct the final digital twin mesh model from the globally consistent PointMap module. Specifically, first fuse all PointMaps into a dense point cloud, then mesh the point cloud to obtain a triangular mesh surface. At the same time, map the pixel color recorded in the PointMap to the mesh vertex to generate texture information. In this way, the digital twin model restores the geometric shape and real appearance of the scene.

[0160] Let T t be the pose transformation matrix of the point cloud at time t+1 in the camera coordinate system at time t. First, transform P t+1 to time t, then concatenate with P {t+1} :

[0161]

[0162] where T t An initial estimate can be calculated by GPS / IMU integration. The two point clouds P t and P {t+1} are weighted averaged based on their respective confidence maps c {t+1} , c t , and the result is:

[0163]

[0164] where denotes element-wise multiplication and denotes element-wise division. This fusion method can adaptively suppress depth noise with high uncertainty.

[0165] Then, P {t+1} and P are registered using the ICP algorithm to obtain an optimized pose transformation matrix Then, the pose of P is updated. By repeating this process, a globally consistent point cloud map P is obtained:

[0166]

[0167] The global point cloud map P {global} obtained by the above method only contains geometric information. To realize more advanced scene interaction functions, the global point cloud map needs to be further endowed with semantic attributes. A common approach is to classify images pixel by pixel, map the segmentation mask to the point cloud map, and associate a class label with each 3D point, as shown in Figure 5 The flowchart provided by the embodiments of the present application for semantic segmentation and target detection based on the global point cloud map and the image sequence includes the following steps:

[0168] Step 501: Project the global point cloud map P {global} onto each frame of image I t in the image sequence.

[0169] Step 502: Output the corresponding depth map d t for each frame of image I t .

[0170] Step 503: Concatenate each frame of image I t and its corresponding depth map d t in the channel dimension to obtain an RGB-D image.

[0171] Step 504: Perform semantic segmentation on the RGB-D image using the PSPNet semantic segmentation network to obtain a segmentation mask.

[0172] Step 505: Map the segmentation mask back to the global point cloud map P {global} .

[0173] Step 506: Perform object detection on each frame of image I t to obtain the object detection result.

[0174] Step 507: Project the 2D bounding box of the object in the object detection result onto the global point cloud map P {global} .

[0175] Step 508: Extract the 3D points corresponding to the in-frame object from the global point cloud map P {global} to obtain the 3D bounding box of the object.

[0176] Step 509: Add the 3D bounding box of the object to the global point cloud map P {global} .

[0177] Step 510: Output the global semantic point cloud map.

[0178] Network input: Project the global point cloud map P {global} onto each frame of image I t in the image sequence to obtain the corresponding 2D depth map d t , concatenate each frame of image I t with its corresponding 2D depth map d t in the channel dimension to obtain a color-depth (RGB-Depth, RGBD) image.

[0179] Network structure: The pyramid pooling module extracts global context information from the input RGBD image at different sampling rates, fuses local details and global semantics, and finally restores the segmentation map size through an upsampling module.

[0180] Network training: Pre-training on common datasets such as Cityscapes, Scene Understanding Benchmark (ADE20K), etc., and fine-tuning on a small amount of manually annotated autonomous driving scene data.

[0181] Point cloud coloring: Map the segmentation result back to P {global} pixel by pixel, and each 3D point will additionally contain a class ID.

[0182] In addition to semantic segmentation, the embodiments of the present application also run YOLOv3 object detection on each frame of image I t to obtain each frame of image I t2D bounding boxes of various types of traffic participants (e.g. pedestrians, vehicles, etc.). Project the 2D bounding boxes of these targets onto the corresponding point cloud map P t , we can obtain the 3D bounding boxes of these targets. Add the 3D boxes of each target to the global point cloud map:

[0183]

[0184] where, is the set of 3D bounding boxes of the i-th target. Each box is described by its category label l i and position, size, orientation parameters (x, y, z, l, w, h, θ). Through the above process, we finally construct a 3D map containing rich geometric and semantic information, i.e. the global semantic point cloud map.

[0185] (3) Localization module

[0186] PnP pose estimation: In the global point cloud map P {global} , if the projection of the image I t is contained, we can solve the camera pose T t through 2D-3D feature point matching. Specifically, extract the ORB feature points {x i} in I t , get their corresponding 3D point coordinates {X i}, and then construct the PnP problem:

[0187] x i = K x T t x X i (9)

[0188] where K is the camera intrinsic.

[0189] Then solve T t using the Random Sample Consensus (RANSAC) and Efficient PnP (EPnP) algorithm, remove dynamic outliers, and further refine T t using nonlinear optimization methods such as ceres. Repeat the above PnP solving process, and we can obtain the real-time pose trajectory {T t} of the camera in the point cloud map P {global} .

[0190] Incremental mapping: As the camera moves, new areas are constantly exposed to the camera's field of view. To maintain the real-time nature of the map, we need to overlay newly collected point cloud data online to expand the coverage of the map. Specifically, we can fix a time step Δt (e.g. 5s), and add the newly added point cloud P newSuperimposed into the global point cloud map:

[0191]

[0192] Wherein, in order to avoid the map being too redundant, P {global} and P new may be voxel down-sampled before superimposition.

[0193] To cope with dynamic environment, a time decay factor is introduced to forget past data. A timestamp is given to each 3D point, and a forgetting time threshold (e.g. 10 min) is set. If the current timestamp of a 3D point exceeds the forgetting time threshold, the data of the 3D point is considered expired, and the expired point cloud data will be removed from the global point cloud map.

[0194] (4) Planning module

[0195] On the basis of semantic point cloud map, autonomous vehicle can achieve more intelligent decision planning. Taking path planning as an example, traditional methods often first binarize point cloud or grid map, extract passable area, and then search for optimal path using A-star algorithm (A-Star, A*), rapidly-exploring random tree (Rapidly-exploring Random Tree, RRT) and other algorithms. This kind of method only considers the geometric passability of the road, ignoring many important semantic factors such as:

[0196] 1) Lane line: the vehicle should strictly follow the lane line, rather than freely cross multiple lanes.

[0197] 2) Pedestrian crossing and traffic signs: slow down and give way to pedestrians when encountering pedestrian crossing; stop and wait when seeing red light or stop sign.

[0198] 3) Dynamic obstacles: be extra careful to small and uncertainly moving objects such as pedestrians and bicycles, and slow down and keep a safe distance.

[0199] Considering these factors, a better path planning process is:

[0200] First, extract the road and lane line on the semantic point cloud map, and construct a road network topology graph considering semantics; second, search a global navigation path on the road network topology graph as a reference line R, combining traffic rules; then, slide a space-time window on the reference line R with the vehicle as the center, copy the point cloud data in the region; analyze the obstacle distribution in the region, and introduce vehicle kinematics and dynamics constraints to solve a locally optimal trajectory A* near the reference line R through quadratic programming:

[0201]

[0202] where w(A) is the smoothness cost of trajectory A*, and T is the planning horizon. The above formula needs to satisfy the vehicle dynamic constraints and the anti-collision constraints.

[0203] The first few time steps of trajectory A* are executed while the obstacle information is constantly updated, the trajectory of the next time window is rolled and optimized, and the model predictive control is realized.

[0204] The above process can balance global path and local trajectory optimization, and introduce rich prior knowledge to guide decision-making. In addition, semantic information can also be used for traffic sign detection and recognition, vehicle intention prediction and other automatic driving tasks.

[0205] In summary, the technical scheme provided by the embodiments of the present application has the following advantages:

[0206] (1) Only one low-cost monocular camera is needed to obtain a dense, accurate and semantic-rich three-dimensional scene reconstruction, and the perception hardware cost is greatly reduced.

[0207] (2) The DUSt3R algorithm can learn depth estimation from monocular video end-to-end, without the need for manually designed features or camera calibration, greatly reducing the algorithm complexity. Its depth estimation accuracy and robustness surpasses traditional algorithms.

[0208] (3) The PointMap stitching and incremental mapping strategy can obtain a large-scale dense point cloud map with centimeter-level accuracy, and the calculation time and space complexity is linearly related to the map scale.

[0209] (4) The PnP positioning method directly uses 2D-3D feature matching, which is simple and efficient, and is suitable for real-time applications. Combined with incremental mapping, sub-meter positioning can be achieved in a million square meter scene.

[0210] (5) The semantic segmentation and object detection module further gives the point cloud pixel-level scene understanding, so that it contains the category and location information of key elements such as roads, vehicles and pedestrians.

[0211] (6) The semantic point cloud map combined with vehicle motion constraints can realize seamless connection of global path planning and local trajectory optimization, as well as traffic sign detection, vehicle intention prediction and other automatic driving functions.

[0212] (7) The modular design makes the system easy to integrate and extend. Developers can flexibly tailor or combine different algorithm modules according to needs.

[0213] The embodiments of the present application also provide a three-dimensional positioning and mapping device, Figure 6 is a structural schematic diagram of the three-dimensional positioning and mapping device provided by the embodiments of the present application, as Figure 6 shown, the device comprises:

[0214] The acquisition unit 601 is configured to acquire a first image sequence; the first image sequence comprises a plurality of images captured by a camera at continuous time instants.

[0215] The conversion unit 602 is configured to perform optical flow decoding, depth decoding and back projection on adjacent frame images in the first image sequence to obtain a first point cloud map sequence corresponding to the first image sequence.

[0216] The splicing unit 603 is configured to perform point cloud splicing on a plurality of point cloud maps in the first point cloud map sequence to obtain a first global point cloud map corresponding to the first point cloud map sequence.

[0217] The segmentation and detection unit 604 is configured to perform semantic segmentation and target detection based on the first image sequence and the first global point cloud map to obtain a target global semantic point cloud map.

[0218] In some embodiments, the apparatus further comprises an estimation unit, a determination unit and an updating unit; wherein,

[0219] The estimation unit is configured to perform pose estimation on the camera based on the first global point cloud map and the first image sequence to obtain a first pose trajectory of the camera in the first global point cloud map.

[0220] The determination unit is configured to determine, based on the first pose trajectory, a second image sequence newly added in a first time step.

[0221] The conversion unit 602 is further configured to perform optical flow decoding, depth decoding and back projection on adjacent frame images in the second image sequence to obtain a second point cloud map sequence corresponding to the second image sequence.

[0222] The updating unit is configured to update the first global point cloud map based on the second point cloud map sequence to obtain a second global point cloud map.

[0223] In some embodiments, the conversion unit 602 is specifically configured to:

[0224] For each pair of adjacent frame images in the first image sequence, performing feature extraction on each pair of adjacent frame images to obtain a first feature map and a second feature map corresponding to each pair of adjacent frame images;

[0225] Constructing a matching cost volume based on the first feature map and the second feature map to obtain a matching cost between the first feature map and the second feature map;

[0226] Calculating an optical flow field between each pair of adjacent frame images based on the matching cost, and transforming the first feature map and the second feature map based on the optical flow field to obtain a third feature map and a fourth feature map corresponding to each pair of adjacent frame images after image registration;

[0227] performing deep decoding on the third feature map and the fourth feature map to obtain first depth information and second depth information corresponding to each pair of adjacent frame image pairs;

[0228] projecting the first depth information and the second depth information corresponding to each pair of adjacent frame image pairs into a three-dimensional space to obtain a first point cloud map sequence.

[0229] wherein the first point cloud map corresponding to each pair of adjacent frame image pairs at a t-1 moment and the second point cloud map at a t moment, the first depth information and the second depth information corresponding to each pair of adjacent frame image pairs include a first confidence map of an image at a t-1 moment and a second confidence map of an image at a t moment, and t is greater than or equal to 1.

[0230] In some embodiments, the splicing unit 603 is specifically configured to:

[0231] iteratively performing the following steps until t is equal to the length of the first image sequence to obtain a first global point cloud map:

[0232] transforming the second point cloud map based on a first pose transformation matrix to obtain a third point cloud map; the first pose transformation matrix is a pose transformation matrix of the second point cloud map in a camera coordinate system at a t-1 moment;

[0233] performing weighted averaging on the first point cloud map and the third point cloud map based on the first confidence map and the second confidence map to obtain a fourth point cloud map;

[0234] performing point cloud registration on the first point cloud map and the third point cloud map to obtain a second pose transformation matrix; the second pose transformation matrix is an optimized first pose transformation matrix;

[0235] updating the pose of the fourth point cloud map based on the second pose transformation matrix, and adding 1 to t.

[0236] In some embodiments, the segmentation and detection unit 604 is specifically configured to:

[0237] projecting the first global point cloud map onto each frame of image in the first image sequence to obtain a depth map corresponding to each frame of image;

[0238] splicing each frame of image and the depth map corresponding to each frame of image in the channel dimension to obtain a plurality of combined images;

[0239] performing semantic segmentation on each combined image in the plurality of combined images to obtain a plurality of segmentation results;

[0240] performing target detection on each frame of image to obtain a plurality of first target detection results;

[0241] mapping each first target detection result in the plurality of first target detection results into a corresponding point cloud map to obtain a plurality of second target detection results.

[0242] map the plurality of segmentation results and the plurality of second target detection results into the first global point cloud map to obtain a target global semantic point cloud map.

[0243] In some embodiments, the updating unit is specifically configured to:

[0244] set a timestamp for each three-dimensional point in the first global point cloud map, and set a forgetting time threshold for the timestamp;

[0245] for each three-dimensional point, if the timestamp of the three-dimensional point exceeds the forgetting time threshold, eliminate the three-dimensional point in the first global point cloud map, and superimpose the second point cloud map sequence on the first global point cloud map to obtain a second global point cloud map.

[0246] In some embodiments, the device further comprises an extraction unit, a construction unit, a searching unit, a sliding unit, a planning unit, and an execution unit, wherein:

[0247] The extraction unit is configured to extract a target object on the target global semantic point cloud map.

[0248] The construction unit is configured to construct a network topology graph based on the target object.

[0249] The searching unit is configured to perform path searching on the network topology graph based on a preset rule to obtain a global reference path.

[0250] The planning unit is configured to perform path planning based on the global reference path and a point cloud map in a target region to obtain a local target path.

[0251] The execution unit is configured to execute the local target path within a second time step.

[0252] The updating unit is further configured to perform rolling optimization on the local target path based on a preset space-time window.

[0253] Those skilled in the art should understand that, Figure 6 The implementation functions of each unit in the three-dimensional positioning and mapping device shown can be understood with reference to the related descriptions of the foregoing methods. Figure 6 The functions of each unit in the three-dimensional positioning and mapping device shown can be implemented through a program running on a processor, or through a specific logic circuit.

[0254] Figure 7 is a structural schematic diagram of a processing device provided by an embodiment of the present application. The processing device can be a terminal device or a network device, Figure 7 The processing device shown comprises a processor 701, which can call and run a computer program from a memory to implement the method in the embodiments of the present application.

[0255] Optionally, as shown in Figure 7 The processing device can further include a memory 702. The processor 701 can call and run a computer program from the memory 702 to implement the method in the embodiments of the present application.

[0256] The memory 702 can be a separate device independent of the processor 701, or can be integrated in the processor 701.

[0257] Optionally, as shown in Figure 7 The processing device can further include a transceiver 703, and the processor 701 can control the transceiver 703 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices.

[0258] The transceiver 703 can include a transmitter and a receiver. The transceiver 703 can further include an antenna, and the number of antennas can be one or more.

[0259] The processing device can be a three-dimensional positioning and mapping device of the embodiments of the present application, and the processing device can implement the corresponding processes of the methods implemented by the embodiments of the present application. For brevity, details are not repeated here.

[0260] It should be understood that the processor of the embodiments of the present application can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method embodiments can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software. The processor described above can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0261] It can be appreciated that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a random access memory (Random Access Memory, RAM) used as an external cache. By way of example and not limitation, many forms of RAM can be used, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM). It should be noted that the memory of the system and method described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0262] The embodiments of the present application also provide a computer readable storage medium for storing a computer program. The computer readable storage medium can be applied to the processing device in the embodiments of the present application, and the computer program makes the computer execute the corresponding processes of the various method implementations of the embodiments of the present application. For brevity, details are not repeated here.

[0263] The embodiments of the present application also provide a computer program product including computer program instructions. The computer program product can be applied to the processing device in the embodiments of the present application, and the computer program instructions make the computer execute the corresponding processes of the various method implementations of the embodiments of the present application. For brevity, details are not repeated here.

[0264] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0265] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0266] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0267] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0268] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0269] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0270] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A three-dimensional positioning and mapping method, characterized in that, The method includes: Acquire a first image sequence; the first image sequence includes multiple frames of images captured by the camera at different consecutive times. Optical flow decoding, depth decoding, and back projection are performed on adjacent frame images in the first image sequence to obtain the first point cloud image sequence corresponding to the first image sequence. Multiple point cloud maps in the first point cloud map sequence are stitched together to obtain the first global point cloud map corresponding to the first point cloud map sequence; Semantic segmentation and target detection are performed based on the first image sequence and the first global point cloud map to obtain a global semantic point cloud map of the target.

2. The method according to claim 1, characterized in that, The method further includes: Based on the first global point cloud map and the first image sequence, the pose of the camera is estimated to obtain the first pose trajectory of the camera in the first global point cloud map. Based on the first pose trajectory, determine the newly added second image sequence within the first time step; Optical flow decoding, depth decoding, and back projection are performed on adjacent frame images in the second image sequence to obtain the second point cloud image sequence corresponding to the second image sequence. The first global point cloud map is updated based on the second point cloud map sequence to obtain the second global point cloud map.

3. The method according to claim 1, characterized in that, The step of performing optical flow decoding, depth decoding, and back projection on adjacent frames in the first image sequence to obtain the first point cloud sequence corresponding to the first image sequence includes: For each pair of adjacent frame images in the first image sequence, feature extraction is performed on each pair of adjacent frame images to obtain a first feature map and a second feature map corresponding to each pair of adjacent frame images; A matching cost body is constructed for the first feature map and the second feature map to obtain the matching cost between the first feature map and the second feature map; The optical flow field between each pair of adjacent frame images is calculated based on the matching cost, and the first feature map and the second feature map are transformed based on the optical flow field to obtain the third feature map and the fourth feature map after image registration for each pair of adjacent frame images. Depth decoding is performed on the third feature map and the fourth feature map to obtain the first depth information and the second depth information corresponding to each pair of adjacent frame images; The first depth information and the second depth information corresponding to each pair of adjacent frame images are back-projected into three-dimensional space to obtain the first point cloud map sequence.

4. The method according to claim 3, characterized in that, The first point cloud map and the second point cloud map at time t-1 correspond to each pair of adjacent frame images. The first depth information and the second depth information corresponding to each pair of adjacent frame images include the first confidence map of the image at time t-1 and the second confidence map of the image at time t, where t is greater than or equal to 1. The step of stitching together multiple point cloud images from the first point cloud image sequence to obtain the first global point cloud map corresponding to the first point cloud image sequence includes: Iteratively execute the following steps until t equals the length of the first image sequence to obtain the first global point cloud map: The second point cloud map is transformed based on the first pose transformation matrix to obtain the third point cloud map; the first pose transformation matrix is ​​the pose transformation matrix of the second point cloud map in the camera coordinate system at time t-1. A fourth point cloud map is obtained by weighted averaging the first point cloud map and the third point cloud map based on the first confidence map and the second confidence map; Point cloud registration is performed on the first point cloud map and the third point cloud map to obtain a second pose transformation matrix; the second pose transformation matrix is ​​the optimized first pose transformation matrix; The pose of the fourth point cloud map is updated based on the second pose transformation matrix, and t is incremented by 1.

5. The method according to claim 1, characterized in that, The step of performing semantic segmentation and target detection based on the first image sequence and the first global point cloud map to obtain a target global semantic point cloud map includes: The first global point cloud map is projected onto each frame of the first image sequence to obtain a depth map corresponding to each frame. Each frame image is stitched together with its corresponding depth map along the channel dimension to obtain multiple combined images; Semantic segmentation is performed on each of the multiple combined images to obtain multiple segmentation results; Target detection is performed on each frame of the image to obtain multiple first target detection results; Each of the plurality of first target detection results is mapped to a corresponding point cloud map to obtain a plurality of second target detection results; The multiple segmentation results and the multiple second target detection results are mapped to the first global point cloud map to obtain the target global semantic point cloud map.

6. The method according to claim 2, characterized in that, The step of updating the first global point cloud map based on the second point cloud map sequence to obtain the second global point cloud map includes: Set a timestamp for each 3D point in the first global point cloud map, and set a forgetting time threshold for the timestamp; For each 3D point, if the timestamp of the 3D point exceeds the forgetting time threshold, the 3D point is removed from the first global point cloud map, and the second point cloud map sequence is superimposed on the first global point cloud map to obtain the second global point cloud map.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Extract target objects from the target global semantic point cloud map and construct a network topology graph based on the target objects; Based on preset rules, a path search is performed on the network topology map to obtain a global reference path; A preset spatiotemporal window is slid across the periphery of the camera to obtain a point cloud map within the target area; Based on the global reference path and the point cloud map within the target area, a path planning is performed to obtain a local target path; The local target path is executed within the second time step, and the local target path is optimized by scrolling based on the preset spatiotemporal window.

8. A processing device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer program instructions that cause a computer to perform the method as described in any one of claims 1 to 7.

Images