Three-dimensional vector map generation method, device and equipment based on data closed-loop optimization

Abstract

The present invention discloses a method, device and equipment for generating a three-dimensional vector map based on data closed-loop optimization, the method comprising: a forward link, obtaining a three-dimensional point cloud map and first environment image data, and extracting two-dimensional vector elements from the first environment image data based on a deep learning perception model; converting two-dimensional vector elements into three-dimensional vector elements by associating the first environment image data with the three-dimensional point cloud map to form a three-dimensional vector map; a reverse link, based on the positioning result and the back projection of the three-dimensional vector elements, generating image information with two-dimensional vector annotations, and feeding back the training deep learning perception model. The present invention is applied to the field of autonomous driving, not only to solve the problem that the lack of depth information in the image data makes it difficult to project two-dimensional vector elements into three-dimensional vector elements, but also to generate a large amount of sample data using the generated three-dimensional vector map, thereby realizing a data closed loop and improving the generalization ability and recognition accuracy of the perception model.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving and intelligent transportation technology, and specifically to a three-dimensional vector map generation method, device and equipment based on data closed-loop optimization. Background Art

[0002] In recent years, with the rapid development of autonomous driving technology, 3D vector maps have been increasingly used in the fields of autonomous driving, intelligent transportation, etc. However, traditional 3D vector map generation methods mainly rely on LiDAR point cloud map data and manual annotation, which is expensive and complex in data processing, limiting the large-scale application of 3D vector maps.

[0003] With the development of computer vision and deep learning technology, vector feature extraction methods based on image data have gradually emerged. Basic feature information such as lane lines and road boundaries are also common recognition targets in autonomous driving perception tasks. By extracting two-dimensional vector data of lane lines and road boundaries from real-time images through deep learning models and converting them to a three-dimensional coordinate system through spatial transformation, the cost of generating three-dimensional vector feature extraction can be greatly reduced.

[0004] However, training an image perception model to extract high-precision two-dimensional vector elements from an image requires a large amount of high-quality training data. For example, patent CN116753936A discloses an online construction method for an autonomous driving vector map based on multi-source visual images, which estimates the relative position of the current target vehicle and adjacent vehicles based on the image and processes them to obtain multi-perspective visual features, which are finally converted into bird's-eye view features and then vectorized. However, the method cannot obtain an accurate description of the vector elements in three-dimensional space, and the method strongly relies on the accuracy and reasoning ability of the deep learning model, requiring a large amount of sample data training, but the problem of difficult sample acquisition and whether the model accuracy can meet the complex task requirements is still a problem that needs to be solved. Summary of the invention

[0005] In view of the deficiencies in the above-mentioned prior art, the present invention provides a three-dimensional vector map generation method, device and equipment based on data closed-loop optimization, which can not only solve the problem that the lack of depth information in the image data makes it difficult to project two-dimensional vector elements into three-dimensional vector elements, but also use the generated three-dimensional vector map to generate a large amount of sample data, thereby realizing data closure and improving the generalization ability and recognition accuracy of the perception model.

[0006] To achieve the above object, the present invention provides a three-dimensional vector map generation method based on data closed-loop optimization, including a forward link and a reverse link;

[0007] In the forward link:

[0008] Acquire a three-dimensional point cloud map and first environment image data of a target map scene, and extract two-dimensional vector elements from the first environment image data based on a deep learning perception model;

[0009] By associating the first environment image data with the three-dimensional point cloud map, the two-dimensional vector elements are converted into three-dimensional vector elements to form a three-dimensional vector map;

[0010] In the reverse link:

[0011] Based on real-time positioning of a carrier equipped with a positioning sensor and a camera on a three-dimensional vector map, the positioning result is combined with the back projection of the three-dimensional vector elements in the three-dimensional vector map to generate image information with two-dimensional vector annotations, which is used to feedback and train the deep learning perception model to complete the data closure and optimization.

[0012] In one embodiment, the forward link comprises:

[0013] When the first carrier equipped with the laser radar, the first camera, and the first positioning sensor is driving in the target map scene, three-dimensional point cloud data of the target map scene, first environment image data, and position information of the first carrier are synchronously collected;

[0014] Generate a globally consistent three-dimensional point cloud map based on the three-dimensional point cloud data of the target map scene and the position information of the first carrier;

[0015] Extracting two-dimensional vector elements from the first environment image data based on the deep learning perception model, wherein the two-dimensional vector elements are represented in the form of a set of pixel point coordinates;

[0016] Based on the external parameter transformation between the first camera and the laser radar, the global pose of the first camera in the three-dimensional point cloud map at the sampling time of the first environment image data is obtained, and the map points of the corresponding area in the three-dimensional point cloud map are back-projected to the image coordinate system according to the first camera pose and the first camera internal parameters at that time, so as to obtain the depth value corresponding to each pixel point in the first environment image data;

[0017] Convert pixel point coordinates and depth values ​​of all two-dimensional vector elements in the first environment image data from the image coordinate system to the world coordinate system to obtain three-dimensional vector elements corresponding to the first environment image data, wherein the three-dimensional vector elements are represented in the form of a set of three-dimensional coordinate points;

[0018] By geometrically associating the three-dimensional vector elements obtained after transforming multiple frames of first environment image data, a three-dimensional vector map can be obtained.

[0019] In one of the embodiments, after generating a globally consistent three-dimensional point cloud map, the three-dimensional point cloud map is further smoothed to remove noise and errors and improve map accuracy.

[0020] In one of the embodiments, before extracting two-dimensional vector elements from the first environment image data based on the deep learning perception model, it also includes performing image denoising and image enhancement processing on the first environment image data.

[0021] In one embodiment, the pixel coordinates and depth values ​​of all two-dimensional vector elements in the first environment image data are converted from the image coordinate system to the world coordinate system, specifically:

[0022] The two-dimensional vector elements in the first environment image data are described as a sequence point set:

[0023] {(u 1 , v 1 , Zc 1 ), (u 2 , v 2 , Zc 2 ),…,(u n , v n , Zc n )}

[0024] Where n is the number of pixels corresponding to the two-dimensional vector elements in the first environment image data, u 1 ~u n is the horizontal coordinate of the pixel point corresponding to the two-dimensional vector element, v 1 ~v n Zc is the vertical coordinate of the pixel corresponding to the two-dimensional vector element. 1 ~Zc n is the depth value of the pixel corresponding to the two-dimensional vector element;

[0025] Based on the intrinsic parameters of the first camera, each point in the sequence point set is back-projected from the image coordinate system to the first camera coordinate system to obtain:

[0026] {(Xc 1 , Yc 1 , Zc 1 ), (Xc 2 , Yc 2 , Zc 2 ),…,(Xc n , Yc n , Zc n )}

[0027] Among them, Xc 1 ~Xc n is the X-axis coordinate of each point in the sequence point set in the first camera coordinate system, Yc 1 ~Yc n is the Y-axis coordinate of each point in the sequence point set in the first camera coordinate system;

[0028] Based on the external parameters between the first camera and the laser radar, each point in the sequence point set is converted from the first camera coordinate system to the laser radar coordinate system to obtain:

[0029] {(Xv 1 , Yv 1 , Zv 1 ), (Xv 2 , Yv 2 , Zv 2 ),…,(Xv n , Yv n , Zv n )}

[0030] Among them, Xv 1 ~Xv n is the X-axis coordinate of each point in the sequence point set in the laser radar coordinate system, Yv 1 ~Yv n is the Y-axis coordinate of each point in the sequence point set in the laser radar coordinate system, Zv 1 ~Zv n is the Z-axis coordinate of each point in the sequence point set in the laser radar coordinate system;

[0031] Based on the position of the laser radar in the global coordinate system, each point in the sequence point set is converted from the laser radar coordinate system to the world coordinate system to obtain:

[0032] {(Xw 1 , Y 1 , Zw 1 ), (Xw 2 , Y 2 , Zw 2 ),…,(Xw n , Y n , Zw n )}

[0033] Among them, Xw 1 ~Xw n is the X-axis coordinate of the three-dimensional vector element, Yw 1 ~Yw n is the Y-axis coordinate of the three-dimensional vector element, Zw 1 ~Zw n It is the Z-axis coordinate of the three-dimensional vector element.

[0034] In one embodiment, the reverse link comprises:

[0035] When a second carrier equipped with a second camera and a second positioning sensor is traveling in a target map scene, second environment image data of the target map scene and position information of the second carrier are synchronously collected;

[0036] Based on the position information of the second carrier, extracting a set of three-dimensional vector feature points corresponding to the second environment image data in the three-dimensional vector map;

[0037] Converting the extracted three-dimensional vector feature point set to the image coordinate system of the second camera, and combining it with the second environment image data to obtain the second environment image data with annotation information;

[0038] The deep learning perception model is trained using the feedback of the second environment image data with annotated information.

[0039] In one of the embodiments, before using the second environment image data with labeled information as feedback to train the deep learning perception model, it also includes data cleaning of the second environment image data with labeled information to remove elements that display incomplete images and erroneous, inaccurate labeling information.

[0040] To achieve the above object, the present invention also provides a three-dimensional vector map generation device based on data closed-loop optimization, which uses the above method to generate a three-dimensional vector map;

[0041] The three-dimensional vector map generating device includes a forward link module and a reverse link module;

[0042] In the forward link module, after obtaining the three-dimensional point cloud map and the first environment image data of the target map scene, two-dimensional vector elements are extracted from the first environment image data based on the deep learning perception model, and the two-dimensional vector elements are converted into three-dimensional vector elements by associating the first environment image data with the three-dimensional point cloud map, and finally a three-dimensional vector map is formed;

[0043] In the reverse link module, the real-time combination of positioning results and back-projection of three-dimensional vector elements in the three-dimensional vector map is used to generate image information with two-dimensional vector annotations, and used for feedback training of the deep learning perception model.

[0044] To achieve the above object, the present invention further provides a terminal device, wherein the terminal device is provided with:

[0045] Memory, used to store programs;

[0046] The processor is used to execute the program stored in the memory. When the program is executed, the processor is used to execute the method as described above.

[0047] To achieve the above object, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions; when the computer-executable instructions are executed by a processor, they are used to implement the above method.

[0048] Compared with the prior art, the present invention has the following beneficial technical effects:

[0049] 1. The present invention can generate three-dimensional vector elements using the perceived and recognized two-dimensional vector elements by combining information such as three-dimensional point cloud maps, carrier positioning information, and environmental image data, thereby solving the problem that the lack of depth information in image data makes it difficult to project two-dimensional vector elements into three-dimensional vector elements;

[0050] 2. The present invention forms a complete closed loop of data collection, processing, conversion and feedback in the process of generating a three-dimensional vector map by setting up a forward link and a reverse link respectively, which can continuously optimize and improve the accuracy and reliability of the deep learning perception model and ensure the consistency and quality of the three-dimensional vector map and model training data;

[0051] 3. The present invention generates annotation information by back-projecting the three-dimensional vector elements in the three-dimensional vector map into a two-dimensional image, which can automatically generate a large amount of training data with precise annotations, significantly reduce the workload of manual annotation, and improve the training efficiency of the deep learning perception model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0053] Figure 1 is a flow chart of a method for generating a three-dimensional vector map in an embodiment of the present invention;

[0054] Figure 2 A flow chart of an implementation method of a forward link in an embodiment of the present invention;

[0055] Figure 3 A flowchart of an implementation method of a reverse link in an embodiment of the present invention;

[0056] Figure 4 It is a structural block diagram of a three-dimensional vector map generating device in an embodiment of the present invention;

[0057] Figure 5 4 is a structural block diagram of a terminal device in an embodiment of the present invention.

[0058] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0061] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0062] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0063] Example 1

[0064] This embodiment discloses a method for generating a three-dimensional vector map based on data closed-loop optimization, which realizes data closed-loop and optimization through two links, forward and reverse. The forward link generates a three-dimensional vector map using a deep learning perception model and a point cloud map with consistent global coordinates. The reverse link performs real-time positioning in the map through other vehicle carriers equipped with positioning sensors and cameras, and generates image information with two-dimensional vector annotations through back projection by combining the positioning results and three-dimensional vector elements. The image information is used for feedback training of the deep learning perception model.

[0065] refer to Figure 1 The three-dimensional vector map generation method based on data closed-loop optimization in this embodiment specifically includes the following steps:

[0066] In the forward link:

[0067] First, a three-dimensional point cloud map and first environment image data of a target map scene are obtained, wherein the three-dimensional point cloud map can be generated in real time using information such as three-dimensional point cloud data collected synchronously with the first environment image data, or can be directly obtained in advance from an external data source;

[0068] Secondly, extracting two-dimensional vector elements from the first environment image data based on a pre-trained deep learning perception model, wherein the deep learning perception model can be constructed using a convolutional neural network;

[0069] Finally, after obtaining the depth information of each pixel point in the first environment image data by associating the first environment image data with the three-dimensional point cloud map, the two-dimensional vector elements are converted into three-dimensional vector elements based on coordinate transformation to form a three-dimensional vector map;

[0070] In the reverse link:

[0071] First, the carrier equipped with positioning sensors and cameras performs real-time positioning in the target map scene, and collects image data of the target map scene;

[0072] Then, the three-dimensional vector elements in the area corresponding to the positioning result in the three-dimensional vector map are back-projected to generate image information with two-dimensional vector annotations, and used to feedback train the deep learning perception model. The deep learning perception model after feedback training can be used again for two-dimensional vector element extraction in the forward link.

[0073] In this embodiment, the forward link is mainly used to realize the generation of a three-dimensional vector map. Taking the use of the three-dimensional point cloud data and other information collected synchronously with the first environment image data to generate a three-dimensional point cloud map in real time as an example, refer to Figure 2 , the specific implementation process of the forward link includes the following steps:

[0074] Step 101, prepare a first carrier equipped with a laser radar, a first camera, and a first positioning sensor such as a GNSS and an IMU, obtain the internal parameters of the first camera and the installation external parameters between the laser radar, the first camera, and the first positioning sensor, verify the accuracy of each internal and external parameter, and complete the time synchronization between the sensors;

[0075] Step 102, while the first carrier is driving in the target map scene, synchronously collecting the three-dimensional point cloud data of the target map scene, the first environment image data and the position and posture information of the first carrier;

[0076] Step 103, using a SLAM (Simultaneous Localization and Mapping) method, based on the three-dimensional point cloud data collected in the target map scene and the position information of the first carrier, a globally consistent three-dimensional point cloud map is generated, and the three-dimensional point cloud map is post-processed and smoothed to remove noise and errors, thereby improving map accuracy;

[0077] Step 104, performing pre-processing such as denoising and enhancement on the collected first environment image data, and then extracting two-dimensional vector elements such as lane lines and road boundaries from the first environment image data using a pre-trained deep learning perception model, wherein the two-dimensional vector elements such as lane lines and road boundaries are represented in the form of a set of pixel point coordinates;

[0078] Step 105, based on the external parameter transformation between the first camera and the laser radar, the global pose of the first camera in the three-dimensional point cloud map at the sampling time of the first environment image data is obtained, and the map points corresponding to the global pose area of ​​the first camera in the three-dimensional point cloud map are back-projected to the image coordinate system according to the first camera pose at that time and the first camera internal parameters, so as to obtain the depth value corresponding to each pixel point in the first environment image data, that is, the depth value of the pixel point corresponding to the two-dimensional vector elements such as lane lines and road boundaries is obtained;

[0079] Step 106, based on the intrinsic parameters of the first camera and the installation extrinsic parameters among the laser radar, the first camera, and the first positioning sensor, the pixel point coordinates and depth values ​​of all two-dimensional vector elements in the first environment image data are converted from the image coordinate system to the first camera coordinate system, the laser radar coordinate system, and the world coordinate system in sequence, so as to obtain the three-dimensional vector elements corresponding to the first environment image data, wherein the three-dimensional vector elements are represented in the form of a set of three-dimensional coordinate points;

[0080] Step 107, geometrically consistent association is performed on the three-dimensional vector elements obtained after the transformation of the multiple frames of the first environment image data. Manual verification and optimization may be performed optionally to ensure the accuracy and reliability of the map data, and a three-dimensional vector map may be obtained.

[0081] In the specific implementation process, the process of converting the pixel coordinates and depth values ​​of all two-dimensional vector elements in the first environment image data from the image coordinate system to the first camera coordinate system, the laser radar coordinate system, and the world coordinate system in sequence is as follows:

[0082] First, the two-dimensional vector elements in the first environment image data are described as the following sequence point set:

[0083] {(u 1 , v 1 , Zc 1 ), (u 2 , v 2, Zc 2 ),…,(u n , v n , Zc n )}

[0084] Where n is the number of pixels corresponding to the two-dimensional vector elements in the first environment image data, u 1 ~u n is the horizontal coordinate of the pixel point corresponding to the two-dimensional vector element, v 1 ~v n Zc is the vertical coordinate of the pixel corresponding to the two-dimensional vector element. 1 ~Zc n is the depth value of the pixel corresponding to the two-dimensional vector element;

[0085] Secondly, based on the intrinsic parameters of the first camera, each point in the sequence point set is back-projected from the image coordinate system to the first camera coordinate system to obtain:

[0086] {(Xc 1 , Yc 1 , Zc 1 ), (Xc 2 , Yc 2 , Zc 2 ),…,(Xc n , Yc n , Zc n )}

[0087] Among them, Xc 1 ~Xc n is the X-axis coordinate of each point in the sequence point set in the first camera coordinate system, Yc 1 ~Yc n is the Y-axis coordinate of each point in the sequence point set in the first camera coordinate system, and the Z-axis coordinate of each point in the sequence point set in the first camera coordinate system is the depth value of each pixel;

[0088] Then, based on the external parameters between the first camera and the laser radar, each point in the sequence point set is transformed from the first camera coordinate system to the laser radar coordinate system to obtain:

[0089] {(Xv 1 , Yv 1 , Zv 1 ), (Xv 2 , Yv 2 , Zv 2 ),…,(Xv n , Yv n , Zv n )}

[0090] Among them, Xv 1 ~Xv nis the X-axis coordinate of each point in the sequence point set in the laser radar coordinate system, Yv 1 ~Yv n is the Y-axis coordinate of each point in the sequence point set in the laser radar coordinate system, Zv 1 ~Zv n is the Z-axis coordinate of each point in the sequence point set in the laser radar coordinate system;

[0091] Finally, based on the position of the lidar in the global coordinate system, each point in the sequence point set is converted from the lidar coordinate system to the world coordinate system to obtain:

[0092] {(Xw 1 , Y 1 , Zw 1 ), (Xw 2 , Y 2 , Zw 2 ),…,(Xw n , Y n , Zw n )}

[0093] Three-dimensional vector elements such as lane lines and road boundaries in three-dimensional space can be represented as the coordinates of each point in the sequence point set in the world coordinate system, that is, Xw 1 ~Xw n is the X-axis coordinate of the three-dimensional vector element, Yw 1 ~Yw n is the Y-axis coordinate of the three-dimensional vector element, Zw 1 ~Zw n It is the Z-axis coordinate of the three-dimensional vector element.

[0094] In this embodiment, the reverse link is mainly used to perform real-time positioning in the target map scene corresponding to the three-dimensional vector map by installing sensors such as GNSS or laser radar that can provide high-precision positioning of the vehicle and camera sensors after the three-dimensional vector map is generated in the forward link, and image information with two-dimensional vector annotations is generated by back-projecting the three-dimensional vector elements in the map for feedback perception model training. Figure 3 , the specific implementation process of the reverse link includes the following steps:

[0095] Step 201, install a single or multiple second cameras on the second carrier, and deploy a second positioning sensor that can provide high-precision positioning and attitude information on the second carrier, such as a laser radar or a GNSS sensor, etc., to obtain the internal parameters of the second camera and the installation external parameters between the second camera and the second positioning sensor, and verify the accuracy of the internal and external parameters to complete the time synchronization of the second camera and the second positioning sensor;

[0096] Step 202, when the second carrier is traveling in the target map scene of the existing three-dimensional vector map, synchronously collecting the second environment image data of the target map scene and the position and posture information of the second carrier;

[0097] Step 203, based on the position information of the second carrier, extract the three-dimensional vector elements of the area around the position information of the second carrier in the three-dimensional vector map, so as to obtain a set of three-dimensional vector element points corresponding to the second environment image data;

[0098] Step 204, converting the extracted three-dimensional vector feature point set to the image coordinate system of the second camera, and combining it with the second environment image data to obtain the second environment image data with annotation information, wherein the process of converting the three-dimensional vector feature point set to the image coordinate system of the second camera is opposite to the specific implementation of the aforementioned step 106, so it will not be described in detail in this embodiment;

[0099] Step 205, clean the second environment image data with the labeled information to remove incomplete images and erroneous and inaccurate labeling information, and then use the cleaned second environment image data with the labeled information to feed back and train the deep learning perception model.

[0100] As a preferred implementation mode, diversified two-dimensional annotated image data can be obtained by adjusting the model and installation position of the second camera on the second carrier. The multi-perspective and diversified annotated data can be used for deep learning perception model training to continuously optimize and improve the accuracy and reliability of the deep learning perception model, and improve the element perception and reasoning capabilities of the deep learning perception model in the forward link, thereby forming positive feedback and a data closed loop.

[0101] It is worth noting that although this embodiment Figures 1 to 3 The steps in the process are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Figures 1 to 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0102] Example 2

[0103] Based on the three-dimensional vector map generation method in Example 1, this embodiment discloses a three-dimensional vector map generation device based on data closed-loop optimization, referring to Figure 4 The three-dimensional vector map generating device includes a forward link module and a reverse link module.

[0104] The forward link module is composed of a three-dimensional point cloud map acquisition unit, a first image data acquisition unit, a deep learning perception model and a three-dimensional vector map construction unit. Specifically: after the three-dimensional point cloud map acquisition unit and the first image data acquisition unit respectively acquire the three-dimensional point cloud map and the first environmental image data of the target map scene, the deep learning perception model extracts two-dimensional vector elements from the first environmental image data, associates the first environmental image data with the three-dimensional point cloud map through the three-dimensional vector map construction unit, and then converts the two-dimensional vector elements into three-dimensional vector elements, and finally forms a three-dimensional vector map.

[0105] The reverse link module consists of a labeling data generation unit and a feedback training unit. Specifically: the labeling data generation unit generates image information with two-dimensional vector annotations by combining the positioning results with the back projection of the three-dimensional vector elements in the three-dimensional vector map in real time. The feedback training unit then uses the image information with two-dimensional vector annotations to feedback train the deep learning perception model.

[0106] In this embodiment, the specific working process and working principle of the forward link module and the reverse link module are the same as those in the method of Embodiment 1, so they are not described in detail in this embodiment. Each unit module can be implemented in whole or in part by software, hardware, or a combination thereof. Each unit module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above unit modules.

[0107] Example 3

[0108] like Figure 5 The terminal device disclosed in this embodiment includes a transmitter, a receiver, a memory, and a processor. The transmitter is used to send instructions and data, the receiver is used to receive instructions and data, the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions stored in the memory to implement the method in the above-mentioned embodiment 1.

[0109] It should be noted that the above memory can be independent or integrated with the processor. When the memory is independently provided, the terminal device further includes a bus for connecting the memory and the processor.

[0110] Example 4

[0111] This embodiment discloses a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method in the above-mentioned embodiment 1 is implemented.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0113] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A three-dimensional vector map generation method based on data closed-loop optimization, characterized in that: Including forward link and reverse link; In the forward link: Acquire a three-dimensional point cloud map and first environment image data of a target map scene, and extract two-dimensional vector elements from the first environment image data based on a deep learning perception model; By associating the first environment image data with the three-dimensional point cloud map, the two-dimensional vector elements are converted into three-dimensional vector elements to form a three-dimensional vector map; In the reverse link: Based on the real-time positioning of the carrier equipped with positioning sensors and cameras on the three-dimensional vector map, the positioning results are combined with the back projection of the three-dimensional vector elements in the three-dimensional vector map to generate image information with two-dimensional vector annotations, which is used to feed back the training of the deep learning perception model to complete the data closed loop and optimization; The forward link comprises: When the first carrier equipped with the laser radar, the first camera, and the first positioning sensor is driving in the target map scene, three-dimensional point cloud data of the target map scene, first environment image data, and position information of the first carrier are synchronously collected; Generate a globally consistent three-dimensional point cloud map based on the three-dimensional point cloud data of the target map scene and the position information of the first carrier; Extracting two-dimensional vector elements from the first environment image data based on the deep learning perception model, wherein the two-dimensional vector elements are represented in the form of a set of pixel point coordinates; Based on the external parameter transformation between the first camera and the laser radar, the global pose of the first camera in the three-dimensional point cloud map at the sampling time of the first environment image data is obtained, and the map points of the corresponding area in the three-dimensional point cloud map are back-projected to the image coordinate system according to the first camera pose and the first camera internal parameters at that time, so as to obtain the depth value corresponding to each pixel point in the first environment image data; Convert pixel point coordinates and depth values ​​of all two-dimensional vector elements in the first environment image data from the image coordinate system to the world coordinate system to obtain three-dimensional vector elements corresponding to the first environment image data, wherein the three-dimensional vector elements are represented in the form of a set of three-dimensional coordinate points; By geometrically associating the three-dimensional vector elements obtained after transforming multiple frames of first environment image data, a three-dimensional vector map can be obtained.

2. The method for generating a three-dimensional vector map based on data closed-loop optimization according to claim 1, characterized in that: After generating a globally consistent three-dimensional point cloud map, the three-dimensional point cloud map is also post-smoothed to remove noise and errors and improve map accuracy.

3. The method for generating a three-dimensional vector map based on data closed-loop optimization according to claim 1, characterized in that: Before extracting two-dimensional vector elements from the first environment image data based on the deep learning perception model, it also includes image denoising and image enhancement processing on the first environment image data.

4. The method for generating a three-dimensional vector map based on data closed-loop optimization according to claim 1, characterized in that: The pixel coordinates and depth values ​​of all two-dimensional vector elements in the first environment image data are converted from the image coordinate system to the world coordinate system, specifically: The two-dimensional vector elements in the first environment image data are described as a sequence point set: in, is the number of pixels corresponding to the two-dimensional vector elements in the first environment image data, is the horizontal coordinate of the pixel point corresponding to the two-dimensional vector element, is the vertical coordinate of the pixel point corresponding to the two-dimensional vector element, is the depth value of the pixel corresponding to the two-dimensional vector element; Based on the intrinsic parameters of the first camera, each point in the sequence point set is back-projected from the image coordinate system to the first camera coordinate system to obtain: in, is the X-axis coordinate of each point in the sequence point set in the first camera coordinate system, is the Y-axis coordinate of each point in the sequence point set in the first camera coordinate system; Based on the external parameters between the first camera and the laser radar, each point in the sequence point set is converted from the first camera coordinate system to the laser radar coordinate system to obtain: in, is the X-axis coordinate of each point in the sequence point set in the laser radar coordinate system, is the Y-axis coordinate of each point in the sequence point set in the laser radar coordinate system, is the Z-axis coordinate of each point in the sequence point set in the laser radar coordinate system; Based on the position of the laser radar in the global coordinate system, each point in the sequence point set is converted from the laser radar coordinate system to the world coordinate system to obtain: in, is the X-axis coordinate of the three-dimensional vector element, is the Y-axis coordinate of the three-dimensional vector element, It is the Z-axis coordinate of the three-dimensional vector element.

5. The method for generating a three-dimensional vector map based on data closed-loop optimization according to any one of claims 1 to 4, characterized in that: The reverse link includes: When a second carrier equipped with a second camera and a second positioning sensor is traveling in a target map scene, second environment image data of the target map scene and position information of the second carrier are synchronously collected; Based on the position information of the second carrier, extracting a set of three-dimensional vector feature points corresponding to the second environment image data in the three-dimensional vector map; Converting the extracted three-dimensional vector feature point set to the image coordinate system of the second camera, and combining it with the second environment image data to obtain the second environment image data with annotation information; The deep learning perception model is trained using the feedback of the second environment image data with annotated information.

6. The method for generating a three-dimensional vector map based on data closed-loop optimization according to claim 5, characterized in that: Before using the second environment image data with labeled information as feedback to train the deep learning perception model, it also includes data cleaning of the second environment image data with labeled information to remove incomplete images of elements and erroneous and inaccurate labeling information.

7. A three-dimensional vector map generation device based on data closed-loop optimization, characterized in that: Generating a three-dimensional vector map using the method described in any one of claims 1 to 6; The three-dimensional vector map generating device includes a forward link module and a reverse link module; In the forward link module, after obtaining the three-dimensional point cloud map and the first environment image data of the target map scene, two-dimensional vector elements are extracted from the first environment image data based on the deep learning perception model, and the two-dimensional vector elements are converted into three-dimensional vector elements by associating the first environment image data with the three-dimensional point cloud map, and finally a three-dimensional vector map is formed; In the reverse link module, the real-time combination of positioning results and back-projection of three-dimensional vector elements in the three-dimensional vector map is used to generate image information with two-dimensional vector annotations, and used for feedback training of the deep learning perception model.

8. A terminal device, characterized in that: The terminal device is provided with: Memory, used to store programs; A processor, configured to execute the program stored in the memory; when the program is executed, the processor is configured to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions; when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 6.

Images