Data processing method

Abstract

The embodiment of the invention provides a data processing method. The data processing method comprises the following steps: acquiring three-dimensional point cloud data and two-dimensional image data associated with mobile equipment; calibrating the three-dimensional point cloud data and the two-dimensional image data to obtain target three-dimensional point cloud data and target two-dimensional image data; optimizing a plurality of historical laser radar sub-graphs associated with the mobile device by using the target three-dimensional point cloud data, and generating a target laser radar sub-graph according to an optimization result; and mapping the target two-dimensional image data to the target laser radar sub-graph, and constructing driving data for automatic driving of the mobile equipment according to a mapping result.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of autonomous driving technology, and in particular, to data processing methods. Background Art

[0002] With the development of computer and Internet technology, autonomous driving technology has become more and more mature. It can effectively improve the driving experience by replacing or alleviating the driver's driving operation. In the prior art, view synthesis and world model technology are mainly applied in the development of autonomous driving systems in two ways. One is to promote the transmission of trained perception or end-to-end data between different vehicle products and sensors; the other is to generate sensor frames with real geometry and appearance from different perspectives for closed-loop simulation, especially in sensor-to-control scenarios. However, most existing autonomous driving NVS methods focus on evaluating new views based on difference quality, rather than focusing on changes in lateral viewpoints, i.e., cross-lane NVS. This leads to the simulation platform in the prior art mainly verifying the effectiveness of test strategy changes in longitudinal (speed) behavior and motion planning, but having low interpretability in lateral (path) planning evaluation. Ultimately, although the mobile device can achieve the purpose of autonomous driving under the drive of self-driving technology, there is still a certain error between it and the expected driving effect, so an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, an embodiment of this specification provides a data processing method. One or more embodiments of this specification also relate to a data processing device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a data processing method is provided, including: Collecting 3D point cloud data and 2D image data associated with a mobile device; Calibrate the three-dimensional point cloud data and the two-dimensional image data to obtain target three-dimensional point cloud data and target two-dimensional image data; Optimizing multiple historical lidar sub-images associated with the mobile device using the target three-dimensional point cloud data, and generating a target lidar sub-image according to the optimization result; The target two-dimensional image data is mapped to the target lidar sub-image, and driving data for autonomous driving of the mobile device is constructed according to the mapping result.

[0005] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including: A collection module configured to collect three-dimensional point cloud data and two-dimensional image data associated with a mobile device; A calibration module is configured to calibrate the three-dimensional point cloud data and the two-dimensional image data to obtain target three-dimensional point cloud data and target two-dimensional image data; an optimization module, configured to optimize a plurality of historical lidar sub-images associated with the mobile device using the target three-dimensional point cloud data, and generate a target lidar sub-image according to the optimization result; A mapping module is configured to map the target two-dimensional image data to the target lidar sub-image, and construct driving data for automatic driving of the mobile device according to the mapping result.

[0006] According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above data processing method are implemented.

[0007] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0008] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0009] The data processing method provided in this embodiment can first collect the three-dimensional point cloud data and two-dimensional image data associated with the mobile device in order to realize the ability of automatic driving technology to evaluate the perspective synthesis based on real data, thereby reducing the impact of automatic driving caused by errors. Considering that there may be some special weather effects in the actual driving environment, although the collected data is real, it may affect safety if the impact of changes in the external driving environment is not considered when used for driving. Therefore, the three-dimensional point cloud data and the two-dimensional image data can be calibrated to obtain the calibrated target three-dimensional point cloud data and target two-dimensional image data. On this basis, the target three-dimensional point cloud data can be used to optimize multiple historical laser radar sub-images associated with the mobile device, and the target laser radar sub-image is generated according to the optimization results. In this way, a radar sub-image matching the current driving scene is constructed, and the error impact caused by the driving environment is integrated. Then the target two-dimensional image data can be mapped to the target laser radar sub-image, and then the driving data for the automatic driving of the mobile device can be constructed according to the mapping results, and when the three-dimensional point cloud data and the two-dimensional image data are aligned, the influence of external factors can be considered to reduce the error, thereby ensuring the authenticity of the data to meet the use of automatic driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of a data processing method provided by an embodiment of this specification; Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification; Figure 3 is a processing flow chart of a data processing method provided by an embodiment of this specification; Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of this specification; Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0011] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0012] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0013] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0014] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0015] First, the terms involved in one or more embodiments of this specification are explained.

[0016] Autonomous driving: Autonomous driving means that through various sensors (such as lidar, cameras, radar, etc.) and artificial intelligence technology, the vehicle can autonomously perceive the environment, understand road conditions, and make driving decisions, thereby completing driving tasks without a human driver. Autonomous driving technology is usually divided into multiple levels, from fully manual driving to fully autonomous driving (Level 0 to Level 5).

[0017] World model: A world model is an internal model used to represent and understand the environment, and is commonly used in robotics and autonomous driving systems. It creates a dynamic and accurate representation of the environment by fusing various sensor data (such as vision, laser, and IMU) to help the system plan and make decisions. The world model usually includes the location, state, and relationship information of objects.

[0018] Closed-loop simulation: Closed-loop simulation means that in a simulation system, the output of the system can be fed back to the input to form a feedback control loop. This simulation method is often used in fields such as autonomous driving to test and verify the effectiveness of the algorithm. Through real-time feedback, closed-loop simulation can more realistically simulate the performance of the system in actual operation, thereby improving the reliability of the algorithm.

[0019] New perspective synthesis: New perspective synthesis is a technology in the field of computer vision that generates new perspective images that have never been taken through existing image data. This technology uses methods such as deep learning and multi-view geometry to provide richer visual information and immersive experience in applications such as virtual reality, augmented reality and autonomous driving.

[0020] Simultaneous Localization and Mapping (SLAM): Simultaneous Localization and Mapping is a technology that simultaneously handles the problems of position estimation and environment mapping. SLAM systems usually work in unknown environments and track their own positions and map the surrounding environment by analyzing sensor data (such as lidar, cameras, etc.) in real time. SLAM has a wide range of applications in robot navigation, autonomous vehicles, virtual reality and other fields.

[0021] NVS: (Navigation and Vehicle Surveillance) is a comprehensive system that integrates navigation technology and vehicle monitoring functions.

[0022] LiDAR (Light Detection and Ranging) map, also known as laser radar map, is a high-precision three-dimensional map created using laser radar technology. This map uses the laser radar sensor to emit laser pulses and receive reflected signals to measure the distance, position and shape of the target object, thereby generating a detailed three-dimensional terrain and object model.

[0023] Odometry: Odometry is a technique or device that measures the distance traveled by an object, especially a vehicle, by using data from motion sensors to estimate the change in position over time.

[0024] LOAM: (Lidar Odometry and Mapping) is a highly efficient real-time SLAM (Simultaneous Localization and Mapping) technology.

[0025] GICP (Generalized-ICP) registration, or generalized iterative near-point registration, is a point cloud registration method. GICP registration unifies the traditional Point-To-Point and Point-To-Plane ICP under a probabilistic framework, and further expands it to Plane-to-Plane ICP. It introduces probabilistic information, uses the covariance matrix, and proposes a unified model of ICP. GICP re-examines and derives the objective function of the ICP algorithm from a probabilistic perspective. Specifically, assume that there are two matched point sets A and B, and each point obeys a Gaussian distribution. GICP uses the point-to-point distance as the loss function to solve the point-to-point loss function, and the point-to-local target point local fitting plane distance as the point-to-plane loss function, and also introduces the plane-to-plane loss function.

[0026] Loop Closure Optimization is an important concept in SLAM (Simultaneous Localization and Mapping), which aims to eliminate cumulative errors and improve the accuracy of positioning and mapping.

[0027] SuperGlue: is an image matching method that uses an attention layer for information propagation. It is a network model that combines a graph neural network (GNN) and an optimal matching layer.

[0028] RTK: (Real-Time Kinematic) is a high-precision positioning technology based on differential GPS. Through real-time communication and data processing, it can provide centimeter-level or even sub-meter-level positioning accuracy.

[0029] INS: Inertial Navigation System is an autonomous navigation system that does not rely on external information and does not radiate energy to the outside. It is based on Newton's laws of mechanics and measures the acceleration of the carrier in the inertial reference system, integrates it over time, and transforms it into the navigation coordinate system to obtain information such as the carrier's speed, yaw angle, and position.

[0030] In this specification, a data processing method is provided. This specification also relates to a data processing device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0031] In practice, it is inherently very difficult to collect real data for multiple lanes in real scenes, so simulation platforms are chosen to present synthetic data with parameters, such as internal parameters, external parameters, rolling shutter effects, and sensor frame poses. They have two main limitations in further use: First, creating synthetic scenes using high-definition mesh models and materials requires meticulous adjustments for artistry and consistent shading. This process is costly and must be completed before the efficient generation of realistic frames for evaluating driving failure cases. Second, there are still some challenging realistic issues, such as the inherent noise of real sensors, and dynamic effects caused by fog, wind, and fluids, which lead to artifacts and domain migration costs. Therefore, even though sensor data obtained from real recordings may have problems with imprecise parameters, they are still an indispensable part in evaluating the quality of cross-lane NVS. If you want to collect data in a single shot, you need an oversized structure to rigidly fix multiple cameras, and the width of a typical circular lane is about 3-4 meters. It is very complicated to design and manufacture such a structure with stable dynamics. Therefore, an effective solution is urgently needed to solve the above problems.

[0032] See also Figure 1 As shown in the schematic diagram, the data processing method provided in this embodiment can first collect the three-dimensional point cloud data and two-dimensional image data associated with the mobile device in order to realize the automatic driving technology to evaluate the view synthesis capability based on real data, thereby reducing the impact of automatic driving caused by errors. Considering that there may be some special weather influences in the actual driving environment, although the collected data is real, it may affect safety if the changes in the external driving environment are not considered when used for driving. Therefore, the three-dimensional point cloud data and the two-dimensional image data can be calibrated to obtain the calibrated target three-dimensional point cloud data and target two-dimensional image data; on this basis, the target three-dimensional point cloud data can be used to optimize the multiple historical laser radar sub-images associated with the mobile device, and the target laser radar sub-image is generated according to the optimization results; thereby, a radar sub-image matching the current driving scene is constructed, and the error influence caused by the driving environment is integrated. Then, the target two-dimensional image data can be mapped to the target laser radar sub-image, and then the driving data for the automatic driving of the mobile device can be constructed according to the mapping results, and when the three-dimensional point cloud data and the two-dimensional image data are aligned, the influence of external factors can be considered to reduce the error, thereby ensuring the authenticity of the data to meet the use of automatic driving.

[0033] See also Figure 2 , Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0034] Step S202, collecting three-dimensional point cloud data and two-dimensional image data associated with the mobile device.

[0035] The data processing method provided in this embodiment can be applied to any mobile device to perform SLAM algorithm optimization processing on the results of multiple data collection in an autonomous driving scenario, where the mobile devices include but are not limited to user-driven vehicles, unmanned delivery vehicles, autonomous driving buses, autonomous driving trucks, industrial robots, shelf handling robots, sorting robots, sweeping robots, etc. By installing laser sensors, cameras, inertial sensors, etc. on the mobile devices, parallel multiple road data collection and desensitization are performed, and then the data is processed by the data processing method provided in this embodiment, thereby achieving the purpose of SLAM algorithm optimization, so as to obtain driving data for autonomous driving of mobile devices for autonomous driving operations of mobile devices.

[0036] This embodiment uses the mobile device as an unmanned delivery vehicle as an example to illustrate the data processing method. The data processing methods corresponding to mobile devices in other autonomous driving scenarios can refer to the same or corresponding description content in this embodiment, and this embodiment will not be elaborated in detail here.

[0037] Specifically, the three-dimensional point cloud data refers to the point cloud data collected by the lidar sensor carried by the mobile device. Correspondingly, the two-dimensional image data refers to the image data collected by the camera carried by the mobile device. By subsequently calibrating and aligning the three-dimensional point cloud data and the two-dimensional image data, it can be ensured that they can be used for autonomous driving while eliminating errors, thereby ensuring the safety and accuracy of autonomous driving.

[0038] Based on this, in order to realize the ability of autonomous driving technology to evaluate the perspective synthesis based on real data, thereby reducing the impact of autonomous driving caused by errors, the three-dimensional point cloud data and two-dimensional image data associated with the mobile device can be collected first; considering that there may be some special weather effects in the actual driving environment, the collected data is real, but if the changes in the external driving environment are not considered when used for driving, it may affect safety; therefore, the three-dimensional point cloud data and two-dimensional image data can be calibrated to obtain the calibrated target three-dimensional point cloud data and target two-dimensional image data; on this basis, the target three-dimensional point cloud data can be used to optimize multiple historical laser radar sub-images associated with the mobile device, and the target laser radar sub-image is generated according to the optimization results; in this way, a radar sub-image matching the current driving scene is constructed, and the error impact caused by the driving environment is integrated. Then the target two-dimensional image data can be mapped to the target laser radar sub-image, and then the driving data for the mobile device's autonomous driving can be constructed according to the mapping results, and when the three-dimensional point cloud data and the two-dimensional image data are aligned, the influence of external factors can be considered to reduce the error, thereby ensuring the authenticity of the data to meet the use of autonomous driving.

[0039] Step S204 , calibrating the three-dimensional point cloud data and the two-dimensional image data to obtain target three-dimensional point cloud data and target two-dimensional image data.

[0040] Specifically, after obtaining the three-dimensional point cloud data and two-dimensional image data as mentioned above, considering that the three-dimensional point cloud data and two-dimensional image data are the initial real data collected by the sensors of the mobile device, which do not take into account the impact of the actual driving environment, after obtaining the above data, the three-dimensional point cloud data and two-dimensional image data can be calibrated according to the influencing factors of the actual driving environment, so as to eliminate some influencing errors through preliminary calibration operations, and then obtain the target three-dimensional point cloud data and target two-dimensional image data, and subsequently perform visual-laser posture alignment processing based on the target three-dimensional point cloud data and target two-dimensional image data.

[0041] Among them, the target three-dimensional point cloud data and target two-dimensional image data specifically refer to the three-dimensional point cloud data and two-dimensional image data that are calibrated according to the external calibration results of the sensor of the mobile device to eliminate the external parameter errors.

[0042] Furthermore, when calibrating the three-dimensional point cloud data and the two-dimensional image data, in order to ensure the calibration accuracy, point-to-point calibration can be used. In this embodiment, the specific implementation method is as follows: Determine target feature point data in the two-dimensional image data, and extract target point cloud data corresponding to the target feature point data in the three-dimensional point cloud data; perform point-to-point calibration on the three-dimensional point cloud data and the two-dimensional image data based on the target feature point data and the target point cloud data; determine target three-dimensional point cloud data and target two-dimensional image data according to the calibration result.

[0043] Specifically, the target feature point data specifically refers to the feature point data specified in the two-dimensional image data. Correspondingly, the target point cloud data is the point cloud data corresponding to the target feature point data. Correspondingly, the point-to-point calibration specifically refers to the operation of performing calibration processing according to the points.

[0044] Based on this, in order to improve the calibration accuracy of three-dimensional point cloud data and two-dimensional image data, the target feature point data can be first determined in the two-dimensional image data, and at the same time, the target point cloud data corresponding to the target feature point data can be extracted from the three-dimensional point cloud data; considering that the target feature point data and the target point cloud data obtained at this time have a corresponding relationship, the three-dimensional point cloud data and the two-dimensional image data can be calibrated point-to-point based on the target feature point data and the target point cloud data; and then the target three-dimensional point cloud data and the target two-dimensional image data can be determined according to the calibration results.

[0045] In specific implementation, the calibration of 3D point cloud data and 2D image data is actually a data correction process based on the results of multi-sensor external parameter calibration. In practical applications, in order to accurately solve scene information using multiple sensors, it is necessary to calibrate the external parameters of multiple sensors on board. The purpose is to be able to effectively align the 3D point cloud obtained by laser measurement with the 2D image obtained by the camera, so as to achieve more accurate environmental perception and scene reconstruction. In specific implementation, it can be achieved through the following processing: (1) Material preparation: Make sure the laser sensor and camera are installed and functioning properly. Confirm the fixed relationship between the devices. At the same time, prepare a standard calibration plate (such as a chessboard or Apri-lTag) with known geometric features.

[0046] (2) Data acquisition: The laser sensor and camera work synchronously to acquire data in the same scene. Usually, the laser sensor is scanned across the calibration plate while the camera is used to capture the image of the calibration plate, while ensuring that the collected point cloud data and image data are aligned, while ensuring that the sensor combination remains rigidly connected.

[0047] (3) Information processing: Process the captured camera images and extract the feature points (such as corner points) on the calibration plate. Extract the three-dimensional point cloud data corresponding to the calibration plate from the laser scan. By combining the information of the two modalities of the laser sensor and the camera, point-to-point matching is performed using the extracted feature points, so that the preliminary external parameter estimation can be completed. After that, the iterative optimization algorithm (such as the nonlinear least squares method) can be used to minimize the error between the laser point cloud data and the image feature points, thereby obtaining more accurate external parameters.

[0048] Based on this, the two-dimensional image data and the three-dimensional point cloud data are calibrated to obtain the target three-dimensional point cloud data and the target two-dimensional image data with some errors eliminated. In specific implementation, the images and laser point clouds of the validation set can also be used to check the accuracy of the external parameter calibration results, so that the registration results can be judged to decide whether to calibrate the collected two-dimensional image data and three-dimensional point cloud data.

[0049] In summary, by calibrating the three-dimensional point cloud data and the two-dimensional image data in a point-to-point manner, the two can have higher accuracy before visual and laser alignment, thus providing a basis for subsequent processing.

[0050] Step S206, optimizing multiple historical lidar sub-images associated with the mobile device using the target three-dimensional point cloud data, and generating a target lidar sub-image according to the optimization result.

[0051] Specifically, after obtaining the target three-dimensional point cloud data and target two-dimensional image data determined after calibration as mentioned above, further, in order to provide an accurate target lidar sub-image as the basis for mapping image data in the laser and vision alignment processing, the target three-dimensional point cloud data can be used to optimize multiple historical lidar sub-images associated with the mobile device, so as to eliminate the accumulated errors caused by the odometer process according to the optimization results, and then generate the target lidar sub-image according to the optimization results for subsequent use.

[0052] The multiple historical LiDAR sub-maps specifically refer to the LiDAR sub-maps constructed by the historical point cloud data collected by the mobile device in the previous cycle, and based on this, combined with the currently collected target three-dimensional point cloud data, the target LiDAR sub-map corresponding to the current frame can be constructed. The target LiDAR sub-map specifically refers to the LiDAR sub-map constructed by combining the point cloud data after eliminating the accumulated error of the odometer.

[0053] Furthermore, when constructing the target laser radar sub-image, in order to ensure that the constructed target laser radar sub-image can eliminate the cumulative error in the odometer process, the sub-image optimization processing operation can be completed by combining the first registration relationship and the second registration relationship. In this embodiment, the specific implementation method is as follows: Extract target point cloud data features corresponding to the target three-dimensional point cloud data, and obtain multiple historical lidar sub-images associated with the mobile device; construct a first registration relationship and a second registration relationship based on the target point cloud data features and the multiple historical lidar sub-images; use the first registration relationship and the second registration relationship to optimize the multiple historical lidar sub-images, and generate a target lidar sub-image based on the optimization results.

[0054] Specifically, the target point cloud data feature specifically refers to the LOAM feature, which refers to the vector expression corresponding to the edge feature points and plane feature points extracted from the point cloud data. Correspondingly, the multiple historical lidar sub-images specifically refer to the lidar sub-images constructed by the mobile device before this time. Correspondingly, the first registration relationship specifically refers to the association relationship used for GCIP registration processing; the second registration relationship specifically refers to the association relationship used for GICP / NDT registration processing between sub-images in the candidate lidar sub-image pair in the loop closure optimization stage, thereby improving the optimization effect.

[0055] Based on this, when constructing a target lidar sub-image, the target point cloud data features corresponding to the target three-dimensional point cloud data can be extracted first, and multiple historical lidar sub-images associated with the mobile device can be obtained; on this basis, in order to avoid the impact of errors, the first registration relationship and the second registration relationship can be constructed according to the target point cloud data features and multiple historical lidar sub-images; the odometer optimization is achieved through the first registration relationship, and the loop optimization can be achieved through the second registration relationship. Therefore, the first registration relationship and the second registration relationship can be used to optimize multiple historical lidar sub-images, and then the target lidar sub-image can be generated according to the optimization results for subsequent use.

[0056] In summary, by optimizing multiple historical lidar sub-images in combination with the first registration relationship and the second registration relationship, it is possible to ensure that the construction of the target lidar sub-image eliminates the accumulated error of the odometer, thereby further improving the construction accuracy of the target lidar sub-image.

[0057] Furthermore, considering that the first registration relationship and the second configuration relationship are the main parameters that determine whether the lidar sub-image eliminates the cumulative error, the first registration relationship and the second registration relationship can be constructed in the following way: The construction of the first registration relationship includes: performing point cloud registration processing according to the target point cloud data features and the multiple historical lidar sub-images to obtain the first registration relationship; Among them, the construction of the second registration relationship includes: using a matching model to screen at least two candidate lidar subimages from the multiple historical lidar subimages, and determining the matching scores corresponding to the at least two candidate lidar subimages respectively; determining a candidate lidar subimage pair from the at least two candidate lidar subimages according to the matching scores; and performing point cloud registration processing on the first lidar subimage and the second lidar subimage in the candidate lidar subimage pair to obtain the second registration relationship.

[0058] Specifically, the matching model specifically refers to a model for calculating the confidence of each historical lidar sub-image, and the confidence of the calculated output is the matching score. Accordingly, the candidate lidar sub-image is the lidar sub-image selected according to the matching score. Accordingly, the candidate lidar sub-image pair specifically refers to a queue of lidar sub-images with corresponding relationships selected from at least two candidate lidar sub-images.

[0059] Based on this, when constructing the first registration relationship, point cloud registration processing can be directly performed according to the target point cloud data features and multiple historical laser radar sub-images, thereby determining the first registration relationship. The second registration relationship can be realized by using a matching model, through which at least two candidate laser radar sub-images can be screened from multiple historical laser radar sub-images, and the matching scores corresponding to the at least two candidate laser radar sub-images can be determined; the matching score can represent the confidence of the laser radar sub-image being selected, so the candidate laser radar sub-image pair can be determined from the at least two candidate laser radar sub-images according to the matching score; thereafter, point cloud registration processing can be performed according to the first laser radar sub-image and the second laser radar sub-image in the candidate laser radar sub-image pair, thereby obtaining the second registration relationship.

[0060] In specific implementation, the construction of the target LiDAR sub-map is essentially the calculation of the laser dense pose. Given the initial trajectory from the RTK / INS sensor, a target LiDAR sub-map (LiDAR map) in the reference coordinate system G can be constructed through the laser SLAM algorithm. This process can be achieved through odometer and loop closure optimization.

[0061] For the odometry part, LOAM features and LiDAR sub-images accumulated over a period of time can be used for GICP registration to enhance the robustness in various mapping scenarios.

[0062] The loop optimization part can be understood as including loop extraction and pose graph optimization. Among them, for loop extraction, the matching model can be used to perform rough matching of multiple historical lidar sub-images and calculate the matching score of each sub-image. Then, the candidate lidar sub-image pairs can be screened out from multiple historical lidar sub-images by the matching score. Then, GICP registration can be performed based on the candidate lidar sub-image pairs. In this way, accurate relative constraints can be established for the pose graph optimization problem. In the pose graph optimization process, the relative constraints generated in the extraction process and the continuous registration relative constraints implemented in the above-mentioned odometer part can be reused to perform global consistency optimization on multiple historical lidar sub-images, thereby eliminating the accumulated errors in the odometer process and obtaining the final target lidar sub-image (LiDAR map).

[0063] It can be understood that, through the pose graph optimization problem, it is possible to consider both the meter-level approximate position of each sub-image obtained through satellite inertia and the fine relative position between sub-images obtained through laser point cloud registration, so as to determine the precise position corresponding to the target lidar sub-image.

[0064] In summary, by constructing the first registration relationship and the second registration relationship through the above processing, it can be ensured that the two registration relationships complete the construction and optimization of the sub-image from different granularities, thereby ensuring that the real data is not affected while eliminating errors, thereby facilitating the construction and use of the target lidar sub-image.

[0065] In addition, after obtaining the target LiDAR sub-image through the above processing, in order to verify the quality of the target LiDAR sub-image, the target LiDAR sub-image can also be detected using the reference index corresponding to the structure, thereby ensuring the subsequent use of high-quality LiDAR sub-images. In this embodiment, the specific implementation method is as follows: Determine the frame pose corresponding to the target laser radar sub-image, and construct a triangular mesh according to the frame pose; compare the triangular mesh with the spliced ​​point cloud data contained in the target laser radar sub-image, and calculate error information according to the comparison result; compare the error information with a reference index corresponding to a preset structure, and when it is determined according to the comparison result that the target laser radar sub-image meets the processing conditions, execute the step of mapping the target two-dimensional image data to the target laser radar sub-image, and construct driving data for autonomous driving of the mobile device according to the mapping result.

[0066] Specifically, the frame pose specifically refers to the position and posture corresponding to the viewing camera of the target lidar sub-image when constructing the sub-image. Correspondingly, the triangular mesh specifically refers to the mesh constructed according to the voxel size and the cut-off distance, which is used to represent the posterior most likely position of the surface of a closed object. Correspondingly, the spliced ​​point cloud data specifically refers to the point cloud data used to construct the target lidar sub-image. Correspondingly, the error information specifically refers to the error value before and after optimization. Correspondingly, the reference index specifically refers to the reference standard used to verify the quality of the target lidar sub-image, which can be set according to actual needs, and this embodiment does not make any limitation here.

[0067] Based on this, when verifying the quality of the target laser radar sub-image, in order to ensure the accuracy of verification and use high-quality laser radar sub-images for subsequent processing, the frame pose corresponding to the target laser radar sub-image can be determined first, and a triangular mesh can be constructed based on the frame pose; then the triangular mesh can be compared with the spliced ​​point cloud data contained in the target laser radar sub-image to calculate the error information based on the comparison result; the error information can reflect the thickness information of the object in the currently constructed target laser radar sub-image. Therefore, the error information can be compared with the reference index corresponding to the preset structure, and when it is determined according to the comparison result that the target laser radar sub-image meets the processing conditions, the step of mapping the target two-dimensional image data to the target laser radar sub-image and constructing the driving data for the automatic driving of the mobile device can be executed according to the mapping result.

[0068] In addition, when it is determined according to the comparison result that the target lidar sub-image does not meet the processing conditions, the above-mentioned processing operation can be performed again to improve the quality of the target lidar sub-image for subsequent use.

[0069] In specific implementation, in order to verify the quality of the currently constructed LiDAR map, the thickness of the structure can be used as a reference indicator. Specifically, VDBFusion and Marching Cubes can be used to reconstruct the mesh of the LiDAR frame pose obtained by solving the pose graph (both the voxel size and the cutoff distance are set to 5 cm) to create a triangular mesh representing the posterior maximum possible position of the surface of the enclosed object. After that, the thickness can be evaluated by comparing the mesh with the spliced ​​point cloud, and calculating the mean error (MAE) and root mean square error (RMSE). By comparing this thickness with the thickness of the structure, the quality of the currently constructed LiDAR map can be determined. The smaller the comparison result between the two, the higher the quality of the LiDAR map.

[0070] In summary, after obtaining the target lidar subimage, in order to provide high-quality lidar subimages for subsequent mapping processing, the quality of the target lidar subimage can also be verified. By comparing it with the reference index, it can be determined whether the error is within the acceptable range, which is convenient for subsequent use.

[0071] Step S208, mapping the target two-dimensional image data to the target lidar sub-image, and constructing driving data for automatic driving of the mobile device according to the mapping result.

[0072] Specifically, after obtaining the target lidar sub-image as described above, in order to achieve alignment of vision and laser, the target two-dimensional image data can be mapped to the target lidar sub-image, so that driving data for autonomous driving of the mobile device can be constructed based on the mapping results for use when the mobile device is moving.

[0073] Among them, driving data specifically refers to the data that realizes visual and laser alignment after mapping the target two-dimensional image data to the target lidar sub-image, which can be used as decision-making data for mobile device driving in autonomous driving scenarios.

[0074] Furthermore, when mapping the target two-dimensional image data to the target lidar sub-image, considering that there may be a problem of visual-laser pose misalignment between the target two-dimensional image data and the target lidar sub-image, direct mapping may lead to result deviation, so alignment correction can be achieved by calculating a multi-dimensional error factor. In this embodiment, the specific implementation method is as follows: Determine an image frame sequence corresponding to the two-dimensional image data and a three-dimensional sparse position associated with the mobile device; calculate a projection error based on a positional relationship between a plurality of image frames included in the image frame sequence and the three-dimensional sparse position; select candidate point cloud data in the target lidar sub-image based on the three-dimensional sparse position, and calculate a tangent distance between the candidate point cloud data and a collection device corresponding to the target two-dimensional image data, and use the tangent distance as a dense error; update the target two-dimensional image data according to a preset sparse error, the projection error, and the dense error; map the updated target two-dimensional image data to the target lidar sub-image, and construct driving data for autonomous driving of the mobile device based on the mapping result.

[0075] Specifically, the image frame sequence specifically refers to a sequence consisting of multiple two-dimensional images, and the images contained in the sequence are sorted according to the acquisition time. Correspondingly, the projection error specifically refers to the error of quantifying the image frame pose by projecting the three-dimensional sparse position onto the image plane of the frame pose. Correspondingly, the dense error specifically refers to the tangent distance between the calculated point cloud data and the acquisition device.

[0076] Based on this, in order to eliminate the impact of errors, we can first determine the image frame sequence corresponding to the two-dimensional image data and the three-dimensional sparse position associated with the mobile device. On this basis, we can calculate the projection error based on the positional relationship between the multiple image frames contained in the image frame sequence and the three-dimensional sparse position. The projection error can be used to eliminate the impact of the image frame pose error.

[0077] At the same time, candidate point cloud data can be selected in the target lidar sub-image according to the three-dimensional sparse position, and the tangent distance between the candidate point cloud data and the target two-dimensional image data corresponding to the acquisition device can be calculated. The tangent distance is used as the dense error, and the dense error is used to optimize the retrieval of sparse positions and neighboring points in the laser point cloud data. Finally, the target two-dimensional image data can be updated according to the preset sparse error, projection error and dense error; and the updated target two-dimensional image data can be mapped to the target lidar sub-image, and the driving data for autonomous driving of mobile devices can be constructed according to the mapping results.

[0078] In specific implementation, when mapping the target two-dimensional image data to the LiDAR map constructed above, it specifically refers to solving the pose of each image frame relative to the reference coordinate system, so as to achieve the purpose of vision and laser alignment. In this process, linear spherical interpolation can be used to initialize the above pose given two adjacent LiDAR frames and image-laser extrinsics for each image frame. Secondly, the pose of all cameras can be further optimized by constructing the following factor graph optimization problem and using the expectation maximization (EM) strategy. Specifically, the factors are designed as follows: Camera Structure from Motion (SfM) reconstruction. SuperGlue can be used to associate joint observations of the same 3D sparse position between multiple image frames, and the projection error factor is constructed through the reprojection error, which quantifies the image frame pose error by projecting the 3D position of the landmark onto the image plane of the corresponding frame pose.

[0079] Camera SfM points are registered with the LiDAR map. Longitudinal driving sequences usually lack sufficient parallax to accurately estimate the depth of some 3D sparse locations. Therefore, once these landmarks are effectively associated with nearby LiDAR points, the positions of these landmarks need to be constrained. The single prior factor of each landmark is defined as the error between the 3D sparse position generated by SfM and its neighboring points in the laser point cloud. In this case, the correspondence between the point pairs formed by the 3D sparse position and the neighboring points in the laser point cloud can be used. In order to optimize the retrieval of the above relationship, a set of candidate points can be selected based on the KNN search of the closer points of each 3D sparse position, and the tangent distances between these points and the rays emitted from multiple camera observations are calculated and used as the joint weight of the final prior position.

[0080] Cross-modal sparse and dense. In multimodal closed-loop simulation applications, the above two factors alone are not sufficient to achieve data association between camera pixels and LiDAR point levels. Therefore, we can combine sparse and dense factors and use SuperGlue again to provide sparse constraints to ensure more accurate mapping results.

[0081] In summary, by optimizing the target two-dimensional image data according to the above error factors, it can be ensured that the optimized image data is more accurate when aligned with the target lidar sub-image, thereby achieving safer autonomous driving operations in autonomous driving scenarios.

[0082] In addition, after mapping the target two-dimensional image data to the target laser radar sub-image, in order to detect the mapping result, it can be realized by calculating the loss. In this embodiment, the specific implementation method is as follows: Determine grayscale image data based on the target two-dimensional image data, and determine intensity image data based on the target lidar sub-image; calculate offset loss based on the grayscale image data and the intensity image data, and when the offset loss is less than a preset loss threshold, execute the step of constructing driving data for autonomous driving of the mobile device based on the mapping result.

[0083] Specifically, the grayscale image data specifically refers to the image data obtained after grayscale processing of the target two-dimensional image data, and correspondingly, the intensity image data specifically refers to the intensity image data corresponding to the target lidar sub-image; correspondingly, the offset loss specifically refers to the error loss existing after constructing the sub-image.

[0084] Based on this, when verifying the mapping results, the grayscale image data can be determined based on the target two-dimensional image data, and the intensity image data can be determined based on the target lidar sub-image; at this time, the offset loss can be calculated based on the grayscale image data and the intensity image data, so that when the offset loss is less than a preset loss threshold, the step of constructing driving data for autonomous driving of the mobile device according to the mapping results is executed.

[0085] In specific implementation, after completing the alignment of vision and laser, in order to evaluate the quality of alignment, the normalized information distance (NID) proposed in the following formula (1) can be used as a quantitative indicator. For each pair of original grayscale images and LiDAR intensity images rendered from a given posture, the initial NID is calculated by discretizing continuous pixel values ​​into 64 intervals. Then, the original image is slightly moved in the UV coordinate system to determine a position that can produce a local minimum value of NID. The offset used to find this local minimum value is defined as NID-Loss. In order to achieve when a smaller NID-Loss is calculated, it is determined that the alignment quality meets the requirements of autonomous driving. Among them, formula (1) is as follows: (1) The images formed by the three-dimensional point cloud data and the two-dimensional image data are respectively I r and I s ; H(I r , I s ) indicates I r and I s The joint entropy, MI (I r , I s ) indicates I r and I s The alignment relationship between H (I r ) and H (I s ) indicates I r and I s The marginal entropy of NID is expressed as the metric value.

[0086] The data processing method provided in this embodiment can first collect the three-dimensional point cloud data and two-dimensional image data associated with the mobile device in order to realize the ability of automatic driving technology to evaluate the perspective synthesis based on real data, thereby reducing the impact of automatic driving caused by errors. Considering that there may be some special weather effects in the actual driving environment, although the collected data is real, it may affect safety if the impact of changes in the external driving environment is not considered when used for driving. Therefore, the three-dimensional point cloud data and the two-dimensional image data can be calibrated to obtain the calibrated target three-dimensional point cloud data and target two-dimensional image data. On this basis, the target three-dimensional point cloud data can be used to optimize multiple historical laser radar sub-images associated with the mobile device, and the target laser radar sub-image is generated according to the optimization results. In this way, a radar sub-image matching the current driving scene is constructed, and the error impact caused by the driving environment is integrated. Then the target two-dimensional image data can be mapped to the target laser radar sub-image, and then the driving data for the automatic driving of the mobile device can be constructed according to the mapping results, and when the three-dimensional point cloud data and the two-dimensional image data are aligned, the influence of external factors can be considered to reduce the error, thereby ensuring the authenticity of the data to meet the use of automatic driving.

[0087] The following combination Figure 3 , taking the application of the data processing method provided in this specification in an autonomous driving scenario as an example, the data processing method is further described. Figure 3 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0088] Step S302, collecting three-dimensional point cloud data and two-dimensional image data associated with the mobile device.

[0089] Step S304 , determining target feature point data in the two-dimensional image data, and extracting target point cloud data corresponding to the target feature point data in the three-dimensional point cloud data.

[0090] Step S306 , performing point-to-point calibration on the three-dimensional point cloud data and the two-dimensional image data based on the target feature point data and the target point cloud data.

[0091] Step S308, determining target three-dimensional point cloud data and target two-dimensional image data according to the calibration result.

[0092] In specific implementation, the calibration of 3D point cloud data and 2D image data is actually a data correction process based on the results of multi-sensor external parameter calibration. In practical applications, in order to accurately solve scene information using multiple sensors, it is necessary to calibrate the external parameters of multiple sensors on board. The purpose is to be able to effectively align the 3D point cloud obtained by laser measurement with the 2D image obtained by the camera, so as to achieve more accurate environmental perception and scene reconstruction. In specific implementation, it can be achieved through the following processing: (1) Material preparation: Make sure the laser sensor and camera are installed and functioning properly. Confirm the fixed relationship between the devices. At the same time, prepare a standard calibration plate (such as a chessboard or Apri-lTag) with known geometric features.

[0093] (2) Data acquisition: The laser sensor and camera work synchronously to acquire data in the same scene. Usually, the laser sensor is scanned across the calibration plate while the camera is used to capture the image of the calibration plate, while ensuring that the collected point cloud data and image data are aligned, while ensuring that the sensor combination remains rigidly connected.

[0094] (3) Information processing: Process the captured camera images and extract the feature points (such as corner points) on the calibration plate. Extract the three-dimensional point cloud data corresponding to the calibration plate from the laser scan. By combining the information of the two modalities of the laser sensor and the camera, point-to-point matching is performed using the extracted feature points, so that the preliminary external parameter estimation can be completed. After that, the iterative optimization algorithm (such as the nonlinear least squares method) can be used to minimize the error between the laser point cloud data and the image feature points, thereby obtaining more accurate external parameters.

[0095] Based on this, the two-dimensional image data and the three-dimensional point cloud data are calibrated to obtain the target three-dimensional point cloud data and the target two-dimensional image data with some errors eliminated. In specific implementation, the images and laser point clouds of the validation set can also be used to check the accuracy of the external parameter calibration results, so that the registration results can be judged to decide whether to calibrate the collected two-dimensional image data and three-dimensional point cloud data.

[0096] Step S310, extracting target point cloud data features corresponding to the target three-dimensional point cloud data, and obtaining multiple historical lidar sub-images associated with the mobile device.

[0097] Step S312: construct a first registration relationship and a second registration relationship according to the target point cloud data features and multiple historical lidar sub-images.

[0098] Step S314, optimizing the multiple historical lidar sub-images using the first registration relationship and the second registration relationship, and generating a target lidar sub-image according to the optimization result.

[0099] Among them, the construction of the first registration relationship includes: performing point cloud registration processing according to the target point cloud data characteristics and multiple historical lidar subimages to obtain the first registration relationship; wherein, the construction of the second registration relationship includes: using the matching model to screen at least two candidate lidar subimages from multiple historical lidar subimages, and determining the matching scores corresponding to the at least two candidate lidar subimages respectively; determining a candidate lidar subimage pair from at least two candidate lidar subimages according to the matching scores; performing point cloud registration processing according to the first lidar subimage and the second lidar subimage in the candidate lidar subimage pair to obtain the second registration relationship.

[0100] In specific implementation, the construction of the target LiDAR sub-map is essentially the calculation of the laser dense pose. Given the initial trajectory from the RTK / INS sensor, a target LiDAR sub-map (LiDAR map) in the reference coordinate system G can be constructed through the laser SLAM algorithm. This process can be achieved through odometer and loop closure optimization.

[0101] For the odometry part, LOAM features and LiDAR sub-images accumulated over a period of time can be used for GICP registration to enhance the robustness in various mapping scenarios.

[0102] The loop optimization part can be understood as including loop extraction and pose graph optimization. Among them, for loop extraction, the matching model can be used to perform rough matching of multiple historical lidar sub-images and calculate the matching score of each sub-image. Then, the candidate lidar sub-image pairs can be screened out from multiple historical lidar sub-images by the matching score. Then, GICP registration can be performed based on the candidate lidar sub-image pairs. In this way, accurate relative constraints can be established for the pose graph optimization problem. In the pose graph optimization process, the relative constraints generated in the extraction process and the continuous registration relative constraints implemented in the above-mentioned odometer part can be reused to perform global consistency optimization on multiple historical lidar sub-images, thereby eliminating the accumulated errors in the odometer process and obtaining the final target lidar sub-image (LiDAR map).

[0103] It can be understood that, through the pose graph optimization problem, it is possible to consider both the meter-level approximate position of each sub-image obtained through satellite inertia and the fine relative position between sub-images obtained through laser point cloud registration, so as to determine the precise position corresponding to the target lidar sub-image.

[0104] In addition, in order to verify the quality of the currently constructed LiDAR map, the thickness of the structure can be used as a reference indicator. Specifically, VDBFusion and Marching Cubes can be used to reconstruct the mesh of the LiDAR frame pose obtained by solving the pose graph (both the voxel size and the cutoff distance are set to 5 cm) to create a triangular mesh representing the posterior most likely position of the surface of the enclosed object. After that, the thickness can be evaluated by comparing the mesh with the spliced ​​point cloud, and calculating the mean error (MAE) and root mean square error (RMSE). By comparing this thickness with the thickness of the structure, the quality of the currently constructed LiDAR map can be determined. The smaller the comparison result between the two, the higher the quality of the LiDAR map.

[0105] Step S316, determining an image frame sequence corresponding to the two-dimensional image data and a three-dimensional sparse position associated with the mobile device.

[0106] Step S318: calculating a projection error according to a positional relationship between a plurality of image frames included in the image frame sequence and the three-dimensional sparse position.

[0107] Step S320, select candidate point cloud data in the target lidar sub-image according to the three-dimensional sparse position, and calculate the tangent distance between the candidate point cloud data and the target two-dimensional image data corresponding to the acquisition device, and use the tangent distance as the dense error.

[0108] Step S322: updating the target two-dimensional image data according to preset sparse error, projection error and dense error.

[0109] Step S324, mapping the updated target two-dimensional image data to the target lidar sub-image, and constructing driving data for autonomous driving of the mobile device according to the mapping result.

[0110] In specific implementation, when mapping the target two-dimensional image data to the LiDAR map constructed above, it specifically refers to solving the pose of each image frame relative to the reference coordinate system, so as to achieve the purpose of vision and laser alignment. In this process, linear spherical interpolation can be used to initialize the above pose given two adjacent LiDAR frames and image-laser extrinsics for each image frame. Secondly, the pose of all cameras can be further optimized by constructing the following factor graph optimization problem and using the expectation maximization (EM) strategy. Specifically, the factors are designed as follows: The camera's structure from motion (SfM) reconstruction can use SuperGlue to associate joint observations of the same 3D sparse position between multiple image frames, and construct the projection error factor through the reprojection error. The projection error factor quantifies the image frame pose error by projecting the 3D position of the landmark onto the image plane of the corresponding frame pose.

[0111] Camera SfM points are registered with the LiDAR map. Longitudinal driving sequences usually lack sufficient parallax to accurately estimate the depth of some 3D sparse locations. Therefore, once these landmarks are effectively associated with nearby LiDAR points, the positions of these landmarks need to be constrained. The single prior factor of each landmark is defined as the error between the 3D sparse position generated by SfM and its neighboring points in the laser point cloud. In this case, the correspondence between the point pairs formed by the 3D sparse position and the neighboring points in the laser point cloud can be used. In order to optimize the retrieval of the above relationship, a set of candidate points can be selected based on the KNN search of the nearest point of each 3D sparse position, and the tangent distances between these points and the rays emitted from multiple camera observations are calculated as the joint weights of the final prior positions.

[0112] Cross-modal sparse and dense. In multimodal closed-loop simulation applications, the above two factors alone are not sufficient to achieve data association between camera pixels and LiDAR point levels. Therefore, we can combine sparse and dense factors and use SuperGlue again to provide sparse constraints to ensure more accurate mapping results.

[0113] In addition, after completing the alignment of vision and laser, in order to evaluate the quality of alignment, the normalized information distance (NID) proposed in the above formula (1) can be used as a quantitative indicator. For each pair of original grayscale images and LiDAR intensity images rendered from a given posture, the initial NID is calculated by discretizing continuous pixel values ​​into 64 intervals. Then, the original image is slightly moved in the UV coordinate system to determine a position that can produce a local minimum value of NID. The offset used to find this local minimum value is defined as NID-Loss. In order to achieve when a smaller NID-Loss is calculated, it is determined that the alignment quality meets the requirements of autonomous driving.

[0114] In summary, in order to realize the ability of autonomous driving technology to evaluate the perspective synthesis based on real data, thereby reducing the impact of autonomous driving caused by errors, the three-dimensional point cloud data and two-dimensional image data associated with the mobile device can be collected first; considering that there may be some special weather effects in the actual driving environment, the collected data is real, but if the changes in the external driving environment are not considered when used for driving, it may affect safety; therefore, the three-dimensional point cloud data and two-dimensional image data can be calibrated to obtain the calibrated target three-dimensional point cloud data and target two-dimensional image data; on this basis, the target three-dimensional point cloud data can be used to optimize the multiple historical laser radar sub-images associated with the mobile device, and the target laser radar sub-image is generated according to the optimization results; in this way, a radar sub-image matching the current driving scene is constructed, and the error impact caused by the driving environment is integrated. Then the target two-dimensional image data can be mapped to the target laser radar sub-image, and then the driving data for the mobile device's autonomous driving can be constructed according to the mapping results, and when the three-dimensional point cloud data and the two-dimensional image data are aligned, the influence of external factors can be considered to reduce the error, thereby ensuring the authenticity of the data to meet the use of autonomous driving.

[0115] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises: A collection module 402 is configured to collect three-dimensional point cloud data and two-dimensional image data associated with a mobile device; A calibration module 404 is configured to calibrate the three-dimensional point cloud data and the two-dimensional image data to obtain target three-dimensional point cloud data and target two-dimensional image data; The optimization module 406 is configured to optimize the multiple historical lidar sub-images associated with the mobile device using the target three-dimensional point cloud data, and generate a target lidar sub-image according to the optimization result; The mapping module 408 is configured to map the target two-dimensional image data to the target lidar sub-image, and construct driving data for autonomous driving of the mobile device according to the mapping result.

[0116] In an optional embodiment, the calibration module 404 is further configured to: Determine target feature point data in the two-dimensional image data, and extract target point cloud data corresponding to the target feature point data in the three-dimensional point cloud data; perform point-to-point calibration on the three-dimensional point cloud data and the two-dimensional image data based on the target feature point data and the target point cloud data; determine target three-dimensional point cloud data and target two-dimensional image data according to the calibration result.

[0117] In an optional embodiment, the optimization module 406 is further configured to: Extract target point cloud data features corresponding to the target three-dimensional point cloud data, and obtain multiple historical lidar sub-images associated with the mobile device; construct a first registration relationship and a second registration relationship based on the target point cloud data features and the multiple historical lidar sub-images; use the first registration relationship and the second registration relationship to optimize the multiple historical lidar sub-images, and generate a target lidar sub-image based on the optimization results.

[0118] In an optional embodiment, the construction of the first registration relationship includes: Point cloud registration processing is performed according to the target point cloud data features and the multiple historical lidar subimages to obtain the first registration relationship; wherein the construction of the second registration relationship includes: using a matching model to screen at least two candidate lidar subimages from the multiple historical lidar subimages, and determining the matching scores corresponding to the at least two candidate lidar subimages respectively; determining a candidate lidar subimage pair from the at least two candidate lidar subimages according to the matching scores; and point cloud registration processing is performed according to the first lidar subimage and the second lidar subimage in the candidate lidar subimage pair to obtain the second registration relationship.

[0119] In an optional embodiment, the mapping module 408 is further configured to: Determine an image frame sequence corresponding to the two-dimensional image data and a three-dimensional sparse position associated with the mobile device; calculate a projection error based on a positional relationship between a plurality of image frames included in the image frame sequence and the three-dimensional sparse position; select candidate point cloud data in the target lidar sub-image based on the three-dimensional sparse position, and calculate a tangent distance between the candidate point cloud data and a collection device corresponding to the target two-dimensional image data, and use the tangent distance as a dense error; update the target two-dimensional image data according to a preset sparse error, the projection error, and the dense error; map the updated target two-dimensional image data to the target lidar sub-image, and construct driving data for autonomous driving of the mobile device based on the mapping result.

[0120] In an optional embodiment, the device further includes: A comparison module is configured to determine the frame pose corresponding to the target laser radar sub-image, and construct a triangular mesh according to the frame pose; compare the triangular mesh with the spliced ​​point cloud data contained in the target laser radar sub-image, and calculate error information according to the comparison result; compare the error information with a reference index corresponding to a preset structure, and when it is determined according to the comparison result that the target laser radar sub-image meets the processing conditions, execute the step of mapping the target two-dimensional image data to the target laser radar sub-image, and construct driving data for autonomous driving of the mobile device according to the mapping result.

[0121] In an optional embodiment, the device further comprises: The loss comparison module is configured to determine grayscale image data based on the target two-dimensional image data, and to determine intensity image data based on the target lidar sub-image; calculate the offset loss based on the grayscale image data and the intensity image data, and when the offset loss is less than a preset loss threshold, execute the step of constructing driving data for autonomous driving of the mobile device based on the mapping result.

[0122] The data processing device provided in this embodiment can first collect the three-dimensional point cloud data and two-dimensional image data associated with the mobile device in order to realize the ability of evaluating the perspective synthesis based on real data of the autonomous driving technology, thereby reducing the impact of the autonomous driving caused by the error. Considering that there may be some special weather influences in the actual driving environment, although the collected data is real, it may affect the safety if the influence of the changes in the external driving environment is not considered when it is used for driving. Therefore, the three-dimensional point cloud data and the two-dimensional image data can be calibrated to obtain the calibrated target three-dimensional point cloud data and target two-dimensional image data. On this basis, the target three-dimensional point cloud data can be used to optimize the multiple historical laser radar sub-images associated with the mobile device, and the target laser radar sub-image is generated according to the optimization result. In this way, a radar sub-image matching the current driving scene is constructed, and the error influence caused by the driving environment is integrated. Then the target two-dimensional image data can be mapped to the target laser radar sub-image, and then the driving data for the autonomous driving of the mobile device can be constructed according to the mapping result, and when the three-dimensional point cloud data and the two-dimensional image data are aligned, the influence of external factors can be considered to reduce the error, thereby ensuring the authenticity of the data to meet the use of autonomous driving.

[0123] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0124] Figure 5The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0125] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0126] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0127] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.

[0128] The processor 520 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.

[0129] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0130] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.

[0131] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0132] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0133] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above data processing method.

[0134] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above data processing method when executed by a processor.

[0135] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above data processing method.

[0136] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0137] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0138] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0139] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0140] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Collecting 3D point cloud data and 2D image data associated with a mobile device; Calibrate the three-dimensional point cloud data and the two-dimensional image data to obtain target three-dimensional point cloud data and target two-dimensional image data; Optimizing multiple historical lidar sub-images associated with the mobile device using the target three-dimensional point cloud data, and generating a target lidar sub-image according to the optimization result; The target two-dimensional image data is mapped to the target lidar sub-image, and driving data for autonomous driving of the mobile device is constructed according to the mapping result.

2. The data processing method according to claim 1, wherein calibrating the three-dimensional point cloud data and the two-dimensional image data to obtain target three-dimensional point cloud data and target two-dimensional image data comprises: Determining target feature point data in the two-dimensional image data, and extracting target point cloud data corresponding to the target feature point data in the three-dimensional point cloud data; Based on the target feature point data and the target point cloud data, performing point-to-point calibration on the three-dimensional point cloud data and the two-dimensional image data; The target three-dimensional point cloud data and the target two-dimensional image data are determined according to the calibration results.

3. The data processing method according to claim 1, wherein the step of optimizing the multiple historical lidar sub-images associated with the mobile device using the target three-dimensional point cloud data and generating the target lidar sub-image according to the optimization result comprises: Extracting target point cloud data features corresponding to the target three-dimensional point cloud data, and acquiring a plurality of historical lidar sub-images associated with the mobile device; Constructing a first registration relationship and a second registration relationship according to the target point cloud data features and the multiple historical lidar sub-images; The plurality of historical lidar sub-images are optimized using the first registration relationship and the second registration relationship, and a target lidar sub-image is generated according to the optimization result.

4. The data processing method according to claim 3, wherein the construction of the first registration relationship comprises: Performing point cloud registration processing according to the target point cloud data features and the multiple historical lidar sub-images to obtain the first registration relationship; Wherein, the construction of the second registration relationship includes: Using the matching model to select at least two candidate lidar subimages from the multiple historical lidar subimages, and determining matching scores corresponding to the at least two candidate lidar subimages respectively; Determine a candidate lidar subimage pair in the at least two candidate lidar subimages according to the matching score; Point cloud registration processing is performed on the first lidar subimage and the second lidar subimage in the candidate lidar subimage pair to obtain the second registration relationship.

5. The data processing method according to claim 1, mapping the target two-dimensional image data to the target laser radar sub-image, and constructing driving data for autonomous driving of the mobile device according to the mapping result, comprises: Determining an image frame sequence corresponding to the two-dimensional image data and a three-dimensional sparse position associated with the mobile device; Calculating a projection error according to a positional relationship between a plurality of image frames included in the image frame sequence and a three-dimensional sparse position; Select candidate point cloud data in the target laser radar sub-image according to the three-dimensional sparse position, calculate the tangent distance between the candidate point cloud data and the acquisition device corresponding to the target two-dimensional image data, and use the tangent distance as the dense error; Updating the target two-dimensional image data according to a preset sparse error, the projection error and the dense error; The updated target two-dimensional image data is mapped to the target lidar sub-image, and driving data for autonomous driving of the mobile device is constructed according to the mapping result.

6. The data processing method according to any one of claims 1 to 5, wherein after the step of optimizing the multiple historical lidar sub-images associated with the mobile device using the target three-dimensional point cloud data and generating the target lidar sub-image according to the optimization result is performed, the method further comprises: Determine a frame pose corresponding to the target laser radar sub-image, and construct a triangular mesh according to the frame pose; Comparing the triangular mesh with the spliced ​​point cloud data contained in the target laser radar sub-image, and calculating error information according to the comparison result; The error information is compared with a reference index corresponding to a preset structure. When it is determined based on the comparison result that the target lidar sub-image satisfies the processing conditions, the step of mapping the target two-dimensional image data to the target lidar sub-image is executed, and driving data for autonomous driving of the mobile device is constructed based on the mapping result.

7. The data processing method according to any one of claims 1 to 5, after the step of mapping the target two-dimensional image data to the target laser radar sub-image is performed, further comprising: Determining grayscale image data based on the target two-dimensional image data, and determining intensity image data based on the target lidar sub-image; The offset loss is calculated based on the grayscale image data and the intensity image data, and when the offset loss is less than a preset loss threshold, a step of constructing driving data for autonomous driving of the mobile device according to the mapping result is performed.

8. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, which implement the steps of the method according to any one of claims 1 to 7 when executed by a processor.

Images