Data enhancement method and device, storage medium, terminal and computer program product
By using time-series multi-frame point cloud and image data for data enhancement, the problem that the prior art cannot simulate dynamic driving environment and does not consider multimodal information. The generated enhanced data is suitable for timing and multimodal model training.
Patent Information
- Application Number
- CN202411395997.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-02-18
AI Technical Summary
The existing autonomous driving data augmentation technology cannot effectively simulate the real driving environment of a vehicle during dynamic driving, and fails to consider multimodal information, resulting in the generated enhancement data being unable to be applicable to timing model training and multimodal model training.
By using the multi-frame original point cloud with timing and the multi-frame original image acquired by the camera as the basic data, coordinate space transformation and occlusion situation analysis are performed, and obstacle data are combined to obtain point cloud and image enhancement data.
The generated enhanced data contains the scene timing characteristics of the vehicle in multiple different locations, which can be better applied to timing model training and simulation experiments, while meeting the training needs of multimodal models.
Smart Images
Figure CN119478882A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and specifically to a data enhancement method and device, a storage medium, a terminal, and a computer program product. Background Art
[0002] In the development of autonomous driving technology, data enhancement technology plays a vital role. Autonomous driving systems rely on a large amount of scene data (e.g., point cloud data, image data, etc.) to train deep learning models or conduct autonomous driving simulation tests. However, it is expensive and time-consuming to obtain and annotate sufficiently diverse and high-quality scene data. In addition, there is a serious problem of long-tail data distribution in autonomous driving scenarios, that is, common dynamic obstacles (e.g., pedestrians, motor vehicles, non-motor vehicles, etc.) are easier to collect, while uncommon road abnormal obstacle data (e.g., cardboard boxes, sand piles, garbage bags, etc. on the road) are difficult to collect, but such road abnormal obstacles are a great test for the safety of autonomous driving. Through data enhancement technology, various road conditions can be simulated, and a large number of abnormal obstacles can be generated in the original scene data to obtain more diverse and rich autonomous driving simulation test data, as well as to improve the detection ability of autonomous driving perception models for abnormal obstacles, so that autonomous driving related algorithms or perception models can cope with diverse driving scenarios.
[0003] In the existing data enhancement technology in the field of autonomous driving, obstacle data is often generated in a single frame image collected by a camera at a specific moment, or in a single frame point cloud collected by a laser radar at a specific moment. However, a single frame image or a single frame point cloud only contains the scene information corresponding to the vehicle at a single position, and lacks the scene time sequence characteristics between different positions before and after. Therefore, the enhanced data obtained by the existing technology cannot well simulate the real driving environment of the autonomous driving vehicle during dynamic driving, and cannot be applied to time sequence model training and simulation experiments. In addition, the existing data enhancement technology usually does not consider multimodal information (different modal information can refer to different types of autonomous driving scene data, such as scene images, scene point clouds, etc.), and only generates abnormal obstacles in two-dimensional images or three-dimensional point clouds. There is no time sequence alignment and spatial position alignment for visual sensors and radar sensors. Therefore, the obtained enhanced data cannot be applied to the training of multimodal models. Summary of the invention
[0004] The technical problem solved by the embodiments of the present invention is how to obtain stable, sequential continuous frame enhancement data.
[0005] To solve the above technical problems, an embodiment of the present invention provides a data enhancement method, comprising the following steps: determining initial obstacle data and a sequence of source data to be enhanced, wherein the sequence of source data to be enhanced comprises multiple frames of original point clouds with time sequence collected by a laser radar during the driving process of the vehicle, wherein each frame of original point cloud has its own laser radar coordinate system, and the laser radar coordinate system corresponds to the position of the laser radar at the time of collecting the original point cloud of the frame; for each frame of original point cloud, transforming the initial obstacle data to the laser radar coordinate system to which the original point cloud of the frame belongs to obtain corresponding transformed obstacle data; for each frame of original point cloud, merging it with the corresponding transformed obstacle data to obtain point cloud enhancement data corresponding to the original point cloud of the frame.
[0006] Optionally, before merging each frame of original point cloud with the corresponding transformed obstacle data, the method further includes: performing occlusion analysis on each frame of original point cloud and the corresponding transformed obstacle data, and confirming that the transformed obstacle data is not occluded.
[0007] Optionally, the occlusion analysis for each frame of original point cloud and the corresponding transformed obstacle data includes: using a first transformation matrix between a laser radar coordinate system and a camera coordinate system at the acquisition moment to which the frame of original point cloud belongs, transforming the frame of original point cloud and the transformed obstacle data to the camera coordinate system respectively, to obtain corresponding camera space point cloud and camera space obstacle data; based on the camera space point cloud data, camera space obstacle data, and a second transformation matrix between the camera coordinate system and the corresponding image coordinate system, performing an occlusion analysis on the frame of original point cloud and the corresponding transformed obstacle data; wherein the camera coordinate system at the acquisition moment to which each frame of original point cloud belongs corresponds to the position of the camera at the acquisition moment, and the image coordinate system corresponding to the camera coordinate system is the image coordinate system to which the original image acquired by the camera at the acquisition moment belongs.
[0008] Optionally, based on the camera space point cloud data, the camera space obstacle data, and the second transformation matrix between the camera coordinate system and the corresponding image coordinate system, an occlusion analysis is performed on the frame original point cloud and the corresponding transformed obstacle data, including: performing voxel processing on the camera space point cloud data and the camera space obstacle data respectively to obtain a corresponding plurality of point cloud voxels and a plurality of obstacle pixels; for a direction vector pointing from the origin of the camera coordinate system to each pixel position of the corresponding image coordinate system, determining a plurality of sampling points on the direction vector according to a preset sampling method; selecting a direction vector having at least a first preset number of sampling points falling into the point cloud voxel, recorded as a target vector; if at least a second preset number of target vectors satisfy the following condition, confirming that the transformed obstacle data is occluded: among the sampling points on the target vector falling into the point cloud voxel, at least a third preset number of sampling points have a distance from the origin of the camera coordinate system that is less than a distance between any sampling point on the target vector falling into the obstacle pixel and the origin of the camera coordinate system.
[0009] Optionally, the method also includes: when it is confirmed that the transformed obstacle data corresponding to the original point cloud is occluded, repeatedly offsetting the transformed obstacle data, and performing occlusion analysis on the original point cloud and the obstacle data after the current offset after each offset; if the obstacle data after the current offset is not occluded and the number of repetitions is less than a preset number, merging the obstacle data after the current offset with the original point cloud to obtain point cloud enhanced data corresponding to the original point cloud.
[0010] Optionally, the method further includes: if the number of repetitions reaches the preset number of times and the obstacle data after the last offset is still blocked, replacing the source data sequence to be enhanced; and performing data enhancement based on the replaced source data sequence to be enhanced and the initial obstacle data.
[0011] Optionally, for each frame of original point cloud, the initial obstacle data is transformed into a laser radar coordinate system to which the original point cloud of the frame belongs to obtain corresponding transformed obstacle data, including: selecting a target acquisition moment from each acquisition moment to which the multiple frames of original point clouds belong, and using the self-vehicle coordinate system of the target acquisition moment as the reference coordinate system, wherein the self-vehicle coordinate system of the acquisition moment to which each frame of original point cloud belongs corresponds to the position of the self-vehicle at the acquisition moment; transforming the initial obstacle data into the reference coordinate system to obtain first transformed data; for each frame of original point cloud, using a third transformation matrix between the self-vehicle coordinate system of the acquisition moment to which the original point cloud of the frame belongs and the reference coordinate system, transforming the first transformed data into the self-vehicle coordinate system to obtain second transformed data; based on a fourth transformation matrix between the self-vehicle coordinate system and the laser radar coordinate system to which the original point cloud of the frame belongs, transforming the second transformed data into the laser radar coordinate system to which the original point cloud of the frame belongs to obtain the transformed obstacle data.
[0012] Optionally, before transforming the initial obstacle data to the laser radar coordinate system to which the original point cloud of each frame belongs, the method further includes: selecting a target area in the original point cloud of the frame, counting the total number of three-dimensional points in the original point cloud of the frame that fall into the target area, and calculating the quotient of the total number and the volume of the target area to obtain a unit number; determining the volume of the obstacle area formed by the initial obstacle data, and using the product of the volume of the obstacle area and the unit number as the number of sampling points; and based on the number of sampling points, down-sampling the three-dimensional points on the surface of the obstacle area.
[0013] Optionally, the source data sequence to be enhanced further includes a plurality of frames of original images with time sequence captured by the camera for the same target scene during the driving process of the vehicle, each frame of the original image has its own image coordinate system, and the image coordinate system corresponds to the position of the camera at the time of capturing the original point cloud of the frame, wherein the plurality of frames of original point cloud with time sequence correspond to the plurality of frames of original images one by one, and the corresponding original images are consistent with the time of capturing the original point cloud; the method further includes: for each frame of original point cloud, transforming the obstacle data of the initial obstacle data in the laser radar coordinate system to which the original point cloud of the frame belongs, to the image coordinate system to which the original image corresponding to the original point cloud of the frame belongs, to obtain corresponding image obstacle data; performing a rendering operation based on the image obstacle data to obtain rendered image data, wherein each pixel of the rendered image data has its own pixel value; and replacing the pixel value of the pixel at the same position in the original image of the frame with the pixel value of each pixel of the rendered image data to obtain corresponding image enhancement data.
[0014] An embodiment of the present invention also provides a data enhancement device, including: a data acquisition module to be enhanced, used to determine initial obstacle data and a source data sequence to be enhanced, wherein the source data sequence to be enhanced includes multiple frames of original point clouds with time sequence collected by a laser radar during the driving process of the vehicle, wherein each frame of original point cloud has its own laser radar coordinate system, and the laser radar coordinate system corresponds to the position of the laser radar at the time of collecting the original point cloud of the frame; a coordinate space transformation module, used to transform the initial obstacle data to the laser radar coordinate system to which the original point cloud of the frame belongs for each frame of original point cloud, so as to obtain corresponding transformed obstacle data; an enhanced data determination module, used to merge the original point cloud of each frame with the corresponding transformed obstacle data to obtain point cloud enhancement data corresponding to the original point cloud of the frame.
[0015] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data enhancement method are executed.
[0016] An embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned data enhancement method when running the computer program.
[0017] An embodiment of the present invention further provides a computer program product, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned data enhancement method when running the computer program.
[0018] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:
[0019] The existing data enhancement technology in the field of autonomous driving is usually based on a single-frame image collected by a camera at a specific moment, or a laser radar collects obstacle data in a single-frame point cloud at a specific moment, resulting in the lack of scene temporal features between different positions before and after. Therefore, it is impossible to simulate the real driving environment of the autonomous driving vehicle in the dynamic driving process well, and it is also impossible to be suitable for temporal model training and simulation experiments. In contrast, in an embodiment of the present invention, a multi-frame original point cloud with temporal characteristics is used as the basic data for generating obstacles, and the initial obstacle data is transformed into the laser radar coordinate system to which each frame of the original point cloud belongs by performing a coordinate space transformation on the initial obstacle data, so as to obtain the transformed obstacle data corresponding to each frame of the original point cloud. After the coordinate space transformation, the time sequence and position of the multi-frame original point cloud and the corresponding groups of transformed obstacle data can be aligned, so that the multi-frame enhanced data containing the scene temporal features of the vehicle at multiple different positions can be finally obtained. Further, the multi-frame point cloud enhanced data with temporal characteristics can be well applied to temporal model training and simulation testing of algorithms related to autonomous driving.
[0020] Furthermore, in an embodiment of the present invention, a multi-frame original point cloud with time sequence collected by a lidar and a multi-frame original image with time sequence collected by a camera and aligned one by one (time sequence alignment and spatial position alignment) are used as the basic data for performing data enhancement. This can effectively perform joint data enhancement on the image and the point cloud to meet the conversion of different postures between sensors, and can be applicable to multimodal model training. At the same time, it can effectively generate stable continuous frame enhanced data, better simulate the sensor imaging effect in a real environment, and is suitable for time series model training and simulation experiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 is a flow chart of a data enhancement method in an embodiment of the present invention;
[0022] Figure 2 yes Figure 1 A flowchart of a specific implementation of step S12;
[0023] Figure 3 is a flow chart of another data enhancement method in an embodiment of the present invention;
[0024] Figure 4 yes Figure 3 A flowchart of a specific implementation of step S31;
[0025] Figure 5 yes Figure 4 A flowchart of a specific implementation of step S42;
[0026] Figure 6 It is a structural schematic diagram of a data enhancement device in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0028] Reference Figure 1 , Figure 1 1 is a flow chart of a data enhancement method in an embodiment of the present invention. The method can be applied to a terminal device with a data processing function, such as but not limited to: a computer, a mobile phone, a tablet computer, a vehicle-mounted terminal, a server, a cloud platform, etc.
[0029] The data enhancement method may include steps S11 to S13:
[0030] Step S11: determining initial obstacle data and a sequence of source data to be enhanced, wherein the sequence of source data to be enhanced includes multiple frames of original point clouds with time sequence collected by the laser radar during the driving process of the vehicle, wherein each frame of the original point cloud has its own laser radar coordinate system, and the laser radar coordinate system corresponds to the position of the laser radar at the time of collecting the frame of the original point cloud;
[0031] Step S12: for each frame of original point cloud, transform the initial obstacle data into the laser radar coordinate system to which the original point cloud of the frame belongs, to obtain corresponding transformed obstacle data;
[0032] Step S13: For each frame of original point cloud, merge it with the corresponding transformed obstacle data to obtain point cloud enhancement data corresponding to the original point cloud of the frame.
[0033] In the specific implementation of step S11, the vehicle may be an autonomous driving vehicle equipped with the laser radar, and the laser radar may be installed at an appropriate position of the vehicle (e.g., the top surface of the vehicle) to collect (e.g., collect in real time or periodically) the original point cloud in the preset area in front of or around the vehicle during the driving of the vehicle. The time-sequential multiple frames of original point clouds collected by the laser radar may be collected by the laser radar respectively when the vehicle drives to multiple different positions.
[0034] Each frame of the original point cloud has its own collection time and a laser radar coordinate system corresponding to the position of the laser radar at the collection time. It can be understood that during the driving process of the vehicle, the position of the laser radar is relatively fixed to the position of the vehicle and changes with the position of the vehicle. Accordingly, the laser radar coordinate system is related to the position of the laser radar. In other words, the laser radar coordinate system to which each frame of the original point cloud belongs corresponds to the position of the laser radar at the collection time of the frame of the original point cloud.
[0035] In some embodiments, the laser radar coordinate system to which each frame of the original point cloud belongs can be determined in the following manner: the coordinate origin is the center of mass or geometric center of the laser radar or the laser beam emission center, the x-axis points horizontally to the front of the laser beam emission surface, the y-axis points horizontally to the left side of the laser radar, and the z-axis points vertically upward.
[0036] The initial obstacle data may include multiple three-dimensional point coordinates, which may be determined in the following manner:
[0037] (1) Using actual obstacle images (e.g., images of cartons, sand piles, garbage bags, vehicle parts, large rocks, etc.) or preset obstacle parameter information (e.g., coordinate information of several points, color parameters, texture parameters, etc.) as input data;
[0038] (2) Then, modeling is performed using an existing three-dimensional geometric modeling tool or algorithm to obtain corresponding modeled obstacles, wherein each modeled obstacle has a respective preset three-dimensional coordinate system, and the modeled obstacle is composed of a plurality of three-dimensional points in the preset three-dimensional coordinate system;
[0039] (3) According to the expected obstacle size, the coordinates of each 3D point of the modeled obstacle in step (2) are scaled to meet the expected size, and the modeled obstacle is translated. Specifically, assuming that the coordinates of the center point of the modeled obstacle are (x0, y0, z0), the coordinates of each 3D point of the modeled obstacle can be translated The x-coordinate and y-coordinate of the center point of the obstacle are both 0, and the lowest end of the obstacle is on the two-dimensional plane formed by the x-axis and the y-axis; and the coordinates of each three-dimensional point of the modeled obstacle after translation are used as the initial obstacle data.
[0040] In the specific implementation of step S12, the initial obstacle data is transformed in coordinate space to be transformed into the laser radar coordinate system to which each frame of the original point cloud belongs, so that the transformed obstacle data corresponding to each frame of the original point cloud can be obtained. Through the coordinate space transformation, each group of transformed obstacle data corresponding to the multiple frames of original point clouds can be made to have time sequence.
[0041] Reference Figure 2 , Figure 2 yes Figure 1 Flow chart of a specific implementation of step S12 in the embodiment; in this implementation, the step S12 may specifically include steps S121 to S124.
[0042] In step S121, a target acquisition moment is selected from the acquisition moments to which the multiple frames of original point clouds belong, and the vehicle coordinate system at the target acquisition moment is used as the reference coordinate system, wherein the vehicle coordinate system at the acquisition moment to which each frame of original point cloud belongs corresponds to the position of the vehicle at the acquisition moment.
[0043] The self-vehicle coordinate system may be a special coordinate system preset according to the actual application scenario and used to describe the movement of the vehicle. For example, when the vehicle is on a horizontal road, the self-vehicle coordinate system may be determined by the following method: the origin of the coordinate is the center of mass or geometric center of the vehicle, the x-axis points horizontally to the front of the vehicle, the y-axis points horizontally to the left of the vehicle (or the driver), and the z-axis points vertically upward. It is understandable that during the driving process of the self-vehicle, the self-vehicle coordinate system at different times is related to the position of the self-vehicle at the corresponding time.
[0044] In a specific implementation, the acquisition time of the last frame of original point cloud in the time sequence may be selected from the acquisition times of the multiple frames of original point cloud as the target acquisition time.
[0045] In step S122, the initial obstacle data is transformed into the reference coordinate system to obtain first transformed data.
[0046] In step S123, for each frame of original point cloud, the first transformation data is transformed into the vehicle coordinate system at the time of acquisition of the frame of original point cloud by using the third transformation matrix between the vehicle coordinate system and the reference coordinate system to obtain second transformation data.
[0047] In a specific implementation, the third transformation matrix between the self-vehicle coordinate system at the acquisition moment of each frame of original point cloud and the reference coordinate system can be determined according to the self-vehicle posture corresponding to the self-vehicle coordinate system and the self-vehicle posture corresponding to the reference coordinate system.
[0048] In step S124, based on the fourth transformation matrix between the vehicle coordinate system and the laser radar coordinate system to which the original point cloud of the frame belongs, the second transformed data is transformed to the laser radar coordinate system to which the original point cloud of the frame belongs, so as to obtain the transformed obstacle data.
[0049] In a specific implementation, the fourth transformation matrix between the vehicle coordinate system and the laser radar coordinate system can be determined according to the posture of the vehicle and the posture of the laser radar at the corresponding acquisition time.
[0050] It should be pointed out that in addition to using the above solution to transform the initial obstacle data into the laser radar coordinate system to which each frame of the original point cloud belongs, other appropriate methods may also be used.
[0051] For example, for each frame of original point cloud, the transformation matrix between the preset three-dimensional coordinate system to which the initial obstacle data belongs and the laser radar coordinate system to which the original point cloud of the frame belongs can be directly determined; then, the transformation matrix is used to transform the initial obstacle data into the laser radar coordinate system to which the original point cloud of the frame belongs.
[0052] For another example, for each frame of original point cloud, the initial obstacle data may be first transformed into the camera coordinate system at the time of acquisition of the frame of original point cloud to obtain the corresponding transformed obstacle data; and then, the transformation matrix between the camera coordinate system and the lidar coordinate system to which the frame of original point cloud belongs is used to transform the transformed obstacle data into the lidar coordinate system to which the frame of original point cloud belongs.
[0053] More specifically, the vehicle may also be equipped with a camera (or camera head), which may be installed at an appropriate position of the vehicle (e.g., the top surface, front, rear, side of the vehicle, etc.), and is used to collect (e.g., collect in real time or periodically) original images in front of or within a preset area around the vehicle during the driving process of the vehicle. During the driving process of the vehicle, the positions of the vehicle, the laser radar, and the camera are often relatively fixed.
[0054] In a specific implementation, the camera coordinate system can be determined in the following manner: the origin of the coordinates is the optical center of the camera, the x-axis and the y-axis are respectively parallel to the X-axis and the Y-axis of the image coordinate system to which the original image captured by the camera belongs, and the z-axis is the optical axis of the camera (which is perpendicular to the image plane), wherein the intersection of the optical axis of the camera and the image plane is the origin of the image coordinate system, and the image coordinate system is a two-dimensional rectangular coordinate system. It can be understood that during the driving process of the vehicle, the camera coordinate system at different times is related to the position of the camera at the corresponding time. In other words, the camera coordinate system at the acquisition time of each frame of the original point cloud corresponds to the position of the camera at the acquisition time of the original point cloud of that frame.
[0055] Furthermore, before executing step S12, the method may further include: selecting a target area in the original point cloud of the frame, counting the total number of three-dimensional points in the original point cloud of the frame that fall into the target area, and calculating the quotient of the total number and the volume of the target area to obtain a unit number; determining the volume of the obstacle area formed by the initial obstacle data, and using the product of the volume of the obstacle area and the unit number as the number of sampling points; based on the number of sampling points, performing downsampling processing on the three-dimensional points on the surface of the obstacle area. Further, the downsampling processing may adopt a uniform downsampling method.
[0056] In an embodiment of the present invention, since the number of three-dimensional points in the initial obstacle data is usually large, and the original point cloud collected by the lidar is relatively sparse, the initial obstacle data is downsampled based on the number of three-dimensional points per unit volume in the original point cloud. This helps to effectively reduce the number of three-dimensional points and reduce computing overhead on the one hand, and helps to ensure that the obstacles generated in the original point cloud have a sparsity level consistent with that of the original point cloud as much as possible.
[0057] Continue to refer to Figure 1In the specific implementation of step S13, for each frame of original point cloud, it is merged with the corresponding transformed obstacle data to obtain the point cloud enhancement data corresponding to the frame of original point cloud, so that multi-frame point cloud enhancement data with time sequence can be obtained, which respectively contains obstacle information. Furthermore, the multi-frame point cloud enhancement data with time sequence can better simulate the real driving environment of the autonomous driving vehicle during dynamic driving, so as to be suitable for the training of time sequence models and the simulation test of autonomous driving related algorithms.
[0058] Reference Figure 3 , Figure 3 is a flow chart of another data enhancement method in an embodiment of the present invention; the another data enhancement method may include Figure 1 The steps S11 to S13 shown in the figure may further include the following step S31, which may be performed after step S12 and before step S13. Figure 1 The different contents in the illustrated embodiments are described.
[0059] In step S31, an occlusion analysis is performed on each frame of the original point cloud and the corresponding transformed obstacle data, and it is confirmed that the transformed obstacle data is not occluded.
[0060] In a specific implementation, it is possible to directly determine in the laser radar coordinate system to which each frame of the original point cloud belongs whether there is an overlapping area between the frame's original point cloud and its corresponding transformed obstacle data (for example, manual determination, or automatic determination based on the distance between the projected three-dimensional points) to confirm whether the transformed obstacle data is blocked.
[0061] In another specific method, each frame of the original point cloud and the corresponding transformed obstacle data can be projected into a preset image coordinate system, and it is judged in the image coordinate system (for example, manually judged, or automatically judged based on the distance between the projected pixel points) whether there is an overlapping area between the original point cloud of the frame and the corresponding transformed obstacle data to confirm whether the transformed obstacle data is blocked.
[0062] In another specific implementation, the occlusion analysis can be implemented by combining the coordinate space transformation, the camera coordinate system and its corresponding image coordinate system, the conversion matrix between the two coordinate systems, etc. For details of this implementation, see Figure 4 , Figure 5 and the description below.
[0063] Reference Figure 4 , Figure 4 yes Figure 3A flowchart of a specific implementation of step S31 in the embodiment; in this implementation, the occlusion situation analysis is performed on each frame of the original point cloud and the corresponding transformed obstacle data in step S31, which may specifically include the following steps S41 to S42.
[0064] In step S41, the first transformation matrix between the laser radar coordinate system and the camera coordinate system at the time of acquisition of the frame original point cloud is used to transform the frame original point cloud and the transformed obstacle data to the camera coordinate system respectively to obtain the corresponding camera space point cloud and camera space obstacle data.
[0065] In step S42, based on the camera space point cloud data, the camera space obstacle data, and a second transformation matrix between the camera coordinate system and the corresponding image coordinate system, an occlusion analysis is performed on the frame original point cloud and the corresponding transformed obstacle data.
[0066] The camera coordinate system at the acquisition moment of each frame of the original point cloud corresponds to the position of the camera at the acquisition moment, and the image coordinate system corresponding to the camera coordinate system is the image coordinate system of the original image acquired by the camera at the acquisition moment.
[0067] For the determination method and related instructions of the camera coordinate system and the corresponding image coordinate system, please refer to the previous article and Figure 2 The description of the camera coordinate system and the image coordinate system in the embodiment shown in will not be repeated here.
[0068] Reference Figure 5 , Figure 5 yes Figure 4 FIG. 4 is a flowchart of a specific implementation of step S42 in FIG. 4 . In this implementation, step S42 may specifically include the following steps S421 to S424.
[0069] In step S421, voxelization is performed on the camera space point cloud data and the camera space obstacle data respectively to obtain a corresponding plurality of point cloud voxels and a plurality of obstacle voxels.
[0070] In a specific implementation, the camera space point cloud data and the camera space obstacle data may be voxelized using a preset voxel side length. In a non-limiting manner, the preset voxel side length may be selected from an interval [0.03, 0.06] (in meters) for voxelization, for example, 0.05 meters (i.e., each point cloud voxel and each obstacle voxel may be a voxel with a side length of 0.05 meters).
[0071] In step S422, for a direction vector pointing from the origin of the camera coordinate system to each pixel position of the corresponding image coordinate system, a plurality of sampling points are determined on the direction vector according to a preset sampling method.
[0072] In a specific implementation, the preset sampling method may be: for each direction vector, a sampling point is acquired on the direction vector at every preset distance (for example, every 0.2 meter), thereby obtaining the multiple sampling points on the direction vector.
[0073] In step S423, a direction vector having at least a first preset number of sampling points falling into the point cloud voxel is selected and recorded as a target vector.
[0074] In step S424, if at least a second preset number of target vectors satisfy the following condition, it is confirmed that the transformed obstacle data is occluded: among the sampling points on the target vector that fall into the point cloud voxel, at least a third preset number of sampling points have a distance from the origin of the camera coordinate system that is less than a distance between any sampling point on the target vector that falls into the obstacle pixel and the origin of the camera coordinate system.
[0075] In a specific implementation, the first preset number, the second preset number, and the third preset number can be appropriately set according to actual needs. For example, for a scene with higher accuracy requirements, the smaller the value of each preset number is set. As a non-limiting embodiment, the first preset number, the second preset number, and the third preset number can all be set to 1.
[0076] In an embodiment of the present invention, the camera coordinate system, the corresponding image coordinate system, and the conversion matrix between the two are combined to perform occlusion analysis on each frame of the original point cloud and the corresponding transformed obstacle data. Compared with the occlusion analysis based only on a single coordinate system (such as a lidar coordinate system or a camera coordinate system), the accuracy of the analysis and judgment results can be improved, which is conducive to obtaining better quality and more reliable enhanced data.
[0077] Furthermore, when it is confirmed that the transformed obstacle data corresponding to the original point cloud is occluded, the transformed obstacle data is repeatedly offset, and after each offset, the occlusion situation of the original point cloud and the obstacle data after the current offset is analyzed; if the obstacle data after the current offset is not occluded and the number of repetitions is less than a preset number, the obstacle data after the current offset is merged with the original point cloud to obtain point cloud enhanced data corresponding to the original point cloud.
[0078] In a specific implementation, a different coordinate offset may be used each time to offset the transformed obstacle data.
[0079] Furthermore, if the number of repetitions reaches the preset number of times and the obstacle data after the last offset is still blocked, the source data sequence to be enhanced is replaced; and data enhancement is performed based on the replaced source data sequence to be enhanced and the initial obstacle data.
[0080] Furthermore, the source data sequence to be enhanced also includes a plurality of frames of original images with time sequence collected by the camera for the same target scene during the driving process of the vehicle, each frame of the original image has its own image coordinate system, and the image coordinate system corresponds to the position of the camera at the acquisition time of the original point cloud of the frame, wherein the plurality of frames of original point clouds with time sequence correspond to the plurality of frames of original images one by one, and the corresponding original images are consistent with the acquisition time of the original point clouds; accordingly, the data enhancement method may also include: for each frame of original point cloud, transforming the obstacle data of the initial obstacle data in the laser radar coordinate system to which the original point cloud of the frame belongs, to the image coordinate system to which the original image corresponding to the original point cloud of the frame belongs, to obtain corresponding image obstacle data; performing a rendering operation based on the image obstacle data to obtain rendered image data (also referred to as a two-dimensional rendered image), wherein each pixel of the rendered image data has its own pixel value; using the pixel value of each pixel of the rendered image data to replace the pixel value of the pixel at the same position in the original image of the frame, to obtain corresponding image enhancement data.
[0081] Furthermore, a specific scheme for performing coordinate space transformation on the transformed obstacle data corresponding to each frame of the original point cloud to obtain the corresponding image obstacle data may include: using a first transformation matrix between the laser radar coordinate system and the camera coordinate system at the acquisition time of the frame of the original point cloud, transforming the transformed obstacle data to the camera coordinate system, and obtaining the corresponding camera space point cloud; using a second transformation matrix between the camera coordinate system and the corresponding image coordinate system (i.e., the image coordinate system to which the original image corresponding to the frame of the original point cloud belongs), transforming the camera space point cloud to the image coordinate system, and obtaining the corresponding image obstacle data. In a specific implementation, the rendering operation may be performed in the following manner based on the raster method:
[0082] Step (1.1), set the scene parameters. Before rendering, you need to set the scene's lighting parameters, camera parameters, and material parameters; lighting parameters include the position, color, and intensity of the light source, camera parameters include viewing angle, projection mode, and camera position, and material parameters include the color, texture, and reflection properties of the object surface.
[0083] Step (1.2), vertex coloring. Define the primitive structure as a triangular primitive structure, that is, the three-dimensional model of the object is composed of many triangular vertices; according to the camera parameters, project all vertices of each three-dimensional model onto the image plane, calculate the position of each vertex on the image, and obtain the corresponding triangular facet.
[0084] Step (1.3), fragment coloring. For each pixel, determine whether the pixel is inside the boundary of the triangle; if the judgment result is yes, use the surface normal and color of the triangle to update the color value of the current pixel.
[0085] Step (1.4), render all triangles. Repeat step (1.3) to perform fragment shading operations on all triangles; by updating the color value of each pixel according to the position and attributes of the triangle, a complete rendered image is finally formed. This process will be performed pixel by pixel on the image plane until all triangles are rendered. Finally, a two-dimensional rendered image corresponding to the initial obstacle data will be rendered on a pure white background image.
[0086] Existing data enhancement technologies usually do not take multimodal information into consideration (different modal information may refer to different types of autonomous driving scene data, such as scene images, scene point clouds, etc.), and only generate abnormal obstacles in two-dimensional images or three-dimensional point clouds. There is no time alignment between visual sensors and radar sensors, and no position alignment in three-dimensional space. Therefore, the enhanced data obtained cannot be applied to the training of multimodal models.
[0087] In comparison, in an embodiment of the present invention, a multi-frame original point cloud with time sequence collected by a lidar and a multi-frame original image with time sequence collected by a camera (or camera) and aligned one by one (time sequence alignment and spatial position alignment) are used as the basic data for performing data enhancement. This can effectively perform joint data enhancement on images and point clouds to meet the conversion of different postures between sensors, and is suitable for multimodal model training. At the same time, it can effectively generate stable continuous frame enhanced data, better simulate the sensor imaging effect in a real environment, and is suitable for time series model training and simulation experiments.
[0088] Figure 6 : is a schematic diagram of the structure of a data enhancement device in an embodiment of the present invention. The device may include:
[0089] The to-be-enhanced data acquisition module 61 is used to determine the initial obstacle data and the to-be-enhanced source data sequence, wherein the to-be-enhanced source data sequence includes multiple frames of original point clouds with time sequence collected by the laser radar during the driving process of the vehicle, wherein each frame of the original point cloud has its own laser radar coordinate system, and the laser radar coordinate system corresponds to the position of the laser radar at the time of collecting the frame of the original point cloud;
[0090] A coordinate space transformation module 62 is used to transform the initial obstacle data into the laser radar coordinate system to which the original point cloud of each frame belongs, so as to obtain corresponding transformed obstacle data;
[0091] The enhanced data determination module 63 is used to merge each frame of the original point cloud with the corresponding transformed obstacle data to obtain point cloud enhanced data corresponding to the frame of the original point cloud.
[0092] For the principle, specific implementation and beneficial effects of the data enhancement device, please refer to the previous article and Figures 1 to 5 The relevant description about the data enhancement method shown in any embodiment will not be repeated here.
[0093] The embodiment of the present invention further provides a storage medium, such as a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor. Figures 1 to 5 The steps of the data enhancement method shown in any embodiment. The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.
[0094] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0095] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).
[0096] The embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned Figures 1 to 5 The steps of the data enhancement method shown in any embodiment.
[0097] The embodiment of the present invention further provides a computer program product, including a computer program, wherein the computer program is executed by a processor Figures 1 to 5 The steps of the data enhancement method shown in any embodiment.
[0098] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.
[0099] The "plurality" appearing in the embodiments of the present application refers to two or more.
[0100] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.
[0101] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.
[0102] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.
Claims
1. A data enhancement method, characterized in that: include: Determine initial obstacle data and a sequence of source data to be enhanced, wherein the sequence of source data to be enhanced includes multiple frames of original point clouds with time sequence collected by a laser radar during the driving process of the vehicle, wherein each frame of original point cloud has its own laser radar coordinate system, and the laser radar coordinate system corresponds to the position of the laser radar at the time of collecting the frame of original point cloud; For each frame of original point cloud, the initial obstacle data is transformed into the laser radar coordinate system to which the original point cloud of the frame belongs, so as to obtain corresponding transformed obstacle data; For each frame of original point cloud, it is merged with the corresponding transformed obstacle data to obtain the point cloud enhancement data corresponding to the original point cloud of the frame.
2. The method according to claim 1, characterized in that: Before merging each frame of the original point cloud with the corresponding transformed obstacle data, the method further includes: An occlusion analysis is performed on each frame of the original point cloud and the corresponding transformed obstacle data, and it is confirmed that the transformed obstacle data is not occluded.
3. The method according to claim 2, characterized in that The occlusion analysis of each frame of the original point cloud and the corresponding transformed obstacle data includes: Using a first transformation matrix between a laser radar coordinate system and a camera coordinate system at the time of acquisition of the frame original point cloud, the frame original point cloud and the transformed obstacle data are respectively transformed to the camera coordinate system to obtain corresponding camera space point cloud and camera space obstacle data; Based on the camera space point cloud data, the camera space obstacle data, and a second transformation matrix between the camera coordinate system and the corresponding image coordinate system, performing an occlusion analysis on the original point cloud of the frame and the corresponding transformed obstacle data; The camera coordinate system at the acquisition moment of each frame of the original point cloud corresponds to the position of the camera at the acquisition moment, and the image coordinate system corresponding to the camera coordinate system is the image coordinate system of the original image acquired by the camera at the acquisition moment.
4. The method according to claim 3, characterized in that: Based on the camera space point cloud data, the camera space obstacle data, and a second transformation matrix between the camera coordinate system and the corresponding image coordinate system, an occlusion analysis is performed on the frame original point cloud and the corresponding transformed obstacle data, including: voxelize the camera space point cloud data and the camera space obstacle data to obtain a corresponding plurality of point cloud voxels and a plurality of obstacle voxels; For a direction vector pointing from the origin of the camera coordinate system to each pixel position of the corresponding image coordinate system, a plurality of sampling points are determined on the direction vector according to a preset sampling method; Selecting a direction vector having at least a first preset number of sampling points falling into the point cloud voxel, recorded as a target vector; If at least a second preset number of target vectors satisfy the following condition, it is confirmed that the transformed obstacle data is occluded: among the sampling points on the target vector that fall into the point cloud voxel, at least a third preset number of sampling points have a distance from the origin of the camera coordinate system that is less than a distance between any sampling point on the target vector that falls into the obstacle pixel and the origin of the camera coordinate system.
5. The method according to claim 2, characterized in that: The method further comprises: When it is confirmed that the transformed obstacle data corresponding to the original point cloud is occluded, the transformed obstacle data is repeatedly offset, and after each offset, the occlusion situation of the original point cloud and the obstacle data after the current offset is analyzed; if the obstacle data after the current offset is not occluded and the number of repetitions is less than a preset number, the obstacle data after the current offset is merged with the original point cloud to obtain point cloud enhanced data corresponding to the original point cloud.
6. The method according to claim 5, characterized in that The method further comprises: If the number of repetitions reaches the preset number of times, and the obstacle data after the last offset is still blocked, then the source data sequence to be enhanced is replaced; Data enhancement is performed based on the replaced source data sequence to be enhanced and the initial obstacle data.
7. The method according to claim 1, characterized in that For each frame of the original point cloud, the initial obstacle data is transformed into the laser radar coordinate system to which the original point cloud of the frame belongs to obtain the corresponding transformed obstacle data, including: A target acquisition time is selected from each acquisition time to which the multiple frames of original point clouds belong, and the vehicle coordinate system at the target acquisition time is used as the reference coordinate system, wherein the vehicle coordinate system at the acquisition time to which each frame of original point cloud belongs corresponds to the position of the vehicle at the acquisition time; Transforming the initial obstacle data into the reference coordinate system to obtain first transformed data; For each frame of original point cloud, using a third transformation matrix between the vehicle coordinate system at the time of acquisition of the frame of original point cloud and the reference coordinate system, the first transformation data is transformed into the vehicle coordinate system to obtain second transformation data; Based on a fourth transformation matrix between the vehicle coordinate system and the laser radar coordinate system to which the original point cloud of the frame belongs, the second transformed data is transformed into the laser radar coordinate system to which the original point cloud of the frame belongs to obtain the transformed obstacle data.
8. The method according to claim 1 or 7, characterized in that: Before transforming the initial obstacle data to the laser radar coordinate system to which the original point cloud of each frame belongs, the method further includes: Selecting a target area in the original point cloud of the frame, counting the total number of three-dimensional points in the original point cloud of the frame that fall into the target area, and calculating the quotient of the total number and the volume of the target area to obtain a unit number; Determine the volume of the obstacle area formed by the initial obstacle data, and use the product of the volume of the obstacle area and the number of units as the number of sampling points; Based on the number of sampling points, down-sampling processing is performed on the three-dimensional points of the surface belonging to the obstacle area.
9. The method according to claim 1, characterized in that: The source data sequence to be enhanced further includes multiple frames of original images with time sequence collected by the camera for the same target scene during the driving process of the vehicle, each frame of the original image has its own image coordinate system, and the image coordinate system corresponds to the position of the camera at the time of collecting the original point cloud of the frame, wherein the multiple frames of original point clouds with time sequence correspond to the multiple frames of original images one by one, and the corresponding original images are consistent with the collection time of the original point clouds; The method further comprises: For each frame of original point cloud, transform the initial obstacle data in the laser radar coordinate system to which the original point cloud belongs to into the image coordinate system to which the original image corresponding to the original point cloud belongs to, to obtain the corresponding image obstacle data; Performing a rendering operation based on the image obstacle data to obtain rendered image data, wherein each pixel of the rendered image data has a respective pixel value; The pixel value of each pixel of the rendered image data is used to replace the pixel value of the pixel at the same position in the original image of the frame to obtain corresponding image enhancement data.
10. A data enhancement device, characterized in that: include: The data acquisition module to be enhanced is used to determine the initial obstacle data and the source data sequence to be enhanced, wherein the source data sequence to be enhanced includes multiple frames of original point clouds with time sequence collected by the laser radar during the driving process of the vehicle, wherein each frame of the original point cloud has its own laser radar coordinate system, and the laser radar coordinate system corresponds to the position of the laser radar at the time of collecting the original point cloud of the frame; the coordinate space transformation module is used to transform the initial obstacle data to the laser radar coordinate system to which the original point cloud of the frame belongs for each frame of the original point cloud, so as to obtain the corresponding transformed obstacle data; The enhanced data determination module is used to merge each frame of the original point cloud with the corresponding transformed obstacle data to obtain the point cloud enhanced data corresponding to the original point cloud of the frame.
11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data enhancement method according to any one of claims 1 to 10 are performed.
12. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the data enhancement method according to any one of claims 1 to 10 are performed.
13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data enhancement method according to any one of claims 1 to 10 are performed.