Point cloud data enhancement method and device for mine area automatic driving dataset building

Abstract

The application discloses a point cloud data enhancement method and device for mine area automatic driving dataset construction, a storage medium and a computer device. The method comprises the following steps: determining a to-be-enhanced point cloud frame from a plurality of basic point cloud frames, wherein the basic point cloud frames are obtained based on a conventional mine area environment; determining at least one target point cloud from a plurality of sample point clouds, wherein each sample point cloud corresponds to a target obstacle and is provided with a region label, and the target point cloud is obtained based on a preset environment; and superimposing the at least one target point cloud into the to-be-enhanced point cloud frame based on the region label of the target point cloud, so as to obtain an enhanced point cloud frame. The at least one target point cloud corresponding to the target obstacle is superimposed into the to-be-enhanced point cloud frame, and then the enhanced point cloud frame is obtained, so that the sample richness can be greatly improved, and meanwhile, the operability is relatively high.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a point cloud data augmentation method, device, storage medium, and computer equipment for building autonomous driving datasets in mining areas. Background Technology

[0002] In autonomous driving solutions for mining equipment within mining environments, deep learning methods based on LiDAR point clouds are an effective means to improve 3D environmental perception performance. However, when acquiring sample data for training deep learning models, the richness of the sample data is often affected by the difficulty in collecting data on moving obstacles within the mining area. On the one hand, because mining areas are not open environments, many non-production vehicles, such as vans and pickup trucks, can only enter the limited mining environment under limited right-of-way conditions, and their entry time is strictly controlled. On the other hand, the scenarios and postures of engineering vehicles or non-production vehicles that can be collected in the mining area are limited. Under these circumstances, point cloud data augmentation methods commonly used on public roads, such as random rotation and random flipping, can only increase the richness of sample data by a small amount and cannot fundamentally solve the problem of severely insufficient sample data richness in mining environments. Summary of the Invention

[0003] In view of this, this application provides a point cloud data augmentation method, device, storage medium, and computer equipment for building a mining area autonomous driving dataset. By superimposing at least one target point cloud corresponding to a target obstacle onto the point cloud frame to be augmented, an augmented point cloud frame is obtained, which can greatly improve the sample richness and has strong operability.

[0004] According to one aspect of this application, a point cloud data augmentation method for constructing autonomous driving datasets in mining areas is provided, comprising:

[0005] The point cloud frame to be enhanced is determined from multiple base point cloud frames, which are obtained based on a conventional mining environment;

[0006] At least one target point cloud is determined from multiple sample point clouds, each sample point cloud corresponding to a target obstacle and bearing a region label, the target point cloud being obtained based on a preset environment;

[0007] Based on the region label of the target point cloud, the at least one target point cloud is superimposed onto the point cloud frame to be enhanced to obtain the enhanced point cloud frame.

[0008] According to another aspect of this application, a point cloud data augmentation device for building autonomous driving datasets in mining areas is provided, comprising:

[0009] The first determining module is used to determine the point cloud frame to be enhanced from multiple basic point cloud frames, wherein the basic point cloud frames are obtained based on a conventional mining environment;

[0010] The second determining module is used to determine at least one target point cloud from multiple sample point clouds, each sample point cloud corresponding to a target obstacle and having a region label, the target point cloud being obtained based on a preset environment;

[0011] The point cloud overlay module is used to overlay the at least one target point cloud onto the point cloud frame to be enhanced based on the region label of the target point cloud, so as to obtain the enhanced point cloud frame.

[0012] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described point cloud data augmentation method for building a mining area autonomous driving dataset.

[0013] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. When the processor executes the program, it implements the above-described point cloud data augmentation method for building a mining area autonomous driving dataset.

[0014] Using the above technical solution, this application provides a point cloud data augmentation method, apparatus, storage medium, and computer equipment for building autonomous driving datasets in mining areas. First, a point cloud frame to be augmented can be selected from multiple basic point cloud frames. Next, one or more target point clouds can be determined from multiple sample point clouds. Here, each sample point cloud can carry a region label. After determining one or more target point clouds, the corresponding target point clouds can be superimposed onto the aforementioned point cloud frame to be augmented, based on the region label corresponding to each target point cloud, ultimately obtaining the augmented point cloud frame. This application's embodiment, by superimposing at least one target point cloud corresponding to a target obstacle onto the point cloud frame to be augmented to obtain the augmented point cloud frame, can significantly improve sample richness and has strong operability.

[0015] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 The illustration shows a flowchart of a point cloud data augmentation method for building an autonomous driving dataset in a mining area, as provided in an embodiment of this application.

[0018] Figure 2 This paper illustrates a flowchart of another point cloud data augmentation method for building an autonomous driving dataset in a mining area, as provided in an embodiment of this application.

[0019] Figure 3 This illustration shows a schematic diagram of a target occlusion area provided in an embodiment of this application;

[0020] Figure 4 The diagram shows a schematic of a point cloud data augmentation device for building a dataset for autonomous driving in mining areas, as provided in an embodiment of this application. Detailed Implementation

[0021] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0022] This embodiment provides a point cloud data augmentation method for building a dataset for autonomous driving in mining areas, such as... Figure 1 As shown, the method includes:

[0023] Step 101: Determine the point cloud frame to be enhanced from multiple basic point cloud frames, wherein the basic point cloud frames are obtained based on a conventional mining area environment;

[0024] The point cloud data augmentation method for building a mining area autonomous driving dataset provided in this application firstly selects a point cloud frame to be augmented from multiple basic point cloud frames. Specifically, the point cloud frame to be augmented can be randomly selected from the basic point cloud frames or obtained according to other preset rules. The basic point cloud frames can include typical scenes and vehicles in a typical mining area environment; that is, the object point clouds in the basic point cloud frames can be point clouds corresponding to objects frequently encountered by autonomous driving equipment in the mining area during daily operation.

[0025] Step 102: Determine at least one target point cloud from multiple sample point clouds. Each sample point cloud corresponds to a target obstacle and has a region label. The target point cloud is obtained based on a preset environment.

[0026] In this embodiment, one or more target point clouds can then be determined from multiple sample point clouds. Here, the sample point clouds can be point clouds corresponding to target obstacles. Target obstacles can be objects that the autonomous driving equipment in the mining area can only encounter at specific times or under specific conditions. For example, since the mining area is a non-open environment, many non-production vehicles, such as vans and pickup trucks, can only enter the limited mining environment when they have limited right-of-way, and the entry time is strictly controlled. Therefore, these non-production vehicles, such as vans and pickup trucks, can serve as target obstacles. Furthermore, target obstacles can also be objects in specific locations and postures. For example, some collectable engineering vehicles or non-production vehicles have limited postures in the mining area, so these engineering vehicles or non-production vehicles in specific locations and postures can also serve as target obstacles. This embodiment constructs a preset environment that can include these target obstacles that can only be encountered at specific times and under specific conditions, and can acquire point clouds of these target obstacles, which is very beneficial for increasing sample richness later. Similarly, the target point cloud can be randomly selected from the sample point cloud or obtained according to other preset rules. Each sample point cloud can have a region label. The region label can include the position and size of the target obstacle's 3D detection box or bounding box, as well as the target obstacle's heading.

[0027] Step 103: Based on the region label of the target point cloud, superimpose the at least one target point cloud onto the point cloud frame to be enhanced to obtain the enhanced point cloud frame.

[0028] In this embodiment, after determining one or more target point clouds, the corresponding target point clouds can be superimposed on the aforementioned point cloud frame to be enhanced, based on the region label corresponding to each target point cloud, to finally obtain the enhanced point cloud frame. The enhanced point cloud frame can then be used to train the deep learning model for autonomous driving, that is, as a sample for the deep learning model for autonomous driving.

[0029] By applying the technical solution of this embodiment, firstly, a point cloud frame to be enhanced can be selected from multiple basic point cloud frames. Next, one or more target point clouds can be determined from multiple sample point clouds. Here, each sample point cloud can carry a region label. After determining one or more target point clouds, each target point cloud can be superimposed onto the aforementioned point cloud frame to be enhanced, based on its corresponding region label, ultimately obtaining the enhanced point cloud frame. This embodiment of the application, by superimposing at least one target point cloud corresponding to a target obstacle onto the point cloud frame to be enhanced, thereby obtaining the enhanced point cloud frame, can significantly improve sample richness and offers strong operability.

[0030] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another point cloud data augmentation method for building autonomous driving datasets in mining areas is provided, such as... Figure 2 As shown, the method includes:

[0031] Step 201: Determine the point cloud frame to be enhanced from multiple basic point cloud frames, wherein the basic point cloud frames are obtained based on a conventional mining environment;

[0032] In this embodiment, firstly, a point cloud frame to be enhanced can be selected from multiple base point cloud frames. Specifically, the point cloud frame to be enhanced can be randomly selected from the base point cloud frames, or it can be obtained according to other preset rules. The base point cloud frames can include typical scenes and vehicles in a typical mining environment; that is, the object point clouds in the base point cloud frames can be the point clouds corresponding to objects frequently encountered by autonomous driving equipment in the mining area during daily operation.

[0033] Step 202: Determine at least one target point cloud from multiple sample point clouds. Each sample point cloud corresponds to a target obstacle and has a region label. The target point cloud is obtained based on a preset environment.

[0034] In this embodiment, one or more target point clouds can then be determined from multiple sample point clouds. Here, the sample point clouds can be point clouds corresponding to target obstacles. Target obstacles can be objects that the autonomous driving equipment in the mining area can only encounter at specific times or under specific conditions. For example, since the mining area is a non-open environment, many non-production vehicles, such as vans and pickup trucks, can only enter the limited mining environment when they have limited right-of-way, and the entry time is strictly controlled. Therefore, these non-production vehicles, such as vans and pickup trucks, can serve as target obstacles. Furthermore, target obstacles can also be objects in specific locations and postures. For example, some collectable engineering vehicles or non-production vehicles have limited postures in the mining area, so these engineering vehicles or non-production vehicles in specific locations and postures can also serve as target obstacles. This embodiment constructs a preset environment that can include these target obstacles that can only be encountered at specific times and under specific conditions, and can acquire point clouds of these target obstacles, which is very beneficial for increasing sample richness later. Similarly, the target point cloud can be randomly selected from the sample point cloud or obtained according to other preset rules. Each sample point cloud can have a region label. The region label can include the position and size of the target obstacle's 3D detection box or bounding box, as well as the target obstacle's heading.

[0035] Step 203: Based on the region label of any of the target point clouds, determine the target placement position of any of the target point clouds in the point cloud frame to be enhanced;

[0036] In this embodiment, since each target point cloud corresponds to a region label, the location of the 3D detection box or bounding box of the target obstacle corresponding to each target point cloud can be identified from the region label, and the target placement position of the corresponding target point cloud in the point cloud frame to be enhanced can be determined based on the location of the 3D detection box or bounding box of the target obstacle corresponding to the target point cloud. Since the target point cloud and the point cloud frame to be enhanced can be placed in the same coordinate system, the target placement position in the point cloud frame to be enhanced can be determined based on the location of the 3D detection box or bounding box of the target obstacle corresponding to the target point cloud.

[0037] Step 204: Based on the target placement location, determine whether any of the target point clouds have non-ground point cloud points in the ground placement area of ​​the point cloud frame to be enhanced;

[0038] In this embodiment, after determining the target placement location of any target point cloud, it can be further determined whether there are non-ground point cloud points within the ground placement area of ​​the target point cloud frame to be enhanced. Here, the ground placement area refers to the area occupied by the target point cloud on the ground plane. The point cloud points in the point cloud frame to be enhanced can be divided into ground point cloud points and non-ground point cloud points. Ground point cloud points are those belonging to the ground, while non-ground point cloud points are all other point cloud points besides the ground point cloud points.

[0039] Step 205: When there are no non-ground point cloud points in the ground placement area, overlay any of the target point clouds onto the target placement position in the point cloud frame to be enhanced.

[0040] In this embodiment, if it is determined that there are no non-ground point cloud points in the ground placement area, then it is reasonable to place the target point cloud there. In this case, the target point cloud can be superimposed onto the target placement location in the point cloud frame to be enhanced. Furthermore, the target point cloud can also have a name tag, which can include the name of the target obstacle corresponding to the target point cloud, such as a van or pickup truck. After superimposing the target point cloud onto the point cloud frame to be enhanced, the name tag and region tag corresponding to the target point cloud can be added to the tag file corresponding to the point cloud frame to be enhanced, making the information included in the tag file of the point cloud frame to be enhanced more comprehensive.

[0041] Step 206: Based on the target placement position of any of the target point clouds, determine the target occlusion area corresponding to any of the target point clouds, determine the target point cloud points corresponding to the target occlusion area from the first point cloud points, and delete the target point cloud points from the point cloud frame to be enhanced.

[0042] In this embodiment, after determining the target placement position corresponding to each target point cloud, the target occlusion area of ​​the corresponding target point cloud can also be determined based on the target placement position. Here, the target placement position can be the three-dimensional spatial location occupied by the target point cloud in the point cloud frame to be enhanced. When the autonomous driving equipment in the mining area uses laser scanning to acquire the point cloud frame, the area corresponding to the side of the target point cloud facing the horizontal laser emission can be determined as the target occlusion area. Figure 3 As shown, the right side of the cuboid faces the horizontal laser emission surface, and the area corresponding to this side is the target occlusion region. During the horizontal laser emission process, the target occlusion region obstructs the laser emitted by the laser emitter, preventing objects on the side furthest from the laser emitter from being scanned, using the target occlusion region as the dividing surface. Therefore, target point cloud points that cannot be obtained due to the target occlusion region can be identified from the first point cloud point set and deleted from the point cloud frame to be enhanced, resulting in the enhanced point cloud frame. Here, the first point cloud point set refers to the point cloud points in the point cloud frame to be enhanced. This application, by deleting target point cloud points that cannot appear due to the influence of the target occlusion region from the point cloud frame to be enhanced, makes the enhanced point cloud frame conform to the principle of lidar point cloud imaging, resulting in a more accurate and realistic enhanced point cloud frame.

[0043] Optionally, in this embodiment of the application, before step 204, "determining whether any of the target point clouds has non-ground point cloud points in the ground placement area of ​​the point cloud frame to be enhanced", the method further includes: determining the target attribute corresponding to each first point cloud point in the point cloud frame to be enhanced based on a preset ground fitting algorithm, and marking the first point cloud points one by one using the target attribute, wherein the target attribute includes ground point cloud points and non-ground point cloud points.

[0044] In this embodiment, before determining whether there are non-ground point cloud points in the ground placement area, each first point cloud point in the point cloud frame to be enhanced can be marked with target attributes. These target attributes can include both ground point cloud points and non-ground point cloud points. Specifically, a preset ground fitting algorithm can be used to fit the ground. Since the ground is a flat, open point cloud, the preset ground fitting algorithm can fit the ground plane; for example, RANSAC can be used for global ground fitting, or the ground can be fitted after dividing the area into several regions. Then, based on the ground fitting results, the relationship between each first point cloud point and the fitted ground plane can be determined, ultimately identifying whether each first point cloud point is a ground point cloud point or a non-ground point cloud point.

[0045] Optionally, in this embodiment of the application, when there are non-ground point cloud points in the ground placement area, the method further includes: if the number of non-ground point cloud points is less than or equal to a preset number threshold, superimposing any of the target point clouds onto the target placement position in the point cloud frame to be enhanced; if the number of non-ground point cloud points is greater than the preset number threshold, rotating the target placement position based on a preset angle, and judging the number of non-ground point cloud points in the corresponding ground placement area again based on the rotated target placement position, until the number of non-ground point cloud points in the ground placement area is less than or equal to the preset number threshold, or the number of rotations exceeds a preset number threshold.

[0046] In this embodiment, if non-ground point cloud points exist in the ground placement area, the next step of judgment can be performed. If the number of non-ground point cloud points in the ground placement area is less than or equal to a preset threshold, it may be due to an error in the ground plane fitting. In this case, the corresponding target point cloud can be superimposed onto the point cloud frame to be enhanced. If the number of non-ground point cloud points in the ground placement area is greater than the preset threshold, it indicates that superimposing the target point cloud at this target placement location is unreasonable. In this case, the target placement location of the target point cloud can be rotated. The base point of rotation can be determined according to a preset rotation rule, and a preset angle can be rotated each time. After rotation, the corresponding ground placement area can be determined based on the rotated target placement location, and the number of non-ground point cloud points in the ground placement area corresponding to the rotated target placement location can be judged again. The above judgment can be performed again based on the number until the number of non-ground point cloud points in the ground placement area is less than or equal to the preset threshold, or until the number of rotations of the target placement location exceeds the preset threshold. Each time the target placement location is rotated, the region label corresponding to the target point cloud can be updated. In this embodiment, when the target placement position is unreasonable, the target placement position is continuously rotated to find a reasonable target placement position. This avoids the problem of difficulty in improving sample richness caused by abandoning stacking due to unreasonable target placement position, and is conducive to steadily increasing sample richness.

[0047] Optionally, before step 201, the method further includes: acquiring a first point cloud frame collected by the autonomous driving device in a conventional mining environment, generating a tag file corresponding to the first point cloud frame, and obtaining the basic point cloud frame based on the first point cloud frame and the tag file; acquiring a second point cloud frame collected by the autonomous driving device in a preset environment, and determining the sample point cloud based on the second point cloud frame, wherein the preset environment includes the target obstacle.

[0048] In this embodiment, before determining the point cloud frame to be enhanced, a first point cloud frame can be acquired by the autonomous driving equipment in a conventional mining environment. Here, the conventional mining environment refers to the daily working environment of the autonomous driving equipment. Specifically, the first point cloud frame can be acquired by the autonomous driving equipment under different weather and time conditions in a conventional mining environment. After acquiring the first point cloud frame, different objects in the first point cloud frame can be identified and labeled to generate corresponding tag files. Specifically, the tag files can include object name information, object location information, object size information, and object heading information for different objects. Then, based on the first point cloud frame and the corresponding tag file, a base point cloud frame can be obtained; that is, the base point cloud frame is the first point cloud frame with the tag file. Subsequently, when using the base point cloud frame as the point cloud frame to be enhanced and overlaying the target point cloud onto the point cloud frame to be enhanced, the tag file can be updated using the name and region tags of the target point cloud.

[0049] In addition, a second point cloud frame can be acquired by the autonomous driving equipment in the mining area under a preset environment. Then, a sample point cloud can be determined based on this second point cloud frame. The preset environment may include target obstacles. The second point cloud frame can be a point cloud frame showing the target obstacle at different positions and relative poses from the autonomous driving equipment. Specifically, the autonomous driving equipment for acquiring the second point cloud frame can be set up on an open platform. The autonomous driving equipment is placed in the center of the platform, and the platform size is determined based on the actual mining area conditions, prioritizing openness and a large size. If the platform is small, the position of the autonomous driving equipment on the platform can be adjusted accordingly to allow sufficient space for the movement of the target obstacle. When the position of the autonomous driving equipment is fixed, the target obstacle is placed on the platform, and point cloud frames showing its movement relative to the autonomous driving equipment at various positions and relative poses are acquired. Then, a sample point cloud can be extracted from the second point cloud frame, that is, the point cloud corresponding to the target obstacle can be extracted from the second point cloud frame.

[0050] Optionally, in this embodiment, the step of "determining the sample point cloud based on the second point cloud frame" includes: sorting the second point cloud frames based on their acquisition time, determining reference frames and search frames corresponding to each reference frame based on the sorted second point cloud frames; identifying the target obstacle in the reference frame, and labeling the point cloud corresponding to the target obstacle using first target information to obtain a first sample point cloud, wherein the first target information includes the region label; and determining the target attribute corresponding to each second point cloud point in the search frame based on the preset ground fitting algorithm. Based on the non-ground point cloud points in the second point cloud and the first target information, the largest cluster closest to the target obstacle in the reference frame is determined using the Euclidean distance clustering algorithm. The largest cluster is then used as the point cloud corresponding to the target obstacle in the search frame. Based on the point cloud corresponding to the target obstacle in the search frame, a target bounding box is determined. Based on the target bounding box and the first target information, second target information corresponding to the target obstacle is determined. The second target information is then used to label the point cloud corresponding to the target obstacle in the search frame to obtain a second sample point cloud.

[0051] In this embodiment, when extracting sample point clouds from the second point cloud frames, firstly, the second point cloud frames corresponding to the same target obstacle can be sorted according to their acquisition time, specifically, they can be sorted in chronological order of acquisition time. After sorting, reference frames and search frames corresponding to each reference frame can be determined. For example, a reference frame is determined for every 10 second point cloud frames, specifically frame 1, frame 11, frame 21, etc. Frames 2-10 are search frames corresponding to frame 1, frames 12-20 are search frames corresponding to frame 2, and so on. After determining the reference frames and search frames, the target obstacle in the reference frames can be identified, and the point cloud corresponding to the target obstacle can be labeled based on the first target information. After labeling, the first sample point cloud can be obtained. The first target information can be manually calculated and organized. Subsequently, a pre-defined ground fitting algorithm can be used to determine the target attributes of each second point cloud point in the search frame. Here, the search frame refers to the frame following the reference frame, i.e., the point cloud frame acquired in the frame after the reference frame. The target attributes also include ground point cloud points and non-ground point cloud points. For example, if the reference frame is frame 1, then the search frame is frame 2. After labeling, the largest cluster closest to the target obstacle in the reference frame can be determined from the second point cloud points labeled as non-ground point cloud points. Specifically, the position of the target obstacle in the reference frame can be determined based on the first target information, and the largest cluster closest to the position of the target obstacle in the reference frame can be determined from the non-ground point cloud points in the second point cloud points of the search frame using the Euclidean distance clustering algorithm. Then, this largest cluster can be used as the point cloud corresponding to the target obstacle in the search frame.

[0052] After determining the point cloud corresponding to the target obstacle in the search frame, the target bounding box of the point cloud corresponding to the target obstacle can be further determined. After determining the target bounding box, two heading information can also be determined. Then, based on the target heading of the target obstacle in the first target information, one of the two heading information is finally determined as the target heading corresponding to the target obstacle in the search frame. Then, the length, width, height, and other dimensions and position information of the target bounding box can be obtained. Based on this information and the target heading, the region label corresponding to the target obstacle in the search frame is determined. In addition, some other information can be obtained, and multiple pieces of information, including the region label, are composed of second target information. Finally, the second target information is used to label the point cloud corresponding to the target obstacle in the search frame, thereby obtaining the second sample point cloud corresponding to the search frame. Here, each first sample point cloud and each second sample point cloud constitute the final sample point cloud.

[0053] Since the target obstacles to be added in the mining area are of the type such as pickup trucks and vans, and their outlines are relatively clear, the L-shape fitting method can be used to calculate the target bounding box. The process of this fitting method is as follows: 1) Traverse all possible directions of the rectangle, and find a rectangle pointing in that direction and containing all scan points in each iteration; 2) Obtain the distances from all points to the four sides of the rectangle, divide the points into p and q based on these distances, and calculate the corresponding squared errors as the objective function; 3) After iterating through all directions and obtaining all the corresponding squared errors, find the optimal direction with the minimum squared error, and adjust the rectangle according to this direction. When selecting a suitable target bounding box rectangle, minimizing the squared error from the point to the edge is used as the evaluation criterion. After obtaining the target bounding box, calculate the center of the target bounding box. When determining the length, width, and height of the target bounding box, the heading is first determined. The slopes of two sides of the target bounding box rectangle in the XY plane (parallel to the ground plane) are calculated to obtain the corresponding four orientation angles (each side has two opposite orientation angles). The orientation with the smallest angle difference from the target heading of the obstacle in the reference frame is selected as the target heading of the obstacle in the search frame. This selection is made because the time interval between adjacent frames is small, and the relative motion speed is low, resulting in minimal change in the target's heading. After determining the target heading, the side parallel to the target heading is designated as the longer side, and the side perpendicular to it is designated as the shorter side. The height is determined by the maximum height difference of the point cloud points. Finally, the 3D target bounding box is obtained.

[0054] Optionally, before "sorting the second point cloud frame", the method further includes: downsampling the second point cloud frame and removing the downsampled second point cloud frame that does not contain the target obstacle to obtain an updated second point cloud frame.

[0055] In this embodiment, the original second point cloud frame has a high sampling frequency. Extracting all sample point clouds from consecutive second point cloud frames is not very meaningful because the changes in the point cloud corresponding to the target obstacle are small. Therefore, the original second point cloud frame can be downsampled. For example, if the original second point cloud frame has a sampling frequency of 10Hz, the downsampling process can involve sampling the original second point cloud frame at equal intervals of 1 frame / s, i.e., downsampling to 1Hz. Subsequently, the downsampled second point cloud frames can be filtered to remove those that do not contain the target obstacle, thus obtaining the updated second point cloud frame. This embodiment of the application, by downsampling the original second point cloud frame and removing those that do not contain the target obstacle, ensures that the positions and poses of the target obstacle in the updated second point cloud frames vary, resulting in better sample point cloud extraction.

[0056] Optionally, after the step of "obtaining the second sample point cloud when the reference frame corresponds to multiple search frames", the method further includes: updating the search frame corresponding to the second sample point cloud to the reference frame, using the updated reference frame to determine the second sample point cloud corresponding to the next search frame, until the number of the second sample point cloud is consistent with the number of the multiple search frames.

[0057] In this embodiment, if each reference frame corresponds to multiple search frames, then each time a search frame finishes extracting the second sample point cloud, the next search frame can extract the second sample point cloud based on the second target information of the search frame that finished extraction. This process continues until the number of second sample point clouds matches the number of search frames, indicating that all search frames corresponding to each reference frame have finished extracting the second sample point cloud, at which point the process can end. This embodiment determines a portion of reference frames from the second point cloud frames. After annotating the reference frames using the first target information, subsequent search frames corresponding to those reference frames can automatically acquire the corresponding second target information and annotate them, greatly reducing the workload of manual annotation. Furthermore, by referencing the annotation results of adjacent annotated frames, the target heading of automatically annotated target obstacles can be determined more accurately.

[0058] Furthermore, as Figure 1 In terms of specific implementation, this application provides a point cloud data augmentation device for building a dataset for autonomous driving in mining areas, such as... Figure 4 As shown, the device includes:

[0059] The first determining module is used to determine the point cloud frame to be enhanced from multiple basic point cloud frames, wherein the basic point cloud frames are obtained based on a conventional mining environment;

[0060] The second determining module is used to determine at least one target point cloud from multiple sample point clouds, each sample point cloud corresponding to a target obstacle and having a region label, the target point cloud being obtained based on a preset environment;

[0061] The point cloud overlay module is used to overlay the at least one target point cloud onto the point cloud frame to be enhanced based on the region label of the target point cloud, so as to obtain the enhanced point cloud frame.

[0062] Optionally, the point cloud overlay module includes:

[0063] A location determination unit is used to determine the target placement position of any of the target point clouds in the point cloud frame to be enhanced based on the region label of any of the target point clouds;

[0064] The judgment unit is used to determine, based on the target placement position, whether any of the target point clouds has non-ground point cloud points in the ground placement area of ​​the point cloud frame to be enhanced;

[0065] The overlay unit is used to overlay any of the target point clouds onto the target placement position in the point cloud frame to be enhanced when there are no non-ground point cloud points in the ground placement area.

[0066] Optionally, the device further includes:

[0067] The marking module is used to determine the target attributes corresponding to each first point cloud point in the point cloud frame to be enhanced based on a preset ground fitting algorithm before determining whether any of the target point clouds has non-ground point cloud points in the ground placement area of ​​the point cloud frame to be enhanced, and to mark the first point cloud points one by one using the target attributes. The target attributes include ground point cloud points and non-ground point cloud points.

[0068] Optionally, the device further includes:

[0069] The quantity determination module is used to, when non-ground point cloud points exist in the ground placement area, if the number of non-ground point cloud points is less than or equal to a preset quantity threshold, superimpose any of the target point cloud points onto the target placement position in the point cloud frame to be enhanced; if the number of non-ground point cloud points is greater than the preset quantity threshold, rotate the target placement position based on a preset angle, and determine the number of non-ground point cloud points in the corresponding ground placement area again based on the rotated target placement position, until the number of non-ground point cloud points in the ground placement area is less than or equal to the preset quantity threshold, or the number of rotations exceeds a preset number threshold, and then the process ends.

[0070] Optionally, the device further includes:

[0071] The deletion module is used to, after superimposing any of the target point clouds onto the target placement position in the point cloud frame to be enhanced, determine the target occlusion region corresponding to any of the target point clouds based on the target placement position of any of the target point clouds, determine the target point cloud points corresponding to the target occlusion region from the first point cloud points, and delete the target point cloud points from the point cloud frame to be enhanced.

[0072] Optionally, the device further includes:

[0073] The first acquisition module is used to acquire the first point cloud frame collected by the autonomous driving equipment in a conventional mining environment before determining the point cloud frame to be enhanced from multiple basic point cloud frames, and to generate a tag file corresponding to the first point cloud frame, and to obtain the basic point cloud frame based on the first point cloud frame and the tag file.

[0074] The second acquisition module is used to acquire a second point cloud frame collected by the autonomous driving equipment in the mining area under a preset environment, and to determine the sample point cloud based on the second point cloud frame, wherein the preset environment includes the target obstacle.

[0075] Optionally, the second acquisition module is used for:

[0076] Based on the acquisition time of the second point cloud frames, the second point cloud frames are sorted, and reference frames and search frames corresponding to each reference frame are determined according to the sorted second point cloud frames; the target obstacles in the reference frames are identified, and the point clouds corresponding to the target obstacles are labeled using first target information to obtain a first sample point cloud, wherein the first target information includes the region label; based on the preset ground fitting algorithm, the target attributes corresponding to each second point cloud point in the search frame are determined, and based on the non-ground point cloud points in the second point cloud points and the first target information, the largest cluster closest to the target obstacle in the reference frame is determined using the Euclidean distance clustering algorithm, and the largest cluster is used as the point cloud corresponding to the target obstacle in the search frame; based on the point cloud corresponding to the target obstacle in the search frame, the target bounding box is determined, and based on the target bounding box and the first target information, the second target information corresponding to the target obstacle is determined, and the point cloud corresponding to the target obstacle in the search frame is labeled using the second target information to obtain a second sample point cloud.

[0077] Optionally, the device further includes:

[0078] The update module is used to update the search frame corresponding to the second sample point cloud to the reference frame after obtaining the second sample point cloud when the reference frame corresponds to multiple search frames, and use the updated reference frame to determine the second sample point cloud corresponding to the next search frame, until the number of the second sample point cloud is consistent with the number of the multiple search frames.

[0079] It should be noted that other corresponding descriptions of the functional units involved in the point cloud data augmentation device for building a mining area autonomous driving dataset provided in this application embodiment can be found in the following references. Figures 1 to 3 The corresponding descriptions in the method will not be repeated here.

[0080] Based on the above, Figures 1 to 3 Accordingly, this application also provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described method. Figures 1 to 3 The method shown is a point cloud data augmentation method for building an autonomous driving dataset in a mining area.

[0081] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0082] Based on the above, Figures 1 to 3 The method shown, and Figure 4 To achieve the above objectives, the present application also provides a computer device, specifically a personal computer, server, network device, etc., as shown in the virtual device embodiment. This computer device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figures 1 to 3 The method shown is a point cloud data augmentation method for building an autonomous driving dataset in a mining area.

[0083] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0084] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0085] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of a computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software within the physical device.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. First, a point cloud frame to be enhanced can be selected from multiple basic point cloud frames. Next, one or more target point clouds can be determined from multiple sample point clouds. Here, each sample point cloud can have a region label. After determining one or more target point clouds, the corresponding target point clouds can be superimposed on the above-mentioned point cloud frame to be enhanced based on the region label corresponding to each target point cloud, and finally the enhanced point cloud frame is obtained. The embodiments of this application, by superimposing at least one target point cloud corresponding to a target obstacle onto the point cloud frame to be enhanced, thereby obtaining the enhanced point cloud frame, can greatly improve the sample richness and has strong operability.

[0087] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0088] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A point cloud data enhancement method for a mine site automatic driving dataset construction, characterized in that, The method comprises the following steps: determining a to-be-enhanced point cloud frame from a plurality of basic point cloud frames, the basic point cloud frames being obtained based on a conventional mine environment; determining at least one target point cloud from a plurality of sample point clouds, each sample point cloud corresponding to a target obstacle and having a region label, the target point cloud being obtained based on a preset environment, the target obstacle including a passenger vehicle that can only enter a limited mine environment under limited road conditions, and an engineering vehicle or a non-production vehicle in a specific position and a specific posture; superimposing the at least one target point cloud into the to-be-enhanced point cloud frame based on the region label of the target point cloud, determining a target occlusion region corresponding to any target point cloud according to a target placement position of the target point cloud in the to-be-enhanced point cloud frame, determining a target point cloud point corresponding to the target occlusion region from each first point cloud point of the to-be-enhanced point cloud frame, and deleting the target point cloud point from the to-be-enhanced point cloud frame to obtain an enhanced point cloud frame; Before the step of determining the to-be-enhanced point cloud frame from the plurality of basic point cloud frames, the method further comprises: obtaining a second point cloud frame collected by a mine automatic driving device in a preset environment, the preset environment including the target obstacle; sorting the second point cloud frame based on a collection time of the second point cloud frame, and determining a reference frame and a search frame corresponding to each reference frame according to the sorted second point cloud frame; recognizing the target obstacle in the reference frame, and labeling the point cloud corresponding to the target obstacle by using first target information to obtain a first sample point cloud, the first target information including the region label; determining a target attribute corresponding to each second point cloud point in the search frame based on a preset ground fitting algorithm, and determining a maximum clustering cluster closest to the target obstacle in the reference frame by an Euclidean distance clustering algorithm according to a non-ground point cloud point in the second point cloud point and the first target information, taking the maximum clustering cluster as the point cloud corresponding to the target obstacle in the search frame; determining a target bounding box according to the point cloud corresponding to the target obstacle in the search frame, and determining second target information corresponding to the target obstacle based on the target bounding box and the first target information, labeling the point cloud corresponding to the target obstacle in the search frame by using the second target information to obtain a second sample point cloud; each first sample point cloud and each second sample point cloud constitute a final sample point cloud.

2. The method of claim 1, wherein, The step of superimposing the at least one target point cloud into the to-be-enhanced point cloud frame based on the region label of the target point cloud comprises the following steps: determining a target placement position of any target point cloud in the to-be-enhanced point cloud frame based on the region label of the target point cloud; judging whether there is a non-ground point cloud point in a ground placement region of the target point cloud in the to-be-enhanced point cloud frame based on the target placement position; when there is no non-ground point cloud point in the ground placement region, superimposing the target point cloud into the target placement position in the to-be-enhanced point cloud frame.

3. The method of claim 2, wherein, Before the judging whether the non-ground point cloud points exist in the ground placement region of any of the target point clouds in the to-be-enhanced point cloud frame, the method further comprises: Based on a preset ground fitting algorithm, determining a target attribute corresponding to each first point cloud point in the to-be-enhanced point cloud frame, and marking the first point cloud points one by one by using the target attribute, wherein the target attribute includes ground point cloud points and non-ground point cloud points.

4. The method according to claim 2 or 3, characterized in that, When the non-ground point cloud points exist in the ground placement region, the method further comprises: If the number of non-ground point cloud points is less than or equal to a preset number threshold, superimpose the any of the target point clouds to the target placement position in the to-be-enhanced point cloud frame; If the number of non-ground point cloud points is greater than the preset number threshold, rotate the target placement position based on a preset angle, and determine the number of non-ground point cloud points in the corresponding ground placement region again according to the rotated target placement position, until the number of non-ground point cloud points in the ground placement region is less than or equal to the preset number threshold, or the number of rotations exceeds a preset number threshold.

5. The method of claim 1, wherein, Before the determining the to-be-enhanced point cloud frame from the plurality of basic point cloud frames, the method further comprises: Obtaining a first point cloud frame collected by the mining area automatic driving equipment in a conventional mining area environment, and generating a label file corresponding to the first point cloud frame, obtaining the basic point cloud frame based on the first point cloud frame and the label file.

6. The method of claim 1, wherein, When the reference frame corresponds to a plurality of search frames, after the obtaining the second sample point cloud, the method further comprises: Updating the search frame corresponding to the second sample point cloud to a reference frame, and determining a second sample point cloud corresponding to a next search frame by using the updated reference frame, until the number of second sample point clouds is consistent with the number of search frames.

7. A point cloud data enhancement device for mine area automatic driving dataset construction, characterized in that, Comprise: A first determination module is configured to determine a to-be-enhanced point cloud frame from a plurality of basic point cloud frames, wherein the basic point cloud frames are obtained based on a conventional mining area environment; A second determination module is configured to determine at least one target point cloud from a plurality of sample point clouds, wherein each sample point cloud corresponds to a target obstacle and has a region label, the target point cloud is obtained based on a preset environment, and the target obstacle includes a passenger vehicle that can only enter a limited mining area environment under limited road conditions, and an engineering vehicle or a non-production vehicle in a specific position and a specific posture; A point cloud superimposition module is configured to superimpose the at least one target point cloud into the to-be-enhanced point cloud frame based on the region label of the target point cloud, determine a target occlusion region corresponding to any of the target point clouds in a target placement position of the target point cloud in the to-be-enhanced point cloud frame, determine a target point cloud point corresponding to the target occlusion region from each first point cloud point of the to-be-enhanced point cloud frame, and delete the target point cloud point from the to-be-enhanced point cloud frame to obtain an enhanced point cloud frame; The device further comprises a second acquisition module; the second acquisition module is configured to: Obtain a second point cloud frame collected by a mining area automatic driving equipment in a preset environment, wherein the preset environment includes the target obstacle; sequencing the second point cloud frames based on collection time of the second point cloud frames, and determining a reference frame and a search frame corresponding to each reference frame according to the sequenced second point cloud frames; identifying the target obstacle in the reference frame, and labeling point clouds corresponding to the target obstacle by using first target information, to obtain a first sample point cloud, wherein the first target information comprises the region label; determining a target attribute corresponding to each second point cloud point in the search frame based on a preset ground fitting algorithm, and determining a maximum clustering cluster closest to the target obstacle in the reference frame by an Euclidean distance clustering algorithm according to non-ground point cloud points in the second point cloud points and the first target information, and taking the maximum clustering cluster as point clouds corresponding to the target obstacle in the search frame; determining a target bounding box according to the point clouds corresponding to the target obstacle in the search frame, and determining second target information corresponding to the target obstacle based on the target bounding box and the first target information, and labeling the point clouds corresponding to the target obstacle in the search frame by using the second target information, to obtain a second sample point cloud; each first sample point cloud and each second sample point cloud constitute a final sample point cloud.

8. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements the method of any one of claims 1 to 6.

9. A computer device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor, when executing the computer program, implements the method of any one of claims 1 to 6.

Images