A data processing method and device
By adding preset point cloud data of labeled objects to the laser point cloud of unmanned driving equipment, as a training sample, the problem of high cost of manually labeling laser point clouds is solved, and the accuracy of target object recognition in laser point clouds is improved Sex and training samples richness.
Patent Information
- Application Number
- CN202011104583.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-10-15
AI Technical Summary
When training classification models for existing unmanned driving equipment, it is necessary to manually mark pedestrian and vehicle information in laser point clouds, resulting in higher labor costs and time expenditures.
By determining the point cloud data of the target object from the preset point cloud data corresponding to the marked object, and adding the point cloud data of the target object to the pending laser point cloud according to the relative position relationship between the acquisition device and the target object, the point cloud data of the target object is added to the pending laser point cloud according to the relative position relationship between the acquisition device and the target object. , as a training sample to reduce the need for manual annotation.
It improves the richness of training samples, reduces the cost and time of manual labeling, and improves the accuracy of target object recognition in laser point clouds.
Smart Images

Figure CN112329547B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device. Background Art
[0002] Usually, when driving, unmanned driving equipment needs to detect pedestrians, vehicles and other moving objects in the surrounding environment in real time to avoid obstacles. Specifically, the unmanned driving equipment can collect laser point cloud information in the surrounding environment in real time through the laser radar carried by the equipment, and input the collected laser point cloud into the pre-trained classification model to determine pedestrians and vehicles in the surrounding environment for obstacle avoidance.
[0003] When training the classification model, several frames of laser point clouds collected in the past are usually manually annotated in advance, that is, pedestrians and vehicles are framed in each frame of laser point cloud in the form of 3D annotation frames, and the annotated frames of laser point cloud are used as training samples to train the classification model. However, manually annotating the information of moving objects such as pedestrians and vehicles in each frame of laser point cloud requires a lot of labor costs. Summary of the invention
[0004] The embodiments of this specification provide a data processing method and device to partially solve the above-mentioned problems existing in the prior art.
[0005] The embodiments of this specification adopt the following technical solutions:
[0006] A data processing method provided in this specification includes:
[0007] Determine a frame of laser point cloud to be processed;
[0008] Determining point cloud data corresponding to a target object from point cloud data corresponding to preset annotated objects, wherein the annotated object is an environmental object annotated in a plurality of frames of laser point cloud, and the point cloud data corresponding to the annotated object is extracted from the annotated laser point cloud;
[0009] Determine the position of the point cloud data corresponding to the target object in the laser point cloud to be processed according to the relative posture relationship between the acquisition device and the target object when acquiring the point cloud data corresponding to the target object and the posture of the acquisition device when acquiring the laser point cloud to be processed;
[0010] According to the position of the target object in the laser point cloud to be processed, the point cloud data corresponding to the target object is added to the laser point cloud to be processed, and the added laser point cloud is used as a training sample for training the classification model, so as to identify the target object in the laser point cloud through the trained classification model.
[0011] Optionally, each labeled laser point cloud includes the laser point cloud to be processed, or each labeled laser point cloud does not include the laser point cloud to be processed.
[0012] Optionally, before using the added laser point cloud as a training sample for training the classification model, the method further includes:
[0013] Determine a ground point cloud dataset from the laser point cloud to be processed;
[0014] Determine the ground represented by the laser point cloud to be processed according to the determined ground point cloud data set;
[0015] Determine a projection area of the target object on the ground according to a position of the point cloud data corresponding to the target object in the laser point cloud to be processed;
[0016] When the projection area does not contain the laser points in the ground point cloud dataset, the point cloud data corresponding to the target object is rotated with the acquisition device for acquiring the laser point cloud to be processed as the center, and the position of the point cloud data corresponding to the target object in the laser point cloud to be processed is updated, and the projection area of the target object on the ground is re-determined until the projection area contains the laser points in the ground point cloud dataset.
[0017] Optionally, before using the added laser point cloud as a training sample for training the classification model, the method further includes:
[0018] Determine a ground point cloud dataset from the laser point cloud to be processed, and determine the ground represented by the laser point cloud to be processed according to the determined ground point cloud dataset;
[0019] Determining the distance between the target object and the ground according to the point cloud data corresponding to the target object and the ground point cloud data set;
[0020] According to the determined distance, the point cloud data corresponding to the target object is adjusted to the ground.
[0021] Optionally, the laser point cloud to be processed contains several environmental objects;
[0022] Before using the added laser point cloud as a training sample for training the classification model, the method further includes:
[0023] According to the point cloud data corresponding to the target object and the point cloud data corresponding to the environmental objects in the laser point cloud to be processed, determining whether the target object has an overlapping area with any environmental object in the laser point cloud to be processed;
[0024] When it is determined that the target object has an overlapping area with the environmental objects in the laser point cloud to be processed, the point cloud data corresponding to the target object is rotated with the acquisition device for acquiring the laser point cloud to be processed as the center until the target object has no overlapping area with any environmental object in the laser point cloud to be processed.
[0025] Optionally, before using the added laser point cloud as a training sample for training the classification model, the method further includes:
[0026] Determine, according to the pose of the acquisition device when acquiring the laser point cloud to be processed, the point cloud data of the target object and the point cloud data of the laser point cloud to be processed, the laser points on the target object blocked by the laser point cloud to be processed and the laser points in the laser point cloud to be processed blocked by the target object;
[0027] The laser points blocked on the target object and the laser points blocked by the target object are deleted from the added laser point cloud, and the added laser point cloud is updated.
[0028] Optionally, there are multiple target objects;
[0029] The method further comprises:
[0030] For each target object in turn, determining the position of the point cloud data corresponding to the target object in the laser point cloud to be processed;
[0031] According to the position of the point cloud data corresponding to each target object in the laser point cloud to be processed, the point cloud data corresponding to each target object is added to the laser point cloud to be processed.
[0032] This specification provides a data processing device, including:
[0033] A first determination module determines a frame of laser point cloud to be processed;
[0034] The second determination module determines the point cloud data corresponding to the target object from the point cloud data corresponding to the preset annotated objects, wherein the annotated objects are environmental objects annotated in a plurality of frames of laser point clouds, and the point cloud data corresponding to the annotated objects are extracted from the annotated laser point clouds;
[0035] A third determination module determines the position of the point cloud data corresponding to the target object in the laser point cloud to be processed according to the relative posture relationship between the acquisition device and the target object when acquiring the point cloud data corresponding to the target object and the posture of the acquisition device when acquiring the laser point cloud to be processed;
[0036] An adding module adds point cloud data corresponding to the target object to the laser point cloud to be processed according to the position of the target object in the laser point cloud to be processed, and uses the added laser point cloud as a training sample for training a classification model, so as to identify the target object in the laser point cloud through the trained classification model.
[0037] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above data processing method is implemented.
[0038] The present specification provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data processing method when executing the program.
[0039] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:
[0040] In this specification, a frame of laser point cloud to be processed can be determined first, and the point cloud data corresponding to the target object can be determined from the point cloud data corresponding to each preset annotated object. Afterwards, according to the relative posture relationship between the acquisition device and the target object when acquiring the point cloud data corresponding to the target object, and the posture of the acquisition device when acquiring the laser point cloud to be processed, the position of the point cloud data corresponding to the target object in the laser point cloud to be processed is determined. Finally, the point cloud data corresponding to the target object is added to the position corresponding to the laser point cloud to be processed, and the added laser point cloud is used as a training sample for training the classification model, so as to identify the target object in the laser point cloud through the trained classification model. By adding the point cloud data of the target object in the preset annotated object to the laser point cloud to be processed according to the relative posture relationship at the time of acquisition, the point cloud data of the target object is added to the combined laser point cloud, thereby improving the richness of the training samples. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0042] Figure 1 A flowchart of a data processing method provided in an embodiment of this specification;
[0043] Figure 2 A schematic diagram of a labeled laser point cloud provided in an embodiment of this specification;
[0044] Figure 3 A schematic diagram of determining a ground point cloud data set provided in an embodiment of this specification;
[0045] Figure 4 A schematic diagram of adding a target object provided in an embodiment of this specification;
[0046] Figure 5 A schematic diagram of adding a target object provided in an embodiment of this specification;
[0047] Figure 6 A schematic diagram of adding a target object provided in an embodiment of this specification;
[0048] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of this specification;
[0049] Figure 8 A schematic diagram of an electronic device for implementing a data processing method provided in an embodiment of this specification. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this application will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0051] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0052] Figure 1 A flowchart of a data processing method provided in an embodiment of this specification may include the following steps:
[0053] S100: Determine a frame of laser point cloud to be processed.
[0054] Usually, when unmanned driving equipment is driving, in addition to driving according to the navigation route in the electronic map, it is also necessary to identify the obstacle information of dynamic obstacles such as pedestrians and vehicles in the surrounding environment in real time to avoid obstacles. When unmanned driving equipment identifies the obstacle information of dynamic obstacles in the surrounding environment, it can collect the laser point cloud in the surrounding environment through its own laser radar, and send the collected laser point cloud data to the pre-trained classification model to identify the pedestrian and vehicle information in the laser point cloud data.
[0055] When training the above classification model, it is necessary to manually mark pedestrians and vehicles in each frame of laser point cloud in advance by means of 3D annotation box selection, and use the marked frames of laser point cloud as training samples to train the classification model, so as to identify pedestrians and vehicles in the laser point cloud through the classification model.
[0056] However, since manual labeling of pedestrians and vehicles in each frame of laser point cloud requires a lot of time and manpower costs, this manual proposes a data processing method, which adds pre-labeled pedestrians, vehicles and other labeled objects to the collected laser point cloud to combine them into new laser point cloud data, and uses the combined laser point cloud data as training samples to train the classification model.
[0057] Therefore, in one or more embodiments of the present specification, a frame of laser point cloud to be processed may be determined first, and the laser point cloud to be processed is the laser point cloud to which the labeled object is to be added later. In order to reduce the cost and time of manual labeling, the laser point cloud to be processed may be a laser point cloud that has been collected but does not contain the labeled object, or may be a laser point cloud that has been collected but has been labeled with the labeled object. The present specification does not limit this and may be set as needed.
[0058] It should be noted that the data processing method provided in this specification can be executed by a server used to combine laser point cloud data sets. The server can be a single server or a system composed of multiple servers, such as a distributed server, etc. This specification does not impose any restrictions on this and can be set as needed.
[0059] S102: Determine the point cloud data corresponding to the target object from the preset point cloud data corresponding to each labeled object.
[0060] In one or more embodiments of the present specification, after the laser point cloud to be processed is determined through the above step S100, the point cloud data of the target object to be added to the laser point cloud to be processed can be determined, so that the target object to be added can be added to the laser point cloud to be processed through subsequent steps to form new laser point cloud data.
[0061] Specifically, the server can determine the point cloud data corresponding to the target object from the preset point cloud data corresponding to each labeled object. Among them, the preset point cloud data corresponding to each labeled object is extracted from each frame of the labeled laser point cloud, and each frame of the laser point cloud is composed of the point cloud data corresponding to each collected environmental object. That is, each environmental object marked in each of the several frames of the labeled laser point cloud is used as each labeled object, and the labeled object includes at least one or more of the environmental objects such as pedestrians, vehicles and buildings. Each labeled laser point cloud may include the laser point cloud to be processed, or may not include the laser point cloud to be processed. When the labeled laser point cloud includes the laser point cloud to be processed, the labeled object can be determined from the laser point cloud to be processed as the target object, and the determined target object can be added to the laser point cloud to be processed.
[0062] Figure 2 The diagram of the labeled laser point cloud is shown in Figure 2. Labeling the laser point cloud dataset is actually to mark one or more environmental objects such as pedestrians, vehicles, and buildings in each frame of the laser point cloud in a 3D annotation box selection manner. Figure 2 As shown, Figure 2 In the figure, pedestrians and vehicles are used as examples to illustrate the labeled objects. The four-pointed stars represent the collected laser points, and the ellipse represents the range of a frame of collected laser point cloud. Pedestrians and vehicles are marked in the form of 3D annotation boxes. The pedestrian annotation box contains the laser points on the pedestrians, and the vehicle annotation box contains the laser points on the vehicles. Except for the laser points on the vehicles and pedestrians, the other laser points located inside the ellipse are laser points on the ground. Figure 2 It is just an exemplary representation. The actually collected laser point cloud only contains laser points, but does not contain actual environmental objects such as vehicles and pedestrians, and the number of laser point clouds is relatively dense.
[0063] Furthermore, when determining the point cloud data corresponding to the target object, it is also possible to set which objects need to be determined as target objects from the preset annotated objects according to the requirements of model training. For example, when the classification model is required to more accurately identify vehicles in the laser point cloud, the vehicle can be determined from the preset annotated objects as the target object. When the classification model is required to more accurately identify pedestrians in the laser point cloud, the pedestrian can be determined from the preset annotated objects as the target object.
[0064] Of course, for each laser point cloud to be processed, different target objects can be determined from the preset labeled objects as the target objects corresponding to the laser point cloud to be processed, so as to be added to the laser point cloud to be processed.
[0065] Furthermore, when determining the point cloud data of the target object from the point cloud data of the preset annotated object, this specification does not limit the number of target objects determined, and the specific number can be set as needed. When only one target object is added to the laser point cloud to be processed, a target object can be randomly determined from the preset annotated objects. When only multiple target objects are added to the laser point cloud to be processed, the preset annotated objects can be arbitrarily combined first, the target objects contained in each combination can be determined, and a combination can be randomly determined from each combination for addition. For the convenience of description, the following description will take the addition of a target object to the laser point cloud to be processed as an example.
[0066] S104: Determine the position of the point cloud data corresponding to the target object in the laser point cloud to be processed according to the relative posture relationship between the acquisition device and the target object when acquiring the point cloud data corresponding to the target object and the posture of the acquisition device when acquiring the laser point cloud to be processed.
[0067] In one or more embodiments of the present specification, after the laser point cloud to be processed and the point cloud data of the target object to be added are determined through the above steps S100 and S102, the target object to be added can be added to the laser point cloud to be processed.
[0068] Since when collecting laser point clouds, the position of the collection device is usually used as the origin to determine the coordinates of each laser point in the laser point cloud in the coordinate system with the collection device as the origin. Therefore, in this specification, when adding the point cloud data of the target object to the laser point cloud to be processed, specifically, the server can first determine the relative posture relationship between the collection device and the target object when collecting the point cloud data corresponding to the target object, and determine the posture of the collection device when collecting the laser point cloud to be processed. Afterwards, according to the determined relative posture relationship, determine the position of the point cloud data corresponding to the target object in the laser point cloud to be processed when adding the point cloud data of the target object to the coordinate system with the collection device when collecting the laser point cloud to be processed as the origin.
[0069] S106: According to the position of the target object in the laser point cloud to be processed, the point cloud data corresponding to the target object is added to the laser point cloud to be processed, and the added laser point cloud is used as a training sample for training a classification model, so as to identify the target object in the laser point cloud through the trained classification model.
[0070] In one or more embodiments of the present specification, when the position of the target object to be added in the laser point cloud to be processed is determined, the point cloud data corresponding to the target object can be added to the corresponding position in the laser point cloud to be processed to form new laser point cloud data.
[0071] Specifically, the server can add the point cloud data corresponding to the target object to the laser point cloud to be processed according to the determined position of the target object in the laser point cloud to be processed as the combined new laser point cloud data. The combined laser point cloud data is used as a training sample for training the classification model to perform model training, so that the unmanned driving device can identify the target object in the laser point cloud of the surrounding environment collected through the trained classification model.
[0072] Among them, because when collecting the point cloud data corresponding to the target object, the collection device can only collect the laser points of the target object facing the side of the collection device, and the laser points on the back side are not collected, therefore, when adding the point cloud data corresponding to the target object to the laser point cloud to be processed, it is also necessary to make the point cloud data corresponding to the target object located on the side of the collection device facing the laser point cloud to be processed according to the corresponding posture relationship, so as to determine a laser point cloud that is more in line with the actual situation as a training sample.
[0073] based on Figure 1 The data processing method shown can first determine a frame of laser point cloud to be processed, and determine the point cloud data corresponding to the target object from the point cloud data corresponding to each preset annotated object. Afterwards, according to the relative posture relationship between the acquisition device and the target object when acquiring the point cloud data corresponding to the target object, and the posture of the acquisition device when acquiring the laser point cloud to be processed, the position of the point cloud data corresponding to the target object in the laser point cloud to be processed is determined. Finally, the point cloud data corresponding to the target object is added to the position corresponding to the laser point cloud to be processed, and the added laser point cloud is used as a training sample for training the classification model, so as to identify the target object in the laser point cloud through the trained classification model. By adding the point cloud data of the target object in the preset annotated object to the laser point cloud to be processed according to the relative posture relationship at the time of acquisition, the point cloud data of the target object is added to the combined laser point cloud, thereby improving the richness of the training samples.
[0074] In step S106 of the present specification, when the laser point cloud to be processed is a labeled laser point cloud, when adding the laser point cloud data of the target object to the laser point cloud to be processed, considering the position of the object in the actual environment, it is necessary to add the point cloud data of the target object to the empty space on the ground in the laser point cloud to be processed, that is, on the ground that does not conflict with the position of the labeled object.
[0075] Therefore, in this specification, before using the added laser point cloud as a training sample for training the classification model, the server may also determine whether the target object is added to the open space on the ground in the laser point cloud to be processed. The open space on the ground includes the laser points on the ground in the laser point cloud to be processed, and the non-open space area on the ground (the ground area where the environmental objects are located, such as the area occupied by the building) does not include the laser points on the ground.
[0076] Specifically, the server may first determine the ground point cloud dataset from the laser point cloud to be processed, and determine the ground represented by the laser point cloud to be processed based on the determined ground point cloud dataset, wherein the ground is the plane where the laser points on the ground are located, including open space and non-open space areas. Afterwards, the projection area of the target object on the ground is determined based on the position of the point cloud data corresponding to the target object in the laser point cloud to be processed. When the projection area does not contain the laser points in the ground point cloud dataset, it indicates that the target object is not located on an open ground. Therefore, the acquisition device for collecting the laser point cloud to be processed can be used as the center, and the point cloud data corresponding to the target object can be rotated according to the preset angle step, and the position of the point cloud data corresponding to the target object in the laser point cloud to be processed can be updated to re-determine the projection area of the target object on the ground until the projection area contains the laser points in the ground point cloud dataset. The preset angle step can be set as needed, and this specification does not limit this.
[0077] Similarly, when collecting point cloud data corresponding to the target object, the acquisition device can only collect the laser points on the side of the target object facing the acquisition device, while the laser points on the opposite side are not collected. Therefore, when rotating the point cloud data corresponding to the target object, it is necessary to keep the side of the target object containing the laser points facing the acquisition device, and rotate according to the preset angle step with the acquisition device as the center.
[0078] When determining the ground point cloud data set from the laser point cloud to be processed, a ground segmentation method can be used to determine the point cloud data of the laser points on the ground as the ground point cloud data, and other point cloud data as non-ground point cloud data. Figure 2 Take the laser point cloud shown in the figure as an example. Figure 3 The four-pointed star in the figure represents the laser point. The laser points in the gray area constitute the ground point cloud dataset of the laser point cloud. Figure 3 The medium gray area represents the vacant area, and the white area corresponding to each quadrilateral represents the non-vacant area.
[0079] Figure 4 This is a schematic diagram of rotating the point cloud data corresponding to the target object and adding it to the laser point cloud to be processed. Figure 4The middle ellipse represents the range of the collected laser point cloud, the triangle in the center of the ellipse represents the collection device, and the cylinder represents the height of the collection device when collecting the laser point cloud. When the collection device collects the laser point cloud to be processed, the height from the ground is 2m. Figure 4 In the above figure, the point cloud data corresponding to the target object (pedestrian) is added to the position A in the laser point cloud to be processed, and the horizontal distance between the pedestrian and the acquisition device is a. The projection area of the pedestrian on the ground is the area S1 at the bottom of the pedestrian frame. The area S2 on the ground in the figure is the ground area where the building in the laser point cloud to be processed is located. For the convenience of explanation, the building is Figure 4 It is not drawn in the figure, but the S2 area is used as an example to represent the ground area where the building is located, that is, the non-open area. Since the projection area corresponding to the pedestrian on the ground does not contain the laser point on the ground, it is considered that the pedestrian is not in the open area in the laser point cloud to be processed, so the point cloud data corresponding to the pedestrian can be rotated according to the preset rotation direction and preset angle step, such as Figure 4 As shown in the figure below, the point cloud data corresponding to the pedestrian is rotated from position A in the laser point cloud to be processed to position B. At this time, the projection area corresponding to the pedestrian on the ground is area S3, which contains laser points on the ground. It can be considered that the pedestrian has been added to the open space area on the ground of the laser point cloud.
[0080] Furthermore, when performing ground segmentation, a random sample consensus algorithm (RANSAC) can be used to first determine a plane composed of any three laser points from the laser point cloud to be processed. After that, the distances between other laser points in the laser point cloud to be processed and the plane are determined respectively, and the laser points whose distances are less than the first preset threshold are determined as the laser points in the plane. Finally, the plane containing the largest number of laser points is taken as the ground, and the laser points in the plane are taken as the ground point cloud dataset, and the other laser points in the frame laser point cloud except the ground point cloud dataset are taken as the non-ground point cloud dataset. Among them, the first preset threshold can be set as needed, and this specification does not limit this.
[0081] Furthermore, there are many ways to perform ground segmentation on the frame of laser point cloud, such as normal vector segmentation, horizontal plane calibration, and grid height difference segmentation method, etc. This manual does not limit this, as long as the ground point cloud dataset can be segmented from the frame of laser point cloud, the specific settings can be based on needs.
[0082] In this specification, since the target object is usually selected from the labeled laser point cloud in the form of a 3D annotation box, in order to reduce manual annotation, when adding the laser point cloud to the laser point cloud to be processed, the annotation box of the target object is usually added to the laser point cloud to be processed, and the annotation box of the target object contains the point cloud data corresponding to the target object. When projecting to the ground and determining the projection area of the target object, the projection area can be determined by projecting to the ground according to the size of the annotation box of the target object.
[0083] In step S106 of this specification, since the point cloud data of each laser point in the laser point cloud is determined with the acquisition device as the origin, when the acquisition device for acquiring the point cloud data of the target object and the acquisition device for acquiring the laser point cloud to be processed are located at different heights, the position coordinates of each laser point determined are also different. When the target object is added to the laser point cloud to be processed, it may be placed in the air.
[0084] like Figure 5 As shown, assuming that the target object to be added is a pedestrian, Figure 5 The figure above shows the laser point cloud where the target object is located. Figure 5 The figure below shows the laser point cloud to be processed. The ellipse in the figure shows the range of the collected laser point cloud. The triangle in the center of the ellipse shows the collection device. The cylinder shows the height of the collection device when collecting the laser point cloud. When collecting the laser point cloud where the target object is located, the height of the collection device from the ground is 2m. When collecting the laser point cloud to be processed, the height from the ground is 3m. The four-pointed star in the figure shows the collected laser point. According to the relative posture relationship between the collection device and the target object when collecting the point cloud data corresponding to the target object and the posture of the collection device when collecting the laser point cloud to be processed, the target object is removed from Figure 5 When the upper middle image moves to the lower image, it can be determined that the target object is located 1m above the ground in the processed laser point cloud.
[0085] Therefore, before using the added laser point cloud as a training sample for training the classification model, the server can first determine the ground point cloud data set from the laser point cloud to be processed, and determine the ground represented by the laser point cloud to be processed based on the determined ground point cloud data set, wherein the method for determining the ground point cloud data set has been described in detail above, and this specification will not repeat it here. Afterwards, the height of the target object from the ground is determined based on the point cloud data corresponding to the target object and the point cloud data of each ground laser point contained in the ground point cloud data set. Finally, the point cloud data corresponding to the target object is adjusted to the ground based on the determined distance.
[0086] Among them, when determining the height of the target object from the ground based on the point cloud data corresponding to the target object and the point cloud data of each ground laser point contained in the ground point cloud data set, the coordinates of the laser point located at the bottom in the point cloud data corresponding to the target object can be determined first, and then the height of the target object from the ground can be determined based on the coordinates of the laser point and the coordinates of any laser point among the ground laser points. Alternatively, the height of the target object from the ground can be determined based on the average value of the coordinates of the laser point and the coordinates of each ground laser point. This specification does not limit the method for determining the height of the target object from the ground, which can be set as needed.
[0087] Furthermore, since the collected laser point cloud is relatively sparse, the collected laser points on the target object are not necessarily the laser points where the target object intersects the ground. Therefore, when the height of the target object from the ground is less than the second preset threshold, the target object can be considered to be on the ground and no movement adjustment is required. The second preset threshold can be set as needed, and this specification does not limit this.
[0088] In step S106 of the present specification, when the laser point cloud to be processed is a labeled laser point cloud, that is, the laser point cloud to be processed contains point cloud data corresponding to the labeled object, when adding the laser point cloud data of the target object to the laser point cloud to be processed, considering the actual environment corresponding to the laser point cloud to be processed, the target object cannot be added to the non-empty area corresponding to the labeled object in the laser point cloud to be processed.
[0089] Therefore, in this specification, before using the added laser point cloud as a training sample for training the classification model, the server can determine whether the target object has an overlapping area with any environmental object in the laser point cloud to be processed based on the point cloud data corresponding to the target object and the point cloud data corresponding to the environmental object in the laser point cloud to be processed. When the target object has an overlapping area with the environmental object in the laser point cloud to be processed, the position of the point cloud data corresponding to the target object in the laser point cloud to be processed can be adjusted, and the point cloud data corresponding to the target object can be rotated according to the preset angle step size with the acquisition device for collecting the laser point cloud to be processed as the center until the target object has no overlapping area with any environmental object in the laser point cloud to be processed. Among them, the environmental object can be the same as the annotation object or different, and the preset angle step size can be set according to needs, which is not limited in this specification.
[0090] Of course, in one embodiment of the present specification, the server may first determine whether the projection area contains laser points on the ground. When it is determined that the projection area contains laser points on the ground, the server determines the height of the target object from the ground based on the point cloud data corresponding to the target object and the ground point cloud data set, and adjusts the target object to the ground. Afterwards, since the projection area may contain open areas and non-open areas, it is further possible to determine whether the target object has an overlapping area with any environmental object in the laser point cloud to be processed. When the target object has no overlapping area with any environmental object, the added laser point cloud can be used as a training sample for training the classification model.
[0091] It should be noted that the above method for determining training samples does not require any restrictions on the acquisition device for collecting point cloud data corresponding to the target object, and the position and posture of the acquisition device for collecting the laser point cloud to be processed, and is therefore universal.
[0092] In another embodiment of the present specification, the server may also first determine whether the projection area contains a laser point on the ground. When it is determined that the projection area contains a laser point on the ground, the server determines the height of the target object from the ground based on the point cloud data corresponding to the target object and the ground point cloud data set, and adjusts the target object to the ground.
[0093] like Figure 6 As shown, Figure 6 The triangle in the center of the middle ellipse represents the acquisition device, which is used to collect laser point clouds, and the cylinder represents the height of the acquisition device from the ground. The elliptical area in the figure represents the range of the laser point cloud to be processed, the gray area represents the ground, the four-pointed star represents the laser point, and each four-pointed star on the gray area represents each laser point on the ground. Assuming that the pedestrian at the right rear is the target object, when the pedestrian is moved to the laser point cloud of this frame, the pedestrian is located above the ground of the laser point cloud of this frame. The projection area of the pedestrian on the laser point cloud of this frame can be determined as shown in the dotted white area below the pedestrian in the figure. Since the projection area contains the laser points on the ground, the height of the target object from the ground can be determined based on the point cloud data corresponding to the target object and the ground point cloud data set, and the pedestrian is adjusted to the projection area on the ground of the laser point cloud to be processed.
[0094] In another embodiment of the present specification, the server may also first determine the height of the target object from the ground based on the point cloud data corresponding to the target object and the ground point cloud data set, and adjust the target object to the ground. Afterwards, it may be further determined whether the target object has an overlapping area with any environmental object in the laser point cloud to be processed. When the target object has no overlapping area with any environmental object, the added laser point cloud may be used as a training sample for training the classification model.
[0095] In other embodiments of this specification, one or more of the above-mentioned determining whether the projection area contains laser points on the ground, determining the height of the target object from the ground and adjusting it to the ground, and determining whether the target object has an overlapping area with any environmental object in the laser point cloud to be processed can be arbitrarily combined to determine various implementation methods. This specification does not limit the combination method, which can be set as needed.
[0096] In step S106 of this specification, when the target object is directly added to the laser point cloud to be processed, the target object may block other environmental objects in the laser point cloud frame, or be blocked by other environmental objects. Therefore, before using the added laser point cloud as a training sample for training the classification model, it is necessary to delete the laser points on the target object blocked by other environmental objects and delete the laser points in the laser point cloud to be processed that are blocked by the target object.
[0097] Specifically, the server can determine the laser points on the target object that are blocked by the laser point cloud to be processed, and the laser points in the laser point cloud to be processed that are blocked by the target object based on the posture of the acquisition device when acquiring the laser point cloud to be processed, the point cloud data of the target object, and the point cloud data of the laser point cloud to be processed, and then delete the determined blocked laser points from the point cloud data of the target object, and update the added laser point cloud.
[0098] Among them, since the collected laser points on the target object are usually a number of discrete points on the target object, and in the actual environment the target object is usually a solid object, when determining the laser points blocked by the target object in the laser point cloud to be processed, the contour area of the target object can be determined first based on the point cloud data of the target object, and then, based on the posture of the acquisition device when collecting the laser point cloud to be processed, the contour area of the target object and the point cloud data of the laser point cloud to be processed, the laser points blocked by the contour area of the target object in the laser point cloud to be processed are determined.
[0099] Similarly, when determining the laser points on the target object that are blocked by the laser point cloud to be processed, the foreground object that blocks the target object in the laser point cloud to be processed may be determined first, and the contour area of the foreground object may be determined based on the point cloud data of the foreground object. Afterwards, the laser points on the target object that are blocked by the laser point cloud to be processed may be determined based on the position and posture of the acquisition device when acquiring the laser point cloud to be processed, the point cloud data of the target object, and the contour area of the foreground object in the laser point cloud to be processed.
[0100] In addition, in the present specification, when the number of added target objects is multiple, the server can determine the position of each target object in the laser point cloud to be processed in turn according to the method described in the above step S104, and add each target object to the laser point cloud to be processed according to the determined position of each target object in the laser point cloud to be processed.
[0101] In this specification, when adding multiple target objects to the laser point cloud to be processed, it is also necessary to determine that there is no overlapping area between the target objects based on the determined positions of the target objects in the laser point cloud to be processed. When deleting the blocked laser points in the added laser point cloud, it is also necessary to determine whether the target objects are blocked, and delete the blocked laser points between the determined target objects.
[0102] It should be noted that when the acquisition device is collecting laser point clouds, different environmental objects are at different distances from the acquisition device, resulting in differences in the density of the collected laser points. Therefore, in this specification, the laser point cloud corresponding to the target object is moved to the laser point cloud to be processed according to the relative posture relationship between the acquisition device and the target object when collecting the point cloud data corresponding to the target object, that is, the point cloud data corresponding to the target object is moved to a position in the laser point cloud to be processed that is at the same distance from the acquisition device. This can avoid moving the target object to any position in the laser point cloud to be processed, resulting in the laser point density distribution not conforming to the actual situation, so that when the laser point cloud moved to any position is used as a training sample for model training, the training effect of the model is affected.
[0103] In addition, the laser point cloud data obtained based on the data processing method provided in this specification can be used as a training sample for training a classification model. The classification model can be applied to the delivery process of unmanned driving equipment. When the unmanned driving equipment is used in the delivery field, the server can obtain a number of laser point cloud data as training samples in advance through the data processing method for training the classification model. During the driving process, the unmanned driving equipment can collect the laser point cloud contained in the surrounding environment and input it into the classification model to determine the target objects in the surrounding environment, such as pedestrians and vehicles, so that the unmanned driving equipment can avoid obstacles. It can be used specifically in scenarios such as using unmanned driving equipment for express delivery, take-out delivery, etc.
[0104] Furthermore, in this specification, unmanned driving equipment may include: unmanned vehicles, robots with mobile functions, intelligent vehicles or vehicles with unmanned driving functions, etc., and this specification does not impose any restrictions on this.
[0105] based on Figure 1The data processing method shown in the figure, the embodiment of this specification also provides a structural schematic diagram of a data processing device, such as Figure 7 shown.
[0106] Figure 7 A schematic diagram of a data processing device provided in an embodiment of this specification, the device comprising:
[0107] A first determination module 200 determines a frame of laser point cloud to be processed;
[0108] The second determination module 202 determines the point cloud data corresponding to the target object from the point cloud data corresponding to the preset annotated objects, wherein the annotated objects are environmental objects annotated in a plurality of frames of laser point clouds, and the point cloud data corresponding to the annotated objects are extracted from the annotated laser point clouds;
[0109] A third determination module 204 determines the position of the point cloud data corresponding to the target object in the laser point cloud to be processed according to the relative posture relationship between the acquisition device and the target object when acquiring the point cloud data corresponding to the target object and the posture of the acquisition device when acquiring the laser point cloud to be processed;
[0110] The adding module 206 adds the point cloud data corresponding to the target object to the laser point cloud to be processed according to the position of the target object in the laser point cloud to be processed, and uses the added laser point cloud as a training sample for training the classification model, so as to identify the target object in the laser point cloud through the trained classification model.
[0111] Optionally, each labeled laser point cloud includes the laser point cloud to be processed, or each labeled laser point cloud does not include the laser point cloud to be processed.
[0112] Optionally, the adding module 206 is further used to determine a ground point cloud dataset from the laser point cloud to be processed, determine the ground represented by the laser point cloud to be processed based on the determined ground point cloud dataset, determine the projection area of the target object on the ground based on the position of the point cloud data corresponding to the target object in the laser point cloud to be processed, and when the projection area does not contain the laser points in the ground point cloud dataset, rotate the point cloud data corresponding to the target object with the acquisition device for acquiring the laser point cloud to be processed as the center, update the position of the point cloud data corresponding to the target object in the laser point cloud to be processed, and redetermine the projection area of the target object on the ground until the projection area contains the laser points in the ground point cloud dataset.
[0113] Optionally, the adding module 206 is also used to determine a ground point cloud dataset from the laser point cloud to be processed, and determine the ground represented by the laser point cloud to be processed based on the determined ground point cloud dataset, determine the distance of the target object from the ground based on the point cloud data corresponding to the target object and the ground point cloud dataset, and adjust the point cloud data corresponding to the target object to the ground based on the determined distance.
[0114] Optionally, the laser point cloud to be processed includes several environmental objects, and the adding module 206 is further used to determine whether the target object has an overlapping area with any environmental object in the laser point cloud to be processed according to the point cloud data corresponding to the target object and the point cloud data corresponding to the environmental objects in the laser point cloud to be processed; when it is determined that the target object has an overlapping area with the environmental objects in the laser point cloud to be processed, the point cloud data corresponding to the target object is rotated with the acquisition device for acquiring the laser point cloud to be processed as the center until the target object has no overlapping area with any environmental object in the laser point cloud to be processed.
[0115] Optionally, the adding module 206 is further used to determine the laser points on the target object that are blocked by the laser point cloud to be processed and the laser points in the laser point cloud to be processed that are blocked by the target object according to the posture of the acquisition device when acquiring the laser point cloud to be processed, the point cloud data of the target object and the point cloud data of the laser point cloud to be processed, delete the blocked laser points on the target object and the laser points blocked by the target object from the added laser point cloud, and update the added laser point cloud.
[0116] Optionally, there are multiple target objects, and the third determination module 204 and the adding module 206 are specifically used to determine, for each target object in turn, the position of the point cloud data corresponding to the target object in the laser point cloud to be processed, and add the point cloud data corresponding to each target object to the laser point cloud to be processed according to the position of the point cloud data corresponding to each target object in the laser point cloud to be processed.
[0117] The present specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provided data processing methods.
[0118] based on Figure 1 The data processing method shown in the specification also proposes Figure 8 The schematic structure diagram of the electronic device shown in FIG. Figure 8At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The data processing method shown.
[0119] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0120] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0121] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0122] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0123] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0124] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0129] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0132] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0133] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0134] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0135] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.
Claims
1. A data processing method, characterized in that: include: Determine a frame of laser point cloud to be processed; Determining point cloud data corresponding to a target object from point cloud data corresponding to preset annotated objects, wherein the annotated object is an environmental object annotated in a plurality of frames of laser point cloud, and the point cloud data corresponding to the annotated object is extracted from the annotated laser point cloud; Determine the position of the point cloud data corresponding to the target object in the laser point cloud to be processed according to the relative posture relationship between the acquisition device and the target object when acquiring the point cloud data corresponding to the target object and the posture of the acquisition device when acquiring the laser point cloud to be processed; According to the position of the target object in the laser point cloud to be processed, the point cloud data corresponding to the target object is added to the laser point cloud to be processed, and the added laser point cloud is used as a training sample for training a classification model, so as to identify the target object in the laser point cloud through the trained classification model; Before using the added laser point cloud as a training sample for training the classification model, the method further includes: Determine a ground point cloud dataset from the laser point cloud to be processed, and determine the ground represented by the laser point cloud to be processed according to the determined ground point cloud dataset; Determining the distance between the target object and the ground according to the point cloud data corresponding to the target object and the ground point cloud data set; According to the determined distance, the point cloud data corresponding to the target object is adjusted to the ground; before the added laser point cloud is used as a training sample for training the classification model, the method further includes: Determine a projection area of the target object on the ground according to a position of the point cloud data corresponding to the target object in the laser point cloud to be processed; When the projection area does not contain the laser points in the ground point cloud dataset, the point cloud data corresponding to the target object is rotated with the acquisition device for acquiring the laser point cloud to be processed as the center, and the position of the point cloud data corresponding to the target object in the laser point cloud to be processed is updated, and the projection area of the target object on the ground is re-determined until the projection area contains the laser points in the ground point cloud dataset.
2. The method according to claim 1, characterized in that Each marked laser point cloud includes the laser point cloud to be processed, or each marked laser point cloud does not include the laser point cloud to be processed.
3. The method according to claim 1, characterized in that The laser point cloud to be processed contains a number of environmental objects; Before using the added laser point cloud as a training sample for training the classification model, the method further includes: According to the point cloud data corresponding to the target object and the point cloud data corresponding to the environmental objects in the laser point cloud to be processed, determining whether the target object has an overlapping area with any environmental object in the laser point cloud to be processed; When it is determined that the target object has an overlapping area with the environmental objects in the laser point cloud to be processed, the point cloud data corresponding to the target object is rotated with the acquisition device for acquiring the laser point cloud to be processed as the center until the target object has no overlapping area with any environmental object in the laser point cloud to be processed.
4. The method according to any one of claims 1 to 3, characterized in that: Before using the added laser point cloud as a training sample for training the classification model, the method further includes: Determine, according to the pose of the acquisition device when acquiring the laser point cloud to be processed, the point cloud data of the target object and the point cloud data of the laser point cloud to be processed, the laser points on the target object blocked by the laser point cloud to be processed and the laser points in the laser point cloud to be processed blocked by the target object; The laser points blocked on the target object and the laser points blocked by the target object are deleted from the added laser point cloud, and the added laser point cloud is updated.
5. The method according to claim 1, characterized in that There are multiple target objects; The method further comprises: For each target object in turn, determining the position of the point cloud data corresponding to the target object in the laser point cloud to be processed; According to the position of the point cloud data corresponding to each target object in the laser point cloud to be processed, the point cloud data corresponding to each target object is added to the laser point cloud to be processed.
6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 5 is implemented.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 5 is implemented.
