Point cloud data augmentation method and device

Abstract

The invention provides a point cloud data augmentation method and device, and the method comprises the steps: obtaining the point cloud data of a target vehicle from a point cloud data set, determining an original vehicle hanging frame of the target vehicle according to the point cloud data, and enabling the original vehicle hanging frame to be a marking frame of the target vehicle, the target vehicle is a vehicle needing point cloud data augmentation processing in the point cloud data set; splitting an original vehicle hanging frame of the target vehicle to obtain a vehicle hanging sub-frame of the target vehicle; according to the point cloud data distribution characteristics of the original vehicle hanging frame, the vehicle hanging sub-frames participating in the application data augmentation scheme are determined in all the vehicle hanging sub-frames, that is, on the basis that the original vehicle hanging frame of the target vehicle is split into the multiple vehicle hanging sub-frames, the point cloud data distribution characteristics are combined, and the application data augmentation scheme is obtained. And the vehicle-mounted sub-frame participating in the application data augmentation scheme is selected, so that the problem that point cloud data augmentation is a scene or a single operation is performed on some instance frames in the scene, and the expectation of simulating more scenes cannot be achieved is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method and device for augmenting point cloud data. Background Art

[0002] Data augmentation technology is a method widely used in deep learning training based on point cloud datasets collected by lidar. Its purpose is to simulate and construct a richer scenario with limited labeled data to improve the generalization of the deep learning model.

[0003] The existing data augmentation technologies are roughly as follows: one is to scale, translate, and rotate the point cloud scenario (including the instance boxes therein) of some frames in the point cloud dataset; the other is to scale, translate, and rotate the instance boxes and the points contained in the boxes in some frames of the point cloud dataset, or to take out the above two and put them into the point cloud scenarios of other frames, or to take out the above two and perform copy and paste operations in the current scenario. However, the above methods all perform single operations on the scenario or some instance boxes in the scenario, and cannot achieve the expected effect of simulating more scenarios. Summary of the Invention

[0004] In view of this, the present application provides a method and device for augmenting point cloud data to solve the problem that the existing point cloud data augmentation only performs single operations on the scenario or some instance boxes in the scenario and cannot achieve the expected effect of simulating more scenarios.

[0005] To achieve the above object, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of the present application discloses a method for augmenting point cloud data, including:

[0007] After obtaining the point cloud data of the target vehicle from the point cloud dataset, determining the original vehicle hanging box of the target vehicle according to the point cloud data, where the original vehicle hanging box is the labeled box of the target vehicle, and the target vehicle is the vehicle that needs to perform point cloud data augmentation processing in the point cloud dataset;

[0008] Splitting the original vehicle hanging box of the target vehicle to obtain the vehicle hanging sub-boxes of the target vehicle;

[0009] Determining the vehicle hanging sub-boxes participating in the application of the data augmentation scheme from all the vehicle hanging sub-boxes according to the point cloud data distribution characteristics of the original vehicle hanging box.

[0010] Optionally, in the above method for augmenting point cloud data, after determining the vehicle hanging sub-boxes participating in the application of the data augmentation scheme from all the vehicle hanging sub-boxes according to the point cloud data distribution characteristics of the original vehicle hanging box, it further includes:

[0011] Perform a first process on the point cloud data in the hanging sub-frame of the vehicle that participates in the application data augmentation scheme and a second process on the point cloud data in the original hanging frame of the vehicle;

[0012] Among them, the first process is to randomly remove a certain proportion of the point cloud, and the second process is to mutually add noise points.

[0013] Optionally, in the above point cloud data augmentation method, splitting the original hanging frame of the vehicle to obtain the hanging sub-frame of the target vehicle includes:

[0014] Determine the target splitting scheme of the target vehicle, where the target splitting scheme includes: a first splitting scheme and a second splitting scheme;

[0015] Split the original hanging frame of the vehicle according to the target splitting scheme to obtain the hanging sub-frame of the target vehicle.

[0016] Optionally, in the above point cloud data augmentation method, the target splitting scheme is the first splitting scheme. Determining the hanging sub-frame of the vehicle that participates in the application data augmentation scheme among all the hanging sub-frames according to the point cloud data distribution characteristics of the original hanging frame of the vehicle includes:

[0017] According to the point cloud data distribution characteristics of the original hanging frame of the vehicle, determine the carrying state of the target vehicle, where the carrying state includes: an empty load state and a full load state;

[0018] According to the carrying state of the target vehicle, determine the hanging sub-frame of the vehicle that participates in the application data augmentation scheme among all the hanging sub-frames.

[0019] Optionally, in the above point cloud data augmentation method, determining the carrying state of the target vehicle according to the point cloud data distribution characteristics of the original hanging frame of the vehicle includes:

[0020] Respectively judge whether the point cloud data distribution characteristics of the original hanging frame of the vehicle conform to the preset empty load state distribution characteristics or the preset full load state distribution characteristics;

[0021] If it is judged that the point cloud data distribution characteristics of the original hanging frame of the vehicle conform to the preset empty load state distribution characteristics, then determine that the carrying state of the target vehicle is the empty load state;

[0022] If it is judged that the point cloud data distribution characteristics of the original hanging frame of the vehicle conform to the preset full load state distribution characteristics, then determine that the carrying state of the target vehicle is the full load state.

[0023] Optionally, in the above point cloud data augmentation method, judging whether the point cloud data distribution characteristics of the original hanging frame of the vehicle conform to the preset empty load state distribution characteristics includes:

[0024] Determine the number of point clouds for each of the vehicle hanging sub - frames of the target vehicle;

[0025] Judge whether the first point cloud ratio is greater than the preset no - load ratio. The first point cloud ratio is the ratio of the sum of the number of point cloud data in each vehicle hanging sub - frame in the upper half of the original vehicle hanging frame to the total number of point cloud data in the original vehicle hanging frame;

[0026] If it is judged that the first point cloud ratio is greater than the preset no - load ratio, it is determined that the distribution characteristic of the point cloud data of the original vehicle hanging frame does not conform to the preset no - load state distribution characteristic;

[0027] If it is judged that the first point cloud ratio is not greater than the preset no - load ratio, it is determined that the distribution characteristic of the point cloud data of the original vehicle hanging frame conforms to the preset no - load state distribution characteristic.

[0028] Optionally, in the above - mentioned point cloud data augmentation method, judging whether the distribution characteristic of the point cloud data of the original vehicle hanging frame conforms to the preset no - load state distribution characteristic includes:

[0029] Determine the number of point clouds for each of the vehicle hanging sub - frames of the target vehicle;

[0030] Judge whether the second point cloud ratio is greater than the preset full - load ratio. The second point cloud ratio is the ratio of the sum of the number of point cloud data in each vehicle hanging sub - frame in the lower half of the original vehicle hanging frame to the total number of point cloud data in the original vehicle hanging frame;

[0031] If it is judged that the second point cloud ratio is greater than the preset full - load ratio, it is determined that the distribution characteristic of the point cloud data of the original vehicle hanging frame does not conform to the preset no - load state distribution characteristic;

[0032] If it is judged that the second point cloud ratio is not greater than the preset full - load ratio, it is determined that the distribution characteristic of the point cloud data of the original vehicle hanging frame conforms to the preset no - load state distribution characteristic.

[0033] Optionally, in the above - mentioned point cloud data augmentation method, according to the carrying state of the target vehicle, determining the vehicle hanging sub - frames participating in the application of the data augmentation scheme among all the vehicle hanging sub - frames includes:

[0034] If the carrying state of the target vehicle is the no - load state, determine the vehicle hanging sub - frames corresponding to the no - load state in the original vehicle hanging frame as the vehicle hanging sub - frames participating in the application of the data augmentation scheme;

[0035] If the carrying state of the target vehicle is the full - load state, determine the vehicle hanging sub - frames corresponding to the full - load state in the original vehicle hanging frame as the vehicle hanging sub - frames participating in the application of the data augmentation scheme.

[0036] Optionally, in the above point cloud data augmentation method, the target splitting scheme is the second splitting scheme. According to the point cloud data distribution characteristics of the original car hanging frame, the car hanging sub-frames participating in the application data augmentation scheme are determined among all the car hanging sub-frames, including:

[0037] Determine whether the point cloud data distribution characteristics of the original car hanging frame satisfy the preset complete car hanging contour scanning characteristics. Satisfying the preset complete car hanging contour scanning characteristics indicates that the point cloud distribution within the original car hanging frame is uniform and the car hanging contour scanning is complete;

[0038] If it is determined that the point cloud data distribution characteristics of the original car hanging frame satisfy the preset complete car hanging contour scanning characteristics, then the car hanging sub-frame corresponding to the original car hanging frame and the preset complete car hanging contour scanning characteristics is determined as the car hanging sub-frame participating in the application data augmentation scheme.

[0039] Optionally, in the above point cloud data augmentation method, determining whether the point cloud data distribution characteristics of the original car hanging frame satisfy the preset complete car hanging contour scanning characteristics includes:

[0040] Determine the point cloud quantity of each car hanging sub-frame of the target vehicle;

[0041] Determine whether the third point cloud ratio is greater than the first preset ratio and whether the fourth point cloud ratio is less than the second preset ratio; the third point cloud ratio is the ratio of the sum of the point cloud data quantities of the first car hanging sub-frame and the second car hanging sub-frame to the total quantity of all point cloud data within the original car hanging frame; the fourth point cloud ratio is the ratio of the difference between the point cloud data quantities of the first car hanging sub-frame and the second car hanging sub-frame to the total quantity of all point cloud data within the original car hanging frame; the first car hanging sub-frame and the second car hanging sub-frame are respectively the car hanging sub-frames located on the left and right sides of the original car hanging frame;

[0042] If it is determined that the third point cloud ratio is greater than the first preset ratio and the fourth point cloud ratio is less than the second preset ratio, then it is determined that the point cloud data distribution characteristics of the original car hanging frame satisfy the preset complete car hanging contour scanning characteristics;

[0043] If it is determined that the third point cloud ratio is not greater than the first preset ratio, and / or, the fourth point cloud ratio is not less than the second preset ratio, then it is determined that the point cloud data distribution characteristics of the original car hanging frame do not satisfy the preset complete car hanging contour scanning characteristics;

[0044] Wherein, the first preset ratio is greater than the second preset ratio.

[0045] Optionally, in the above point cloud data augmentation method, determining the point cloud quantity of each car hanging sub-frame of the target vehicle includes:

[0046] Determine the Euclidean distance between each point cloud data in the original car hanging frame and the center points of each of the car hanging sub - frames respectively, where the center point of the car hanging sub - frame is the geometric centroid of the car hanging sub - frame;

[0047] For each piece of point cloud data, select the car hanging sub - frame with the minimum distance from the Euclidean distances between it and the center points of each of the car hanging sub - frames as the car hanging sub - frame to which the point cloud data belongs, and obtain the point cloud quantity of each of the car hanging sub - frames of the target vehicle.

[0048] A second aspect of the present application discloses a point cloud data augmentation device, including:

[0049] A first determination unit, configured to, after obtaining the point cloud data of the target vehicle from the point cloud data set, determine the original car hanging frame of the target vehicle according to the point cloud data, where the original car hanging frame is the labeled frame of the target vehicle, and the target vehicle is the vehicle for which point cloud data augmentation processing needs to be performed in the point cloud data set;

[0050] A splitting unit, configured to split the original car hanging frame of the target vehicle to obtain the car hanging sub - frames of the target vehicle;

[0051] A second determination unit, configured to determine the car hanging sub - frames participating in the application data augmentation scheme among all the car hanging sub - frames according to the point cloud data distribution characteristics of the original car hanging frame.

[0052] A third aspect of the present application discloses an electronic device, including: a memory and a processor;

[0053] Wherein, the memory is used to store a computer program;

[0054] The processor is used to execute the computer program, and specifically used to implement the point cloud data augmentation method as described in any one of the first aspect.

[0055] A fourth aspect of the present application discloses a computer storage medium, used to store a computer program, and when the computer program is executed, it is specifically used to implement the point cloud data augmentation method as described in any one of the first aspect.

[0056] The present invention provides a method, apparatus, electronic device, and storage medium for point cloud data augmentation. After obtaining the point cloud data of a target vehicle from a point cloud dataset, the method for point cloud data augmentation determines the original vehicle hanging frame of the target vehicle according to the point cloud data. The original vehicle hanging frame is the labeled frame of the target vehicle, and the target vehicle is the vehicle in the point cloud dataset that needs to be processed for point cloud data augmentation; the original vehicle hanging frame of the target vehicle is split to obtain the vehicle hanging sub-frames of the target vehicle; according to the point cloud data distribution characteristics of the original vehicle hanging frame, the vehicle hanging sub-frames participating in the application of the data augmentation scheme are determined among all the vehicle hanging sub-frames. That is, based on splitting the original vehicle hanging frame of the target vehicle into multiple vehicle hanging sub-frames, the method selects the vehicle hanging sub-frames participating in the application of the data augmentation scheme in combination with the point cloud data distribution characteristics, solving the problem that existing point cloud data augmentation performs single operations on the entire scene or certain instance frames in the scene and cannot achieve the expected effect of simulating more scenes. Description of the Drawings

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings according to the provided drawings without creative efforts.

[0058] Figure 1 It is a flowchart of a method for point cloud data augmentation provided by this application;

[0059] Figure 2 It is a flowchart for determining the vehicle hanging sub-frames of a target vehicle provided by an embodiment of this application;

[0060] Figure 3 It is a schematic diagram of the spatial position of the original vehicle hanging frame of a target vehicle in the Cartesian right-handed coordinate system provided by an embodiment of this application;

[0061] Figure 4 It is a schematic diagram of dividing the original vehicle hanging frame into vehicle hanging sub-frames provided by an embodiment of this application;

[0062] Figure 5 It is a schematic diagram of the coordinate conversion between the center point and the corner points of a vehicle hanging sub-frame provided by an embodiment of this application;

[0063] Figure 6 It is another schematic diagram of dividing the original vehicle hanging frame into vehicle hanging sub-frames provided by an embodiment of this application;

[0064] Figure 7 It is a flowchart for determining the vehicle hanging sub-frames participating in the application of the data augmentation scheme provided by an embodiment of this application;

[0065] Figure 8Flow chart of another point cloud data augmentation method provided by an embodiment of the present application;

[0066] Figure 9 Structural schematic diagram of a point cloud data augmentation device provided by an embodiment of the present application. Detailed implementation manners

[0067] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0068] The present application provides a point cloud data augmentation method and device to solve the problem that existing point cloud data augmentation only performs single operations on a scene or some instance frames in the scene and cannot achieve the expected effect of simulating more scenes.

[0069] Please refer to Figure 1 , the point cloud data augmentation method may include the following steps:

[0070] S101. After obtaining the point cloud data of the target vehicle from the point cloud dataset, determine the original vehicle hitch frame of the target vehicle according to the point cloud data.

[0071] Among them, the original vehicle hitch frame is the annotation frame of the target vehicle; the target vehicle may be a vehicle in the point cloud dataset that needs to perform point cloud data augmentation-related processing to simulate more point cloud scenes, and the point cloud dataset includes the point cloud data of several vehicles. In practical applications, the target vehicle may be a container truck. Of course, it is not limited thereto, and it may also be other vehicles equipped with vehicle hitches. The present application does not limit the specific type of the target vehicle, and all are within the protection scope of the present application.

[0072] In some embodiments, the point cloud data of the target vehicle can be obtained by a lidar; of course, it is not limited thereto, and the point cloud data of the target vehicle can also be obtained by other existing methods. The present application does not limit the acquisition method of the point cloud data of the target vehicle, and all are within the protection scope of the present application.

[0073] In practical applications, the point cloud data of the target vehicle can be annotated, and the original vehicle hitch frame of the target vehicle can be determined according to each annotated point cloud data.

[0074] Specifically, the manual annotation method can be adopted. According to different vehicle types, the annotation of common original vehicle hitch frames can be divided into the following situations:

[0075] Annotation situation for large container trucks: Large container trucks generally come with trailers, and during manual annotation, the truck head and trailer are usually annotated separately. Specifically, since there is generally a large gap between the truck head and trailer of large container trucks, it is not very reasonable to combine them. In this case, the determination of the original trailer frame is completely done manually.

[0076] Annotation situation for small container trucks: Since most small container trucks have a rigid connection between the truck head and trailer, such as a sprinkler truck, or the distance between them is very small, and it is not easy for the truck head to block the trailer when the vehicle turns. Therefore, for some small container trucks, the truck head and trailer are not separated during the manual annotation of the original trailer frame.

[0077] It should be noted that for small container trucks with a rigid connection, most of the occlusions of the trailer occur at the rear of the vehicle. When the target vehicle and the self-vehicle carrying the lidar are moving towards each other, the rear of the vehicle cannot be seen at all. In this case, the proposed trailer sub-frame splitting method of this application can be directly applied to the original trailer frame, and data augmentation can be performed on the trailer sub-frame at the rear of the vehicle by reducing a certain number of point clouds.

[0078] It should be noted that for the relevant description of determining the original trailer frame of the target vehicle based on point cloud data, reference can also be made to the prior art, and this application will not elaborate on it one by one.

[0079] S102. Split the original trailer frame of the target vehicle to obtain the trailer sub-frame of the target vehicle.

[0080] In practical applications, for the specific process of executing step S102, splitting the original trailer frame of the target vehicle to obtain the trailer sub-frame of the target vehicle, reference can be made to Figure 2 , which mainly includes steps S201 and S202:

[0081] S201. Determine the target splitting scheme for the target vehicle.

[0082] Among them, the target splitting scheme generally includes: the first splitting scheme and the second splitting scheme.

[0083] In practical applications, the first splitting scheme can be a splitting scheme for distinguishing whether the target vehicle is in an empty load state or a full load state; for example, dividing the original trailer frame of the target vehicle into eight hexahedrons with the same heading angle as the original trailer frame, that is, the first splitting scheme can equally divide the original trailer frame into eight hexahedrons; specifically, the hexahedron can be a cuboid or a cube; of course, it is not limited to this, and it can also be determined according to the specific application environment and user requirements. This application does not limit it, and all are within the protection scope of this application.

[0084] The second splitting scheme can be a splitting scheme for distinguishing whether the point cloud data of the original vehicle hanging frame obtained by scanning the target vehicle represents a relatively complete vehicle hanging contour. For example, the original vehicle hanging frame of the target vehicle can be split into multiple pyramids (quadrangular pyramids) sharing a vertex. The second splitting scheme can split the original vehicle hanging frame into six quadrangular pyramids. Specifically, the base of the pyramid can be a square or a rectangle. Of course, it is not limited to this, and it can also be determined according to the specific application environment and user requirements. The present application does not limit it, and all are within the protection scope of the present application.

[0085] S202. Split the original vehicle hanging frame according to the target splitting scheme to obtain the vehicle hanging sub-frame of the target vehicle.

[0086] The following specifically describes the situation of splitting the original vehicle hanging frame according to the first splitting scheme to obtain the vehicle hanging sub-frame of the target vehicle:

[0087] Assume that the original vehicle hanging frame of the target vehicle is as Figure 3 shown. Its position and shape can generally be described by the following seven quantities, namely the central point coordinates (x, y, z) in space, the length, width, and height (l, w, h) of the frame, and the heading angle alpha of the frame. The situation of equally dividing the original vehicle hanging frame into 8 hexahedrons (cuboids) can be as Figure 4 shown. It can be seen from the figure that each hexahedron is a vehicle hanging sub-frame, which are respectively represented as: cube1, cube2, cube3, cube4, cube5, cube6, cube7, cube8. The heading angle of each vehicle hanging sub-frame is the same as that of the original vehicle hanging frame, still heading. The l cube , width w cube , height h cube are specifically calculated as follows:

[0088]

[0089] The following specifically describes the situation of splitting the original vehicle hanging frame according to the second splitting scheme to obtain the vehicle hanging sub-frame of the target vehicle:

[0090] Assume that the original vehicle hanging frame of the target vehicle is as Figure 3 shown. Its position and shape can generally be described by the following seven quantities, namely the central point coordinates (x, y, z) in space, the length, width, and height (l, w, h) of the frame, and the heading angle alpha of the frame. The situation of equally dividing the original vehicle hanging frame into 6 cones can be as Figure 5 and Figure 6 shown. It can be seen from the figure that each cone is a vehicle hanging sub-frame, which are respectively represented as: pyramid1, pyramid2, pyramid3, pyramid4, pyramid5, pyramid6.

[0091] S103. Determine the vehicle hanging sub - frames that participate in the application data augmentation scheme among all vehicle hanging sub - frames according to the distribution characteristics of the point cloud data of the original vehicle hanging frame.

[0092] In practical applications, the target splitting scheme is the first splitting scheme. The specific process of executing step S103, that is, determining the vehicle hanging sub - frames that participate in the application data augmentation scheme among all vehicle hanging sub - frames according to the distribution characteristics of the point cloud data of the original vehicle hanging frame, can be as follows Figure 7 shown, mainly including steps S301 and S302:

[0093] S301. Determine the carrying state of the target vehicle according to the distribution characteristics of the point cloud data of the original vehicle hanging frame.

[0094] Among them, the carrying state may include: unloaded state and fully - loaded state; of course, it is not limited to this, and it can also be determined according to the specific application environment and user requirements. This application does not limit it, and all are within the protection scope of this application.

[0095] In some embodiments, the specific process of executing step S301, that is, determining the carrying state of the target vehicle according to the distribution characteristics of the point cloud data of the original vehicle hanging frame, can be as follows:

[0096] Judge whether the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset unloaded state distribution characteristics.

[0097] If it is judged that the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset unloaded state distribution characteristics, then determine that the carrying state of the target vehicle is the unloaded state; if it is judged that the distribution characteristics of the point cloud data of the original vehicle hanging frame do not conform to the preset unloaded state distribution characteristics, then determine that the carrying state of the target vehicle is the fully - loaded state.

[0098] In some embodiments, the process of judging whether the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset unloaded state distribution characteristics can be as follows, including steps S501 to S504:

[0099] S501. Determine the number of point clouds of each vehicle hanging sub - frame of the target vehicle.

[0100] In practical applications, first, the Euclidean distance between each point cloud data in the original vehicle hanging frame and the center point of each vehicle hanging sub - frame can be determined respectively. The center point of the vehicle hanging sub - frame is the geometric center of gravity of the vehicle hanging sub - frame; then, for each point cloud data, select the vehicle hanging sub - frame with the smallest distance from its Euclidean distances from the center points of each vehicle hanging sub - frame as the vehicle hanging sub - frame to which the point cloud data belongs, and obtain the number of point clouds of each vehicle hanging sub - frame of the target vehicle.

[0101] S502. Judge whether the first point cloud ratio is greater than the preset unloaded ratio.

[0102] Among them, the first point cloud ratio is the ratio of the sum of the amounts of point cloud data at the midpoints of each hanging sub-frame in the lower half of the original vehicle hanging frame to the amount of all point cloud data within the original vehicle hanging frame. Combining Figure 4 , each hanging sub-frame in the lower half of the original vehicle hanging frame can be cube1, cube2, cube3, cube4 in the figure.

[0103] If it is determined that the first point cloud ratio is greater than the preset no-load ratio, step S503 can be executed; if it is determined that the first point cloud ratio is not greater than the preset no-load ratio, step S504 can be executed.

[0104] S503. Determine that the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset no-load state distribution characteristics.

[0105] In practical applications, when it is determined that the first point cloud ratio is greater than the preset no-load ratio, it means that the point cloud in the lower half of the original vehicle hanging frame along the height direction is dense, and it can be considered that the original vehicle hanging frame represents a container truck hanging without cargo.

[0106] S504. Determine that the distribution characteristics of the point cloud data of the original vehicle hanging frame do not conform to the preset no-load state distribution characteristics.

[0107] In practical applications, when it is determined that the first point cloud ratio is not greater than the preset no-load ratio, it means that the density of the point cloud in the upper half of the original vehicle hanging frame along the height direction is about the same as that in the lower half, and it can be considered that the original vehicle hanging frame represents a container truck hanging loaded with goods.

[0108] Similarly, in another embodiment, the process of determining whether the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset no-load state distribution characteristics can be as follows, including steps S601 to S604:

[0109] S601. Determine the amount of point cloud of each hanging sub-frame of the target vehicle.

[0110] It should be noted that the relevant description of step S601, determining the amount of point cloud of each hanging sub-frame of the target vehicle, can refer to the above step S501, and will not be elaborated here one by one.

[0111] S602. Determine whether the second point cloud ratio is greater than the preset full-load ratio.

[0112] Among them, the second point cloud ratio is the ratio of the sum of the amounts of point cloud data at the midpoints of each hanging sub-frame in the upper half of the original vehicle hanging frame to the amount of all point cloud data within the original vehicle hanging frame. Combining Figure 4 , each hanging sub-frame in the upper half of the original vehicle hanging frame can be cube5, cube6, cube7, cube8 in the figure.

[0113] If it is determined that the second point cloud ratio is greater than the preset full load ratio, step S603 can be executed; if it is determined that the second point cloud ratio is not greater than the preset full load ratio, step S604 can be executed.

[0114] S603: Determine whether the distribution characteristics of the point cloud data of the original vehicle hanging frame meet the preset distribution characteristics of the no-load state.

[0115] In practical applications, when it is determined that the proportion of the second point cloud is greater than the preset full load proportion, it means that the density of the point cloud in the middle half of the original vehicle trailer frame in the height direction is almost the same as that of the point cloud in the lower half. It can be considered that the original vehicle trailer frame represents a container trailer loaded with goods.

[0116] S604: Determine whether the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset no-load state distribution characteristics.

[0117] In practical applications, when it is determined that the proportion of the second point cloud is greater than the preset full load proportion, it means that the point cloud in the lower half of the original vehicle trailer frame along the height direction is relatively dense, and it can be considered that the original vehicle trailer frame represents a container trailer without cargo.

[0118] To sum up the above, the distribution characteristics of the point cloud data of the original vehicle hanging frame meet the preset no-load state distribution characteristics, which can be: the sum of the number of point cloud data in each vehicle hanging sub-frame located in the lower half of the original vehicle hanging frame of the target vehicle, and the proportion of the number of all point cloud data in the original vehicle hanging frame exceeds the preset no-load proportion; or, the sum of the number of point cloud data in each vehicle hanging sub-frame located in the upper half of the original vehicle hanging frame of the target vehicle, and the proportion of all point cloud data in the original vehicle hanging frame is lower than the preset full-load proportion.

[0119] Contrary to the empty-load judgment condition, the distribution characteristics of the point cloud data of the original vehicle hanging frame meet the preset full-load state distribution characteristics, which can be: the sum of the number of point cloud data in each vehicle hanging sub-frame located in the lower half of the original vehicle hanging frame of the target vehicle, and the proportion of the number of all point cloud data in the original vehicle hanging frame does not exceed the preset empty-load proportion; or, the sum of the number of point cloud data in each vehicle hanging sub-frame located in the upper half of the original vehicle hanging frame of the target vehicle, and the proportion of all point cloud data in the original vehicle hanging frame exceeds the preset full-load proportion.

[0120] The specific value of the preset no-load ratio can be 70%; of course, it is not limited to this, and it can be determined according to the specific application environment and user needs, all within the scope of protection of this application. The specific value of the preset full-load ratio can be 30%; of course, it is not limited to this, and it can be determined according to the specific application environment and user needs, all within the scope of protection of this application.

[0121] In combination with the above, exemplary, also with Figure 4Taking the shown splitting scheme as an example, the specific process of determining whether the point cloud data distribution characteristics of the original car hanging frame conform to the preset no-load state distribution characteristics is as follows: First, determine the center point coordinates (x cube , y cube , z cube ) of the 8 car hanging sub-frames. For the convenience of calculation, first use the 2D information of the original car hanging frame mentioned above, that is, (x, y, l, w, alpha), to obtain the coordinates of the four corner points corner1, corner2, corner3, and corner4 (only distinguish the four corner points, and their order is not important, and the coordinate order of each corner point is also x, y, z). The specific calculation is as follows:

[0122]

[0123]

[0124]

[0125]

[0126] Then, add the vertical coordinates of the above four points to the height h of the original frame respectively to obtain the coordinates of the remaining four corner points corner5, corner6, corner7, and corner8 of the original frame (the order should be the same as that of corner1, -corner4). The specific calculation is as follows:

[0127]

[0128]

[0129]

[0130]

[0131] The 8 car hanging sub-frames can be named through the above corner point coordinates, that is, the hexahedron where corner1 is located is cube1, the sub-cube where corner2 is located is cube2, and so on. Then, convert the corner point coordinates of each car hanging sub-frame into the center point coordinate representation. The method is the average value of the coordinates of the two points on the space diagonal of the hexahedron. For example, the center point coordinates of the car hanging sub-frame cube1 are obtained from the position of the corner point corner1 and the center point center of the original car hanging frame. The specific calculation is as follows:

[0132]

[0133] Subsequently, the point cloud data of the original car hanging frame can be clustered, that is, for each point cloud data (x point , ypoint ) are all related to the center coordinates center of each trailer sub-frame cube1 , center cube2 , center cube3 , center cube4 , center cube5 , center cube6 , center cube7 , center cube8 Calculate the Euclidean distance, and select the trailer sub-frame corresponding to the smallest distance as the trailer sub-frame to which the point cloud data belongs.

[0134] Assume that the number of point cloud data in each trailer sub-frame is: num points cube1 , numpoints cube2 , num points cube3 , num points cube4 , num points cube5 , num points cube6 , num points cube7 , numpoints cube8 . If the sum of num points in the trailer sub-frame cube5 , num points cube6 , numpoints cube7 , numpoints cube8 is less than 30% of the total number of all point cloud data in the original trailer frame, that is, the point cloud in the lower half of the original trailer frame along the height direction is dense, then it is considered that the original trailer frame represents an empty container truck trailer without cargo, that is, the target vehicle is in an empty state, otherwise it is considered that the original trailer frame is a container truck trailer loaded with goods, that is, the target vehicle is in a full load state.

[0135] S302. Determine the trailer sub-frames participating in the application data augmentation scheme among all trailer sub-frames according to the carrying state of the target vehicle.

[0136] In practical applications, if the carrying state of the target vehicle is an empty state, then determine the trailer sub-frames corresponding to the empty state in the original trailer frame as the trailer sub-frames participating in the application data augmentation scheme; if the carrying state of the target vehicle is a full load state, then determine the trailer sub-frames corresponding to the full load state in the original trailer frame as the trailer sub-frames participating in the application data augmentation scheme.

[0137] In some embodiments, the trailer sub-frames corresponding to the empty state in the original trailer frame can be the trailer sub-frames in the lower half of the original trailer frame; similarly Figure 4For example, the hanging sub - frames at the first position can be cube1, cube2, cube3, cube4 in the figure.

[0138] In some embodiments, the hanging sub - frames corresponding to the full - load state in the original hanging frame can be the hanging sub - frames located in the second half of the hanging direction of the original hanging frame in the direction of the hanging heading; similarly, taking Figure 4 For example, the hanging sub - frames at the second position can be cube2, cube3, cube6, cube7 in the figure.

[0139] In practical applications, when the target splitting scheme is the second splitting scheme, the specific process of performing step S103, that is, determining the hanging sub - frames participating in the application data augmentation scheme among all the hanging sub - frames according to the point - cloud data distribution characteristics of the original hanging frame, can be as follows:

[0140] Judge whether the point - cloud data distribution characteristics of the original hanging frame meet the preset complete hanging - contour scanning characteristics.

[0141] Among them, meeting the preset complete hanging - contour scanning characteristics can indicate that the point - cloud distribution in the original hanging frame is uniform and the hanging contour scanning is complete.

[0142] In some embodiments, the process of judging whether the point - cloud data distribution characteristics of the original hanging frame meet the preset complete hanging - contour scanning characteristics can be as follows, mainly including steps S701 to S704:

[0143] S701: Determine the number of point - clouds of each hanging sub - frame of the target vehicle.

[0144] It should be noted that the relevant description of step S701, determining the number of point - clouds of each hanging sub - frame of the target vehicle, can refer to the above step S501, and will not be elaborated here one by one.

[0145] S702: Judge whether the third point - cloud ratio is greater than the first preset ratio and whether the fourth point - cloud ratio is less than the second preset ratio.

[0146] Among them, the third point - cloud ratio is the ratio of the sum of the number of point - cloud data of the first hanging sub - frame and the second hanging sub - frame to the total number of point - cloud data in the original hanging frame; the fourth point - cloud ratio is the ratio of the difference between the number of point - cloud data of the first hanging sub - frame and the second hanging sub - frame to the total number of point - cloud data in the original hanging frame; the first hanging sub - frame and the second hanging sub - frame are respectively the hanging sub - frames located on the left and right sides of the original hanging frame. Combining Figure 6 , the first hanging sub - frame and the second hanging sub - frame can be percent2 and percent4 in the figure respectively.

[0147] If it is determined that the proportion of the third point cloud is greater than the first preset proportion and the proportion of the fourth point cloud is less than the second preset proportion, then step S703 can be executed; if it is determined that the proportion of the third point cloud is not greater than the first preset proportion, and / or the proportion of the fourth point cloud is not less than the second preset proportion, then step S704 can be executed.

[0148] S703. Determine that the point cloud data distribution characteristics of the original car hanging frame satisfy the preset car hanging complete contour scanning characteristics.

[0149] In practical applications, when it is determined that the proportion of the third point cloud is greater than the first preset proportion and the proportion of the fourth point cloud is less than the second preset proportion, it can be explained that the point cloud distribution in the original car hanging frame is uniform and the lidar scans the contour of the vehicle relatively completely. It can be determined that the point cloud data distribution characteristics of the original car hanging frame satisfy the preset car hanging complete contour scanning characteristics.

[0150] S704. Determine that the point cloud data distribution characteristics of the original car hanging frame do not satisfy the preset car hanging complete contour scanning characteristics.

[0151] Among them, the first preset proportion is greater than the second preset proportion.

[0152] In practical applications, when it is determined that the proportion of the third point cloud is not greater than the first preset proportion, and / or the proportion of the fourth point cloud is less than the second preset proportion, it can be explained that the point cloud distribution in the original car hanging frame is not uniform and the lidar scans the contour of the vehicle incompletely. It can be determined that the point cloud data distribution characteristics of the original car hanging frame do not satisfy the preset car hanging complete contour scanning characteristics.

[0153] Combined with the above, the original car hanging frame is split into 6 car hanging sub-frames of pyramids sharing a vertex according to the second splitting scheme, namely: pyramid1, pyramid2, pyramid3, pyramid4, pyramid5, pyramid6, that is Figure 5 and Figure 6 as shown. Assume that the percentages of the number of point cloud data in each car hanging sub-frame in the total number of point cloud data in the original car hanging frame are: percent1, percent2, percent3, percent4, percent5, percent6. If the sum of percent2 and percent4 is greater than 40% and the difference between them is less than 10%, it can be considered that the point cloud data distribution of the original car hanging frame is relatively uniform and the lidar scans the car hanging contour of the target vehicle relatively completely. It can be judged that the point cloud data distribution characteristics of the original car hanging frame satisfy the preset car hanging complete contour scanning characteristics.

[0154] It should be noted that when the lidar scans a target, some gaps on the target are ignored, and the target is considered to be a solid cube (which is also more in line with the actual shape of objects such as trucks). The laser points are all scanned on the surface of the entire cube. In addition, due to the installation position of the lidar, there are basically no points on the bottom surface of the vehicle because the chassis is very close to the ground and the radar cannot scan it. Also, when the lidar is installed on the vehicle, there are not many points on the top surface of the vehicle when scanning other vehicles. Therefore, the problem of whether the so-called point cloud is uniform is simplified to the problem of whether the points are evenly distributed on the four surrounding surfaces of a cube. Then, according to engineering practice, if some container trucks or dump trucks are calculated with a length of 15 meters, a width of 3 meters, and a height of 2.5 meters, then the surface area occupied by percent2 or percent4 is approximately 40%. However, here, in order not to make the standard too strict, and although the laser generally follows the principle that more points are on the larger surface area during actual scanning, special situations caused by the installation position of the lidar cannot be excluded. Therefore, the standard is relaxed to the sum of the two being greater than 40%. However, the percentage difference between the point clouds they own cannot be too large, that is, a relatively strict standard of 10%, that is, if it is sparse, it should be sparse together, and if it is dense, it should be dense together to be considered uniform.

[0155] If it is determined that the point cloud data distribution characteristics of the original vehicle hanging frame satisfy the preset vehicle hanging complete contour scanning characteristics, then the vehicle hanging sub-frame in the original vehicle hanging frame corresponding to the preset vehicle hanging complete contour scanning characteristics is determined as the vehicle hanging sub-frame participating in the application data augmentation scheme.

[0156] It should be noted that selecting the original vehicle hanging frame with point cloud data distribution characteristics that satisfy the preset vehicle hanging complete contour scanning characteristics for data augmentation can further improve the authenticity of the point cloud data augmentation compared to selecting the original vehicle hanging frame with point cloud data distribution characteristics that do not satisfy the preset vehicle hanging complete contour scanning characteristics.

[0157] In some embodiments, the vehicle hanging sub-frame in the original vehicle hanging frame corresponding to the preset vehicle hanging complete contour scanning characteristics can be the vehicle hanging sub-frames of two adjacent sides in the original vehicle hanging frame away from the vehicle hanging heading in the direction of the vehicle hanging heading. For example, Figure 6 pyramid3 and pyramid4 in ; Of course, it is not limited to this, and it can also be determined according to the specific application environment and user needs. This application does not make a limit and is within the protection scope of this application.

[0158] It should also be noted that the first splitting scheme only considers the shape of the target vehicle itself and does not consider the change in the density of the point cloud data caused by the relationship between the installation position of the lidar and the position where the vehicle travels. Therefore, by considering the change in the density of the point cloud data caused by the relationship between the installation position of the lidar and the position where the vehicle travels through the second splitting scheme, the authenticity of the point cloud data augmentation can be further improved.

[0159] Based on the above principle, after obtaining the point cloud data of the target vehicle from the point cloud dataset, the point cloud data augmentation method provided in this embodiment determines the original vehicle-mounted frame of the target vehicle according to the point cloud data. The original vehicle-mounted frame is the annotation frame of the target vehicle, and the target vehicle is the vehicle that needs to perform point cloud data augmentation processing in the point cloud dataset; the original vehicle-mounted frame of the target vehicle is split to obtain the vehicle-mounted sub-frames of the target vehicle; according to the point cloud data distribution characteristics of the original vehicle-mounted frame, the vehicle-mounted sub-frames participating in the application of the data augmentation scheme are determined among all the vehicle-mounted sub-frames. That is, based on splitting the original vehicle-mounted frame of the target vehicle into multiple vehicle-mounted sub-frames, combined with the point cloud data distribution characteristics, the vehicle-mounted sub-frames participating in the application of the data augmentation scheme are selected, solving the problem that some point cloud data augmentations only perform single operations on the scene or some instance frames in the scene and cannot achieve the expected effect of simulating more scenes.

[0160] Based on the above, optionally, after the point cloud data augmentation method provided in another embodiment of the present application executes step S103, that is, determines the vehicle-mounted sub-frames participating in the application of the data augmentation scheme among all the vehicle-mounted sub-frames according to the point cloud data distribution characteristics of the original vehicle-mounted frame, please refer to Figure 8 , it further includes:

[0161] S401: Perform a first process on the point cloud data in the vehicle-mounted sub-frames participating in the application of the data augmentation scheme and a second process on the point cloud data within the original vehicle-mounted frame respectively to simulate the point cloud scene of the target vehicle in the occluded state.

[0162] Among them, the first process and the second process are complementary processes. Specifically, the first process can be randomly removing a certain proportion of point clouds, and the second process can be adding noise points.

[0163] Specifically, taking the vehicle-mounted sub-frames participating in the application of the data augmentation scheme as cube1, cube2, cube3, cube4, for example:

[0164] Let the set of vehicle-mounted sub-frames participating in the application of the data augmentation scheme be cube parts {cube1, cube2, cube3, cube4}; the set of N points in cube3 is points = {point1,..., pointn}, and a percentage p can be set artificially. Then, randomly select p*N points from the set points to retain, and delete the remaining points from the set. The remaining sub-frames in the set cube_parts are processed in the same way. That is, for the remaining sub-frames in the set cube_parts, first set a percentage p, and then randomly select p*N points from the set points to retain, and delete the remaining points from the set.

[0165] In practical applications, performing a first processing on the point cloud data in the vehicle-mounted sub-frame participating in the application data augmentation scheme can achieve the purpose of simulating the situation where the front of the target vehicle blocks the tail of the vehicle-mounted device during a turn, resulting in sparse laser point clouds. However, performing a second processing on the point cloud data in the original vehicle-mounted frame can prevent, when performing the first processing, completely removing too many point clouds or the randomness of removing point clouds causing all the removed point clouds to be located near a certain position, appearing too deliberate, and thus resulting in unrealistic augmented point clouds.

[0166] Specifically, assume that there are a total of M point cloud data in the original vehicle-mounted frame. Set a relatively small probability q with a value range of 0 - 0.3, and randomly generate some points randompoints = {p r1, p r2,...}. The total number of random points in this set is q * M. However, the coordinates (x r , y r , z r ) of any random noise point must be within the original vehicle-mounted frame, that is, satisfy the following conditions:

[0167]

[0168] ; where abs represents taking the absolute value of this value.

[0169] In this embodiment, by performing a first processing on the point cloud data in the vehicle-mounted sub-frame participating in the application data augmentation scheme, the situation where the front of the target vehicle blocks the tail of the vehicle-mounted device during a turn, resulting in sparse laser point clouds, can be simulated. Performing a second processing on the point cloud data in the original vehicle-mounted frame can prevent, when performing the first processing, completely removing too many point clouds or the randomness of removing point clouds causing all the removed point clouds to be located near a certain position, appearing too deliberate, and thus resulting in unrealistic augmented point clouds, further improving the authenticity of point cloud data augmentation.

[0170] It should be noted that for a certain frame of point cloud scene, the data augmentation method of randomly rotating the entire scene by a certain angle can also simulate more scenes, and operating at the instance box level can also simulate more scenes. This application can make the augmentation process more delicate by performing different augmentation schemes on different parts of the instance box.

[0171] Based on the point cloud data augmentation method provided in the above embodiment, for better illustration of the point cloud data augmentation method, taking the target vehicle as a container truck as an example, this method aims to reasonably divide some container truck instance boxes in the point cloud data according to the point cloud sparsity degree and adopt appropriate and different data augmentation methods for their different regions. This process aims to realistically simulate the actual form of the annotation box of large target vehicles such as container trucks and apply it to other frames of the data set. Specifically, the application process of this point cloud data augmentation method is as follows:

[0172] Step S1, in the point cloud dataset scenario, the position and shape of a container truck trailer (hereinafter simply referred to as the trailer) are generally described by the following seven quantities, namely the center point coordinates (x, y, z) in space, the length, width, and height of the box (l, w, h), and the heading angle alpha of the box, as Figure 3 shown. First, an original trailer box is evenly divided into eight cubes (the above-mentioned hexahedral trailer sub-boxes) in space, as Figure 4 shown. Obviously, the heading angles of the eight sub-cubes are the same as those of the original box cube (the above-mentioned original trailer box), still heading, the length l cube , width w cube , height h cube The specific calculations are as follows:

[0173]

[0174] Step S2, obtain the center point coordinates (x cube , y cube , z cube ) of the eight sub-cubes. For the convenience of calculation, first use the 2D information of the original empty container truck box mentioned in Step S1, that is, (x, y, l, w, alpha), to obtain the coordinates of the four corner points corner1, corner2, corner3, and corner4 (only distinguish the four corner points, and their order is not important, and the coordinate order of each corner point is also x, y, z). The specific calculations are as follows:

[0175]

[0176]

[0177]

[0178]

[0179] Then, add the vertical coordinates of the above four points to the height h of the original box respectively to obtain the coordinates of the remaining four corner points corner5, corner6, corner7, and corner8 of the original box (the order should be the same as that of corner1, -corner4). The specific calculations are as follows:

[0180]

[0181]

[0182]

[0183]

[0184] The eight sub - cubes can be named according to the above corner coordinates. That is, the hexahedron where corner1 is located is cube1, the sub - cube where corner2 is located is cube2, and so on. Then, the corner coordinates of each sub - cube are converted into the coordinates of the center point. The method is the average value of the coordinates of two points on the space diagonal of the sub - cube. For example, the center point coordinates of sub - cube cube1 are obtained from the position of corner1 and the center point center of the original box. The specific calculation is as follows:

[0185]

[0186] Step S3: Cluster the point cloud in the original box. That is, for each point (x point , y point ), calculate the Euclidean distance from it to the center point coordinates center cube1 , center cube2 , center cube3 , center cube4 , center cube5 , center cube6 , center cube7 , center cube8 obtained in step S2, and select the sub - cube corresponding to the minimum distance as the cube to which the point belongs.

[0187] Step S4: Determine the sub - cube boxes participating in the application of the data augmentation scheme according to the sparsity of the point cloud. According to the calculation results of step S3, each point in the original box belongs to a sub - cube. In this way, the number of points in each sub - cube can be counted as num points cube1 , numpoints cube2 , numpoints cube3 , num points cube4 , numpoints cube5 , num points cube6 , numpoints cube7 , num points cube8 . If the number of points in the sub - cube num points cube5 , numpoints cube6 , numpoints cube7 , numpoints cube8The sum of them accounts for less than 30% of the sum of all the points in the original box. That is, if the point cloud in the lower half along the height direction in the original box is dense, it is considered that the original box represents an empty container truck trailer without cargo (hereinafter referred to as an empty trailer), and then the sub-box set cubeparts = {cube1, cube2, cube3, cube4} for the data augmentation scheme is required; otherwise, it is considered that the original box is a container truck trailer loaded with goods (hereinafter referred to as a full trailer), and then the sub-box set cubeparts = {cube2, cube3, cube6, cube7} for the data augmentation scheme is required.

[0188] Step S5: Apply the data augmentation method of "randomly removing a certain proportion of points" to the sub-cube boxes in the set cube parts generated in step S4 to simulate the situation where the front of the container truck blocks the tail of the trailer during turning, resulting in sparse lidar point clouds. The specific method is as follows: For example, if the set of N points in the sub-cube box cube3 is points = {point 1,..., pointn}, a percentage p can be artificially set. Then, p*N points are randomly selected and retained in the set points, and the remaining points are deleted from the set. Apply the data augmentation method of "completely removing points (dropout)" to the remaining sub-boxes in the set cube parts according to this method.

[0189] Step S6: Refine the sub-box division scheme according to the point cloud density again. Since only the vehicle's own shape is considered in step S4 without considering the change in point cloud density caused by the relationship between the lidar installation position and the vehicle's traveling position, it is necessary to re-divide the original box into sub-regions in a pyramid shape (the above-mentioned conical trailer sub-boxes), as Figure 5 and Figure 6 shown. The original box will be divided into six pyramid regions, namely pyramid1,.., pyramid6. By analogy with the methods in step S2 and step S3, the points in the original box obtained in step S5 are divided into each sub-pyramid region. Thus, the percentages of the number of points in each sub-pyramid accounting for the total number of points in the original box can be calculated as percent1,..., percent6. If the sum of percent2 and percent4 is greater than 40% and their difference is less than 10%, it is considered that the point cloud distribution of the original box is relatively uniform at this time and the lidar scans the outline of the trailer relatively completely. That is to say, this box is more suitable for the data augmentation method to simulate the situation of "the front of the vehicle blocking the trailer". Then, the sub-box set pyramid parts = {pyramid 3, pyramid 4} for the data augmentation scheme is required, and the operation in step S5 is repeated.

[0190] Step S7, add noise points to the entire original box. There are M points in the original box. Set a relatively small probability q with a value range of 0 - 0.3, and randomly generate some points randompoints = {p r1 , p r2 , …}, and the number of random points in this set is q * M. However, the coordinates (x r , y r , z r ) of any random noise point must be within the original box, that is, satisfy the following conditions:

[0191]

[0192] It should be noted that for the above steps S4 - S7, the data augmentation methods adopted in each sub - cube can be other reasonable existing methods. In this application, it is only illustrated for the specific container truck instance box scenario. The method of reasonably dividing an overall instance box for different data augmentations and the above - mentioned specific divided sub - cubes and data augmentation schemes for the empty - load vehicle hitch are within the protection scope of this application.

[0193] In summary, it can be understood that the application proposes to divide the empty - load container truck hitch instance box into different regions based on the point cloud data collected by lidar and apply different data augmentation schemes. That is, compared with the traditional method of taking the whole vehicle as the smallest unit and repeatedly performing multiple data augmentation schemes, on the basis of achieving the purpose of "simulating richer scenarios with limited labeled data", it can more realistically simulate the situations that occur when the container truck is moving, such as the front of the truck blocking the hitch when the truck turns, and the corresponding sub - cubes of different hitch instance boxes exchanging with each other to simulate more forms of hitches, etc. In addition to the above advantages, the scheme proposed in this application can integrate multiple data augmentation methods in one instance box, thereby reducing the redundancy in the traditional scheme of repeatedly putting the same box through different processes and then placing it in a new scenario due to the desire to use multiple data augmentation methods. The scheme proposed in this application can also reasonably divide the original box into different - shaped and - numbered sub - boxes according to the sparsity of the actual laser point cloud. In this way, operating in regions at the instance box level can more realistically simulate the situation where some regions of the container truck are blocked while some regions are scanned completely and clearly by the lidar. In the actual operation scenario, the situation where the hitch of the container truck is completely blocked is relatively rare, and in most cases, only a small part of the hitch is blocked. This application is exactly a solution for dealing with this majority of situations. On this basis, this application can also be combined with the existing data augmentation methods that do not divide the original instance box, increasing the flexibility of the scheme.

[0194] Optionally, another embodiment of this application also provides a point cloud data augmentation device. Please refer to Figure 9 , and this device mainly includes:

[0195] A first determination unit 101, configured to, after obtaining the point cloud data of a target vehicle from a point cloud dataset, determine an original vehicle hitch frame of the target vehicle according to the point cloud data, where the original vehicle hitch frame is an annotation frame of the target vehicle, and the target vehicle is a vehicle that needs to perform point cloud data augmentation processing in the point cloud dataset;

[0196] A splitting unit 102, configured to split the original vehicle hitch frame of the target vehicle to obtain vehicle hitch sub - frames of the target vehicle;

[0197] A second determination unit 103, configured to determine, according to the point cloud data distribution characteristics of the original vehicle hitch frame, vehicle hitch sub - frames that participate in applying a data augmentation scheme among all the vehicle hitch sub - frames.

[0198] Optionally, in some embodiments, the apparatus further includes:

[0199] A processing unit, configured to perform a first process on the point cloud data in the vehicle hitch sub - frames that participate in applying the data augmentation scheme and a second process on the point cloud data within the original vehicle hitch frame respectively, so as to simulate a point cloud scene of the target vehicle in an occluded state;

[0200] Wherein, the first process is randomly removing a certain proportion of point clouds, and the second process is adding noise points.

[0201] Optionally, in some embodiments, the splitting unit 102 is specifically configured to:

[0202] Determine a target splitting scheme of the target vehicle, where the target splitting scheme includes: a first splitting scheme and a second splitting scheme;

[0203] Split the original vehicle hitch frame according to the target splitting scheme to obtain vehicle hitch sub - frames of the target vehicle.

[0204] Optionally, in some embodiments, when the target splitting scheme is the first splitting scheme, the second determination unit 103 is specifically configured to:

[0205] Determine the carrying state of the target vehicle according to the point cloud data distribution characteristics of the original vehicle hitch frame, where the carrying state includes: an empty - load state and a full - load state;

[0206] Determine, according to the carrying state of the target vehicle, vehicle hitch sub - frames that participate in applying the data augmentation scheme among all the vehicle hitch sub - frames.

[0207] Optionally, in some embodiments, when performing the process of determining the carrying state of the target vehicle according to the point cloud data distribution characteristics of the original vehicle hitch frame, it is specifically configured to:

[0208] Respectively determine whether the point cloud data distribution characteristics of the original vehicle hitch frame conform to a preset empty - load state distribution characteristic, a preset full - load state distribution characteristic, or a preset turning state distribution characteristic;

[0209] If it is determined that the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset distribution characteristics in the no-load state, the carrying state of the target vehicle is determined to be the no-load state;

[0210] If it is determined that the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset distribution characteristics in the full-load state, the carrying state of the target vehicle is determined to be the full-load state.

[0211] Optionally, in some embodiments, determining whether the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset distribution characteristics in the no-load state includes:

[0212] Determine the number of point clouds of each of the vehicle hanging sub-frames of the target vehicle;

[0213] Judge whether the first point cloud ratio is greater than the preset no-load ratio. The first point cloud ratio is the ratio of the sum of the number of point cloud data in the vehicle hanging sub-frames located in the lower half of the original vehicle hanging frame to the number of all point cloud data in the original vehicle hanging frame;

[0214] If it is determined that the first point cloud ratio is greater than the preset no-load ratio, it is determined that the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset distribution characteristics in the no-load state;

[0215] If it is determined that the first point cloud ratio is not greater than the preset no-load ratio, it is determined that the distribution characteristics of the point cloud data of the original vehicle hanging frame do not conform to the preset distribution characteristics in the no-load state.

[0216] Optionally, in some embodiments, determining whether the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset distribution characteristics in the no-load state includes:

[0217] Determine the number of point clouds of each of the vehicle hanging sub-frames of the target vehicle;

[0218] Judge whether the second point cloud ratio is greater than the preset full-load ratio. The second point cloud ratio is the ratio of the sum of the number of point cloud data in the vehicle hanging sub-frames located in the upper half of the original vehicle hanging frame to the number of all point cloud data in the original vehicle hanging frame;

[0219] If it is determined that the second point cloud ratio is greater than the preset full-load ratio, it is determined that the distribution characteristics of the point cloud data of the original vehicle hanging frame do not conform to the preset distribution characteristics in the no-load state;

[0220] If it is determined that the second point cloud ratio is not greater than the preset full-load ratio, it is determined that the distribution characteristics of the point cloud data of the original vehicle hanging frame conform to the preset distribution characteristics in the no-load state.

[0221] Optionally, in some embodiments, when determining the vehicle hanging sub-frames participating in the application data augmentation scheme among all the vehicle hanging sub-frames according to the carrying state of the target vehicle, specifically:

[0222] If the carrying state of the target vehicle is an empty load state, determine the vehicle hanging sub-frame corresponding to the empty load state in the original vehicle hanging frame as the vehicle hanging sub-frame participating in the application data augmentation scheme;

[0223] If the carrying state of the target vehicle is a full load state, determine the vehicle hanging sub-frame corresponding to the full load state in the original vehicle hanging frame as the vehicle hanging sub-frame participating in the application data augmentation scheme.

[0224] Optionally, in some embodiments, the target splitting scheme is the second splitting scheme, and the second determination unit 103 is specifically configured to:

[0225] Judge whether the point cloud data distribution feature of the original vehicle hanging frame meets the preset vehicle hanging complete contour scanning feature. Meeting the preset vehicle hanging complete contour scanning feature indicates that the point cloud distribution in the original vehicle hanging frame is uniform and the vehicle hanging contour scanning is complete;

[0226] If it is judged that the point cloud data distribution feature of the original vehicle hanging frame meets the preset vehicle hanging complete contour scanning feature, determine the vehicle hanging sub-frame corresponding to the preset vehicle hanging complete contour scanning feature in the original vehicle hanging frame as the vehicle hanging sub-frame participating in the application data augmentation scheme.

[0227] Optionally, in some embodiments, judging whether the point cloud data distribution feature of the original vehicle hanging frame meets the preset vehicle hanging complete contour scanning feature includes:

[0228] Determine the point cloud quantity of each vehicle hanging sub-frame of the target vehicle;

[0229] Judge whether the third point cloud ratio is greater than the first preset ratio and whether the fourth point cloud ratio is less than the second preset ratio; the third point cloud ratio is the ratio of the sum of the point cloud data quantities of the first vehicle hanging sub-frame and the second vehicle hanging sub-frame to the total quantity of all point cloud data in the original vehicle hanging frame; the fourth point cloud ratio is the ratio of the difference between the point cloud data quantities of the first vehicle hanging sub-frame and the second vehicle hanging sub-frame to the total quantity of all point cloud data in the original vehicle hanging frame; the first vehicle hanging sub-frame and the second vehicle hanging sub-frame are respectively the vehicle hanging sub-frames located on the left and right sides of the original vehicle hanging frame;

[0230] If it is judged that the third point cloud ratio is greater than the first preset ratio and the fourth point cloud ratio is less than the second preset ratio, it is determined that the point cloud data distribution feature of the original vehicle hanging frame meets the preset vehicle hanging complete contour scanning feature;

[0231] If it is judged that the third point cloud ratio is not greater than the first preset ratio, and / or, the fourth point cloud ratio is not less than the second preset ratio, it is determined that the point cloud data distribution feature of the original vehicle hanging frame does not meet the preset vehicle hanging complete contour scanning feature;

[0232] Wherein, the first preset ratio is greater than the second preset ratio.

[0233] Optionally, in some embodiments, determining the point cloud quantity of each of the vehicle hanging sub - frames of the target vehicle includes:

[0234] Respectively determine the Euclidean distance between each point cloud data in the original vehicle hanging frame and the center points of each vehicle hanging sub - frame, where the center point of the vehicle hanging sub - frame is the geometric centroid of the vehicle hanging sub - frame;

[0235] For each point cloud data, select the vehicle hanging sub - frame with the minimum distance from the Euclidean distances between it and the center points of each vehicle hanging sub - frame as the vehicle hanging sub - frame to which the point cloud data belongs, and obtain the point cloud quantity of each vehicle hanging sub - frame of the target vehicle.

[0236] Based on the above, the point cloud data augmentation device provided in this embodiment includes: The first determination unit 101 is used to determine the original vehicle hanging frame of the target vehicle according to the point cloud data after obtaining the point cloud data of the target vehicle from the point cloud data set. The original vehicle hanging frame is the labeled frame of the target vehicle, and the target vehicle is the vehicle that needs to perform point cloud data augmentation processing in the point cloud data set; The splitting unit 102 is used to split the original vehicle hanging frame of the target vehicle to obtain the vehicle hanging sub - frames of the target vehicle; The second determination unit 103 is used to determine the vehicle hanging sub - frames participating in the application of the data augmentation scheme among all the vehicle hanging sub - frames according to the point cloud data distribution characteristics of the original vehicle hanging frame. That is, based on splitting the original vehicle hanging frame of the target vehicle into multiple vehicle hanging sub - frames, combined with the point cloud data distribution characteristics of the vehicle hanging sub - frames, the vehicle hanging sub - frames participating in the application of the data augmentation scheme are selected, which solves the problem that single operations are performed on some scenes or some instance frames in the scene for point cloud data augmentation and cannot achieve the expected effect of simulating more scenes.

[0237] It should be noted that the relevant descriptions of each unit of the point cloud data augmentation device can be seen in the corresponding above - mentioned embodiments, and will not be elaborated here one by one.

[0238] Optionally, another embodiment of the present application further provides a computer storage medium for storing a computer program, which when executed, is specifically used to implement the point cloud data augmentation method provided in any embodiment of the present application.

[0239] It should be noted that the relevant descriptions of the point cloud data augmentation method can be seen in the above - mentioned embodiments, and will not be elaborated here.

[0240] Optionally, another embodiment of the present application further provides an electronic device, which includes a memory and a processor.

[0241] Among them, the memory is used to store a computer program;

[0242] The processor is used to execute the computer program, and is specifically used to implement the point cloud data augmentation method provided in any embodiment of the present application.

[0243] It should be noted that the related description of the point cloud data augmentation method can also be referred to the above embodiments, and will not be repeated here.

[0244] The features described in each of the embodiments in this specification can be replaced or combined with each other. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the related parts, reference can be made to the partial description of the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0245] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0246] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0247] It should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

Claims

1. A method for augmenting point cloud data, characterized in that, Including: After obtaining the point cloud data of the target vehicle from the point cloud dataset, determining the original vehicle hanging frame of the target vehicle according to the point cloud data, where the original vehicle hanging frame is the annotation frame of the target vehicle, and the target vehicle is the vehicle in the point cloud dataset that needs to perform point cloud data augmentation processing; Splitting the original vehicle hanging frame of the target vehicle to obtain the vehicle hanging sub-frames of the target vehicle; Determining the vehicle hanging sub-frames participating in the application of the data augmentation scheme among all the vehicle hanging sub-frames according to the point cloud data distribution characteristics of the original vehicle hanging frame.

2. The method for augmenting point cloud data according to claim 1, characterized in that, After determining the vehicle hanging sub-frames participating in the application of the data augmentation scheme among all the vehicle hanging sub-frames according to the point cloud data distribution characteristics of the original vehicle hanging frame, it further includes: Performing a first process on the point cloud data in the vehicle hanging sub-frames participating in the application of the data augmentation scheme and a second process on the point cloud data within the original vehicle hanging frame respectively to simulate the point cloud scene of the target vehicle in the occluded state; Wherein, the first process is randomly removing a certain proportion of point clouds, and the second process is adding noise points.

3. The method for augmenting point cloud data according to claim 1, characterized in that, Splitting the original vehicle hanging frame to obtain the vehicle hanging sub-frames of the target vehicle includes: Determining the target splitting scheme of the target vehicle, where the target splitting scheme includes: a first splitting scheme and a second splitting scheme; Splitting the original vehicle hanging frame according to the target splitting scheme to obtain the vehicle hanging sub-frames of the target vehicle.

4. The method for augmenting point cloud data according to claim 3, characterized in that, When the target splitting scheme is the first splitting scheme, determining the vehicle hanging sub-frames participating in the application of the data augmentation scheme among all the vehicle hanging sub-frames according to the point cloud data distribution characteristics of the original vehicle hanging frame includes: Determining the carrying state of the target vehicle according to the point cloud data distribution characteristics of the original vehicle hanging frame, where the carrying state includes: an empty load state and a full load state; Determining the vehicle hanging sub-frames participating in the application of the data augmentation scheme among all the vehicle hanging sub-frames according to the carrying state of the target vehicle.

5. The method for augmenting point cloud data according to claim 4, characterized in that, Determining the carrying state of the target vehicle according to the point cloud data distribution characteristics of the original vehicle hanging frame includes: Judging whether the point cloud data distribution characteristics of the original vehicle hanging frame conform to the preset empty load state distribution characteristics; If it is judged that the point cloud data distribution characteristics of the original vehicle hanging frame conform to the preset empty load state distribution characteristics, then determining the carrying state of the target vehicle as the empty load state; If it is judged that the point cloud data distribution characteristics of the original vehicle hanging frame do not conform to the preset empty load state distribution characteristics, then determining the carrying state of the target vehicle as the full load state.

6. The method for augmenting point cloud data according to claim 5, characterized in that, Judging whether the point cloud data distribution characteristics of the original vehicle hanging frame conform to the preset empty load state distribution characteristics includes: Determining the number of point clouds of each vehicle hanging sub-frame of the target vehicle; Judging whether the first point cloud ratio is greater than the preset empty load ratio, where the first point cloud ratio is the ratio of the sum of the number of point cloud data in the vehicle hanging sub-frames located in the lower half of the original vehicle hanging frame to the number of all point cloud data within the original vehicle hanging frame; If it is judged that the first point cloud ratio is greater than the preset empty load ratio, then determining that the point cloud data distribution characteristics of the original vehicle hanging frame conform to the preset empty load state distribution characteristics; If it is determined that the first point cloud ratio is not greater than the preset no-load ratio, it is determined that the point cloud data distribution feature of the original car hanging frame does not conform to the preset no-load state distribution feature.

7. The method for augmenting point cloud data according to claim 5, characterized in that, Determining whether the point cloud data distribution feature of the original car hanging frame conforms to the preset no-load state distribution feature includes: Determining the number of point clouds of each of the car hanging sub-frames of the target vehicle; Judging whether the second point cloud ratio is greater than the preset full-load ratio, where the second point cloud ratio is the ratio of the sum of the number of point cloud data in each of the car hanging sub-frames in the upper half of the original car hanging frame to the total number of point cloud data in the original car hanging frame; If it is determined that the second point cloud ratio is greater than the preset full-load ratio, it is determined that the point cloud data distribution feature of the original car hanging frame does not conform to the preset no-load state distribution feature; If it is determined that the second point cloud ratio is not greater than the preset full-load ratio, it is determined that the point cloud data distribution feature of the original car hanging frame conforms to the preset no-load state distribution feature.

8. The method for augmenting point cloud data according to claim 4, characterized in that, According to the carrying state of the target vehicle, determining the car hanging sub-frames participating in the application data augmentation scheme among all the car hanging sub-frames includes: If the carrying state of the target vehicle is the no-load state, determining the car hanging sub-frames corresponding to the no-load state in the original car hanging frame as the car hanging sub-frames participating in the application data augmentation scheme; If the carrying state of the target vehicle is the full-load state, determining the car hanging sub-frames corresponding to the full-load state in the original car hanging frame as the car hanging sub-frames participating in the application data augmentation scheme.

9. The method for augmenting point cloud data according to claim 3, wherein, The target splitting scheme is the second splitting scheme. According to the point cloud data distribution feature of the original car hanging frame, determining the car hanging sub-frames participating in the application data augmentation scheme among all the car hanging sub-frames includes: Judging whether the point cloud data distribution feature of the original car hanging frame satisfies the preset car hanging complete contour scanning feature. Satisfying the preset car hanging complete contour scanning feature indicates that the point cloud distribution in the original car hanging frame is uniform and the car hanging contour scanning is complete; If it is determined that the point cloud data distribution feature of the original car hanging frame satisfies the preset car hanging complete contour scanning feature, determining the car hanging sub-frames corresponding to the preset car hanging complete contour scanning feature in the original car hanging frame as the car hanging sub-frames participating in the application data augmentation scheme.

10. A device for augmenting point cloud data, wherein, Including: A first determination unit, configured to, after obtaining the point cloud data of the target vehicle from the point cloud dataset, determine the original car hanging frame of the target vehicle according to the point cloud data. The original car hanging frame is the annotation frame of the target vehicle, and the target vehicle is the vehicle in the point cloud dataset that needs to perform point cloud data augmentation processing; A splitting unit, configured to split the original car hanging frame of the target vehicle to obtain the car hanging sub-frames of the target vehicle; A second determination unit, configured to determine the car hanging sub-frames participating in the application data augmentation scheme among all the car hanging sub-frames according to the point cloud data distribution feature of the original car hanging frame.

Images