Point cloud data enhancement method and device and terminal equipment

Abstract

The invention is suitable for the technical field of data processing, and provides a point cloud data enhancement method and device, and terminal equipment, and the method comprises the steps: obtaining an initial point cloud data set and a background point cloud corresponding to a to-be-enhanced scene; extracting a point cloud region corresponding to each target from each piece of initial point cloud data according to annotation data corresponding to each piece of initial point cloud data in the initial point cloud data set; placing the point cloud region corresponding to the at least one target in the background point cloud to generate a plurality of pieces of enhanced point cloud data; and adding each piece of enhanced point cloud data into the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced. Therefore, the point cloud data of the existing target in the initial point cloud data set is fused with the background point cloud of the scene to be enhanced, so that the training data of the new scene can be obtained without field acquisition and manual labeling, the difficulty of training data acquisition is reduced, the acquisition cost is saved, and the generalization ability of the training data set is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a point cloud data enhancement method, device, terminal equipment and computer-readable storage medium. Background Art

[0002] By installing equipment such as radars and cameras on the roadside, it is possible to perceive the intersection scene and realize intelligent management of the intersection. Since radar equipment has the advantages of being stable and not affected by weather, the perception results of radar equipment are usually used as the perception subject. For example, using a deep learning-based target detection algorithm to detect the point cloud data scanned by the radar device, information such as the location and category of the target in the scene can be obtained. The target detection algorithm currently used is data-driven, and the detection effect mainly depends on the training set data.

[0003] In the related art, the training data set is usually generated by field collection and manual annotation. However, due to the large differences between scene data of different intersections and the limited types of actual scenes, the training data collected and annotated by the field are difficult to generalize to all scenes. Therefore, the method of generating training data sets by field collection and manual annotation is not only complicated, but also time-consuming and labor-intensive, and the generalization ability of the training data sets is poor. Summary of the invention

[0004] The embodiments of the present application provide a point cloud data enhancement method, apparatus, terminal device and storage medium, which can solve the problem that the method of generating training data sets through field collection and manual annotation is not only complicated in process, but also has high time and labor costs, and also results in poor generalization ability of the training data sets.

[0005] In a first aspect, an embodiment of the present application provides a point cloud data enhancement method, comprising: obtaining an initial point cloud data set, wherein the initial point cloud data set includes multiple initial point cloud data and annotation data corresponding to each initial point cloud data, the annotation data including annotation types and point cloud areas corresponding to each target included in the initial point cloud data; obtaining a background point cloud corresponding to a scene to be enhanced; extracting a point cloud area corresponding to each target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data; placing a point cloud area corresponding to at least one target in the background point cloud to generate multiple enhanced point cloud data; and adding each enhanced point cloud data to the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced.

[0006] In a possible implementation manner of the first aspect, before placing the point cloud area corresponding to the at least one target in the background point cloud to generate a plurality of enhanced point cloud data, the method further includes:

[0007] Determine the area to be enhanced in the background point cloud and the type of target to be enhanced corresponding to the area to be enhanced;

[0008] Accordingly, the above-mentioned placing of the point cloud area corresponding to at least one target in the background point cloud to generate a plurality of enhanced point cloud data includes:

[0009] According to the types of targets to be enhanced and the annotation types corresponding to the targets, a point cloud region corresponding to at least one target is placed in the region to be enhanced in the background point cloud to generate a plurality of enhanced point cloud data.

[0010] Optionally, in another possible implementation manner of the first aspect, the obtaining of a background point cloud corresponding to the scene to be enhanced includes:

[0011] Performing background extraction on each initial point cloud data to determine each background point cloud included in the initial point cloud data set;

[0012] Each background point cloud included in the initial point cloud data set is determined as the background point cloud corresponding to the scene to be enhanced.

[0013] Optionally, in another possible implementation of the first aspect, before extracting the point cloud area corresponding to each target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data, the method further includes:

[0014] Performing data analysis on the initial point cloud data set according to a preset indicator to determine a data distribution ratio of a first type of target in the initial point cloud data set under the preset indicator, wherein the first type of target is a target of any annotated type in the initial point cloud data set;

[0015] According to the data distribution ratio, determine the extraction ratio of the first type of target under the preset indicators;

[0016] Correspondingly, the above-mentioned extraction of point cloud regions corresponding to each target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data includes:

[0017] According to the annotation data corresponding to each initial point cloud data and the extraction ratio of the first type of target under the preset index, the point cloud area corresponding to the first type of target is extracted from each initial point cloud data.

[0018] Optionally, in another possible implementation of the first aspect, the extraction ratio of the first type of target under the preset indicator is negatively correlated with the data distribution ratio of the first type of target under the preset indicator.

[0019] Optionally, in another possible implementation of the first aspect, the above-mentioned initial point cloud data is collected by a radar device; accordingly, the above-mentioned preset indicators include at least one of the following indicators: the distance between the target and the radar device, the number of point clouds on the target surface, and the angle of the target.

[0020] Optionally, in another possible implementation of the first aspect, when the above-mentioned preset indicator includes the distance between the target and the radar device, the extraction ratio of the above-mentioned first type of target under the preset indicator is positively correlated with the distance between the first type of target and the radar device.

[0021] Optionally, in another possible implementation manner of the first aspect, the preset indicator includes a distance between the target and the radar device; accordingly, the performing data analysis on the initial point cloud dataset according to the preset indicator to determine a data distribution ratio of the first type of targets in the initial point cloud dataset under the preset indicator includes:

[0022] Determining the distance range to which each first-type target belongs according to the distance between each first-type target and the radar device;

[0023] The data distribution ratio is determined according to the number of first type targets corresponding to each distance range.

[0024] Optionally, in yet another possible implementation manner of the first aspect, the preset indicator includes the number of point clouds on the target surface; accordingly, the performing data analysis on the initial point cloud dataset according to the preset indicator to determine the data distribution ratio of the first type of targets in the initial point cloud dataset under the preset indicator includes:

[0025] Taking the straight line where the heading angle of each first-type target is located as the axis, each first-type target is divided into a first part and a second part;

[0026] Determine the distribution ratio of the number of surface point clouds corresponding to each first type target according to the number of point clouds included in the first part and the number of point clouds included in the second part of each first type target;

[0027] Determine the proportion range to which each first-type target belongs according to the distribution proportion of the number of surface point clouds corresponding to each first-type target;

[0028] The data distribution ratio is determined according to the number of first type targets corresponding to each ratio range.

[0029] Optionally, in yet another possible implementation manner of the first aspect, the preset indicator includes an angle of the target; accordingly, the performing data analysis on the initial point cloud dataset according to the preset indicator to determine a data distribution ratio of the first type of targets in the initial point cloud dataset under the preset indicator includes:

[0030] According to the angle of each first-type target, determine the angle range to which each first-type target belongs;

[0031] The data distribution ratio is determined according to the number of first type targets corresponding to each angle range.

[0032] Optionally, in another possible implementation manner of the first aspect, placing the point cloud area corresponding to the at least one target in the background point cloud to generate a plurality of enhanced point cloud data includes:

[0033] According to the scene type corresponding to the scene to be enhanced, determine the target insertion number N corresponding to the background point cloud, where N is a positive integer;

[0034] The point cloud areas corresponding to the N targets are placed in the background point cloud to generate multiple enhanced point cloud data.

[0035] Optionally, in another possible implementation manner of the first aspect, placing the point cloud area corresponding to the at least one target in the background point cloud to generate a plurality of enhanced point cloud data includes:

[0036] The point cloud areas corresponding to each target are placed in the background point cloud in sequence so that the point cloud areas corresponding to each target do not overlap.

[0037] In a second aspect, an embodiment of the present application provides a point cloud data enhancement device, comprising: a first acquisition module, used to acquire an initial point cloud data set, wherein the initial point cloud data set includes multiple initial point cloud data and annotation data corresponding to each initial point cloud data, and the annotation data includes annotation types and point cloud areas corresponding to each target included in the initial point cloud data; a second acquisition module, used to acquire a background point cloud corresponding to a scene to be enhanced; a first extraction module, used to extract point cloud areas corresponding to each target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data; a first placement module, used to place a point cloud area corresponding to at least one target in the background point cloud to generate multiple enhanced point cloud data; a first generation module, used to add each enhanced point cloud data to the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced.

[0038] In a possible implementation manner of the second aspect, the apparatus further includes:

[0039] The first determination module is used to determine the area to be enhanced in the background point cloud and the type of the target to be enhanced corresponding to the area to be enhanced;

[0040] Accordingly, the first placement module includes:

[0041] The first placement unit is used to place a point cloud area corresponding to at least one target in the area to be enhanced in the background point cloud according to the type of the target to be enhanced and the annotation type corresponding to each target, so as to generate multiple enhanced point cloud data.

[0042] Optionally, in another possible implementation of the second aspect, the second acquisition module includes:

[0043] A first determining unit is used to perform background extraction on each initial point cloud data set to determine each background point cloud included in the initial point cloud data set;

[0044] The second determining unit is used to determine each background point cloud included in the initial point cloud data set as a background point cloud corresponding to the scene to be enhanced.

[0045] Optionally, in another possible implementation of the second aspect, the apparatus further includes:

[0046] A second determination module is used to perform data analysis on the initial point cloud data set according to a preset indicator to determine a data distribution ratio of a first type of target in the initial point cloud data set under the preset indicator, wherein the first type of target is a target of any annotated type in the initial point cloud data set;

[0047] A third determination module is used to determine the extraction ratio of the first type of target under a preset indicator according to the data distribution ratio;

[0048] Accordingly, the first extraction module includes:

[0049] The first extraction unit is used to extract a point cloud area corresponding to the first type of target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data and the extraction ratio of the first type of target under a preset indicator.

[0050] Optionally, in another possible implementation of the second aspect, the extraction ratio of the above-mentioned first type of targets under the preset indicators is negatively correlated with the data distribution ratio of the first type of targets under the preset indicators.

[0051] Optionally, in another possible implementation of the second aspect, the above-mentioned initial point cloud data is collected by a radar device; accordingly, the above-mentioned preset indicators include at least one of the following indicators: the distance between the target and the radar device, the number of point clouds on the target surface, and the angle of the target.

[0052] Optionally, in another possible implementation of the second aspect, when the above-mentioned preset indicator includes the distance between the target and the radar device, the extraction ratio of the above-mentioned first type of target under the preset indicator is positively correlated with the distance between the first type of target and the radar device.

[0053] Optionally, in another possible implementation manner of the second aspect, the preset indicator includes a distance between the target and the radar device; accordingly, the second determination module includes:

[0054] A third determining unit, configured to determine a distance range to which each first-type target belongs according to a distance between each first-type target and the radar device;

[0055] The fourth determining unit is used to determine the data distribution ratio according to the number of first type targets corresponding to each distance range.

[0056] Optionally, in yet another possible implementation manner of the second aspect, the preset indicator includes the number of point clouds of the target surface; accordingly, the second determination module includes:

[0057] A first division unit is used to divide each first type target into a first part and a second part respectively, with the straight line where the heading angle of each first type target is located as an axis;

[0058] a fifth determining unit, configured to determine a distribution ratio of the number of surface point clouds corresponding to each first type target, according to the number of point clouds included in the first part and the number of point clouds included in the second part of each first type target;

[0059] A sixth determining unit, configured to determine a proportion range to which each first-type target belongs according to a distribution ratio of the number of surface point clouds corresponding to each first-type target;

[0060] The seventh determining unit is used to determine the data distribution ratio according to the number of first type targets corresponding to each ratio range.

[0061] Optionally, in yet another possible implementation manner of the second aspect, the preset indicator includes an angle of the target; accordingly, the second determination module includes:

[0062] An eighth determining unit, configured to determine, according to the angle of each first-type target, an angle range to which each first-type target belongs;

[0063] The ninth determining unit is used to determine the data distribution ratio according to the number of first type targets corresponding to each angle range.

[0064] Optionally, in another possible implementation of the second aspect, the first placement module includes:

[0065] a tenth determining unit, configured to determine, according to a scene type corresponding to the scene to be enhanced, a target insertion number N corresponding to the background point cloud, where N is a positive integer;

[0066] The second placement unit is used to place the point cloud areas corresponding to the N targets in the background point cloud to generate a plurality of enhanced point cloud data.

[0067] Optionally, in another possible implementation of the second aspect, the first placement module includes:

[0068] The third placement unit is used to place the point cloud areas corresponding to the respective targets in the background point cloud in sequence, so that the point cloud areas corresponding to the respective targets do not overlap.

[0069] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the point cloud data enhancement method as described above when executing the computer program.

[0070] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the point cloud data enhancement method as described above.

[0071] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the point cloud data enhancement method as described above.

[0072] Compared with the prior art, the beneficial effect of the embodiments of the present application is that by fusing the point cloud data of the existing targets in the initial point cloud data set with the background point cloud of the scene to be enhanced to generate enhanced point cloud data of the scene to be enhanced, training data of the new scene can be obtained without field collection and manual labeling, thereby not only reducing the difficulty of training data collection, saving time and labor costs, but also improving the generalization ability of the training data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0074] Figure 1 It is a flowchart of a point cloud data enhancement method provided in an embodiment of the present application;

[0075] Figure 2 is a flow chart of a point cloud data enhancement method provided by another embodiment of the present application;

[0076] Figure 3is a schematic diagram of the structure of a point cloud data enhancement device provided in an embodiment of the present application;

[0077] Figure 4 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0079] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0080] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0081] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0082] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0083] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0084] The point cloud data enhancement method, apparatus, terminal device, storage medium and computer program provided in the present application are described in detail below with reference to the accompanying drawings.

[0085] Figure 1 A flow chart of a point cloud data enhancement method provided in an embodiment of the present application is shown.

[0086] like Figure 1 As shown, the point cloud data enhancement method includes the following steps:

[0087] Step 101, obtaining an initial point cloud data set, wherein the initial point cloud data set includes a plurality of initial point cloud data and annotation data corresponding to each initial point cloud data, and the annotation data includes annotation types and point cloud areas corresponding to each target included in the initial point cloud data.

[0088] It should be noted that the point cloud data enhancement method of the embodiment of the present application can be executed by the point cloud data enhancement device of the embodiment of the present application. The point cloud data enhancement device of the embodiment of the present application can be configured in any terminal device to execute the point cloud data enhancement method of the embodiment of the present application.

[0089] Among them, the initial point cloud data set can refer to a data set composed of actual point cloud data collected by radar equipment deployed in various actual traffic scenes; it can also be a data set composed of actual point cloud data collected by radar equipment deployed in various actual traffic scenes and some enhanced point cloud data.

[0090] It should be noted that when the point cloud data collected by the radar equipment is perceived by a deep learning-based perception algorithm (such as a target detection algorithm) (such as detecting the location and category of the target contained in the point cloud data, etc.), the perception algorithm needs to be trained to ensure that the performance of the perception algorithm meets the requirements. Therefore, a large amount of point cloud data can be collected according to the application scenario corresponding to the perception algorithm to form a training data set to train the perception algorithm. However, when generating a training data set by field collection and manual annotation, due to the large differences between different traffic scene data and the limited types of actual scenes, the field-collected and annotated training data is difficult to generalize to all scenes, and the collection cost is high. Therefore, the point cloud data enhancement method of the embodiment of the present application can be used to enhance the training data set collected in the field to reduce the difficulty of training data collection, save time and labor costs, and improve the generalization ability of the training data set. Therefore, the initial point cloud dataset in the embodiment of the present application can be a training dataset for training the perception algorithm, that is, it can be a dataset composed of actual point cloud data collected by the radar device; in addition, when it is necessary to use the point cloud data enhancement method of the embodiment of the present application to enhance the training dataset multiple times, the initial point cloud dataset of the embodiment of the present application can be an enhanced dataset generated after the last data enhancement, that is, the initial point cloud dataset can include the actual point cloud data collected by the radar device, and the enhanced point cloud data generated after the previous data enhancement, and the embodiment of the present application is not limited to this.

[0091] It should be noted that the actual traffic scene for deploying radar equipment mentioned above can be determined according to the specific application scenario, and the embodiments of the present application do not limit this. For example, when the initial point cloud data set of the embodiments of the present application is used to train the target detection algorithm for the intersection scene, the actual traffic scene for deploying radar equipment can be various traffic intersections.

[0092] The initial point cloud data may refer to each point cloud data included in the initial point cloud data set. It should be noted that when the initial point cloud data set is used to train the target detection algorithm, the initial point cloud data may include at least one target, and the type of each target included in the initial point cloud data and the corresponding point cloud area may be annotated.

[0093] The type of annotation corresponding to the target may be related to the actual application scenario, and the present application embodiment does not limit this. For example, in the target detection scenario at the intersection, pedestrians, non-motor vehicles, and motor vehicles usually appear at the intersection. Therefore, the type of annotation corresponding to the target may include pedestrians, non-motor vehicles, and motor vehicles.

[0094] The point cloud area corresponding to the target may refer to the area of ​​the point cloud corresponding to the target in the initial point cloud data. For example, the detection frame corresponding to each target may be marked in the initial point cloud data, and the point cloud area within the detection frame is the point cloud area corresponding to the target.

[0095] It should be noted that when the point cloud area corresponding to the target is marked by the detection frame, the type of the detection frame can be a regular shape such as a rectangle or a circle, or any irregular shape, so as to more accurately represent the point cloud area corresponding to each target. In actual use, the marking method of the point cloud area corresponding to the target can be selected according to actual needs and specific application scenarios, and the embodiments of the present application do not limit this.

[0096] In an embodiment of the present application, radar equipment can be deployed in the corresponding traffic scene before and after the perception algorithm is applied to collect point cloud data corresponding to the corresponding traffic scene in real time, and the point cloud data collected for a long time (such as one day, one week, one month, one year, etc.) is used as the initial point cloud data, and the annotation type and point cloud area corresponding to each target contained in the initial point cloud data are manually annotated to generate an initial point cloud data set.

[0097] It should be noted that when data enhancement is not performed on a certain data set for the first time, the enhanced data set generated by the previous data enhancement can be used as the initial point cloud data set.

[0098] Step 102: Obtain a background point cloud corresponding to the scene to be enhanced.

[0099] The scene to be enhanced may refer to any scene that currently needs to generate new point cloud data. In actual use, the scene to be enhanced may be a new scene not included in the initial data set, so as to expand the point cloud data corresponding to the new scene in the initial point cloud data set according to actual use requirements; the scene to be enhanced may also be a scene included in the initial point cloud data set, so as to expand the point cloud data corresponding to one or more scenes in the initial point cloud data set.

[0100] For example, in the target detection scenario of the intersection scene, if radar equipment has been deployed at intersection A and intersection B, and the initial point cloud data set already contains the actual point cloud data collected by the radar equipment at intersection A and intersection B, and it is currently necessary to realize target detection at the new intersection C, then intersection C can be determined as the scene to be enhanced; for another example, assuming that in the above example, intersection A is located in a remote area with a small flow of people, so the number of targets contained in the point cloud data collected at intersection A is very small, or even contains a large amount of point cloud data that does not contain targets, so it is impossible to obtain enough point cloud data at intersection A to train the target detection algorithm, resulting in target The detection algorithm has poor reliability in target detection at intersection A. Therefore, intersection A can be used as a scene to be enhanced to expand the point cloud data corresponding to intersection A. For example, since the installation position of the radar equipment at the intersection is fixed, and the distribution of the targets collected by the radar equipment is usually related to the installation position of the radar equipment, the point cloud data corresponding to different types of targets collected by the radar equipment at each intersection may be unevenly distributed. Therefore, intersection A and intersection B can also be used as scenes to be enhanced to enhance the data of each scene involved in the initial point cloud data set so that the point cloud data of each scene is evenly distributed.

[0101] The background point cloud may refer to point cloud data corresponding to the scene to be enhanced that does not contain any target.

[0102] For example, when the scene to be enhanced is intersection C in the above example, the point cloud data corresponding to intersection C can be collected as the background point cloud corresponding to intersection C when no target (pedestrians, non-motor vehicles, motor vehicles, etc.) appears at intersection C; or, the point cloud data corresponding to intersection C can be collected at any time, and the point cloud data corresponding to each target in the point cloud data can be removed to generate the background point cloud corresponding to intersection C. For another example, when the scene to be enhanced is intersection A in the above example, any initial point cloud data corresponding to intersection A can be obtained from the initial point cloud data set, and the point cloud data corresponding to each target in the initial point cloud data can be removed to generate the background point cloud corresponding to intersection A.

[0103] In the embodiment of the present application, the scene to be enhanced can be determined and the background point cloud corresponding to the scene to be enhanced can be obtained according to the actual data enhancement requirements. In actual use, the scene to be enhanced can be one or more, and the embodiment of the present application does not limit this.

[0104] Step 103 , extracting point cloud regions corresponding to each target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data.

[0105] In an embodiment of the present application, after obtaining the background point cloud corresponding to the scene to be enhanced, the target fused into the background point cloud can be obtained from the initial point cloud data set, so as to generate enhanced point cloud data after fusing the target and background point cloud data.

[0106] As a possible implementation method, for an initial point cloud data, the point cloud area corresponding to each target can be determined according to the annotation data corresponding to each target contained in the initial point cloud data, and the point cloud area corresponding to each target can be extracted from the initial point cloud data, that is, the point cloud contained in the point cloud area corresponding to each target is extracted from the initial point cloud data to prepare data for the subsequent fusion process. By analogy, the same extraction process can be performed on each initial point cloud data set in the initial point cloud data set to extract the point cloud area corresponding to each target contained in the initial point cloud data set.

[0107] As a possible implementation method, the extraction ratio of each type of target can be determined according to the scene to be enhanced, and the point cloud area corresponding to each type of target can be extracted from the initial point cloud data set according to the extraction ratio, so that the target distribution in the enhanced point cloud data finally generated matches the actual traffic status of the scene to be enhanced.

[0108] For example, assuming that in the target detection scene at the intersection, the target categories include motor vehicles, non-motor vehicles, and pedestrians, and the scene to be enhanced is a traffic scene where the probability of motor vehicles appearing is high, and the probability of non-motor vehicles and pedestrians appearing is low, then the extraction ratio of motor vehicles can be made greater than the extraction ratio of non-motor vehicles and pedestrians, so as to maximize the recognition accuracy of motor vehicles by the target detection model in the scene to be enhanced. For example, the extraction ratio of motor vehicles can be determined to be 50%, and the extraction ratios of non-motor vehicles and pedestrians can be determined to be 25% each.

[0109] Step 104 : placing a point cloud area corresponding to at least one target in the background point cloud to generate a plurality of enhanced point cloud data.

[0110] In the embodiment of the present application, after determining the background point cloud corresponding to the scene to be enhanced and extracting the point cloud areas corresponding to each target from the initial point cloud data set, one or more point cloud areas corresponding to each target can be randomly selected from the point cloud areas corresponding to each target and fused into the background point cloud corresponding to the scene to be enhanced to generate an enhanced point cloud data containing one or more targets; after repeating this process multiple times, multiple enhanced point cloud data corresponding to the scene to be enhanced can be generated. If there are multiple scenes to be enhanced, the background point cloud corresponding to each scene to be enhanced can be processed in turn in the same manner as described above to generate enhanced point cloud data corresponding to each scene to be enhanced.

[0111] It should be noted that, after each time the point cloud area corresponding to the selected target is placed in the background point cloud corresponding to the scene to be enhanced, the selected target can be removed to ensure that the targets in the generated enhanced point cloud data are not repeated; or, the selected target may not be removed, and each time the enhanced point cloud data is generated, a target is randomly selected from all the targets extracted in step 103 and fused into the background point cloud.

[0112] It should be noted that in actual use, the amount of enhanced point cloud data generated can be determined based on actual needs and specific application scenarios, and the embodiments of the present application do not limit this.

[0113] Furthermore, in order to ensure that the various targets in the enhanced point cloud data do not overlap, when multiple targets need to be placed in the background point cloud, the various targets can also be placed in order. That is, in a possible implementation of the embodiment of the present application, the above step 104 may include:

[0114] The point cloud areas corresponding to each target are placed in the background point cloud in sequence so that the point cloud areas corresponding to each target do not overlap.

[0115] As a possible implementation method, when multiple targets need to be placed in the background point cloud, if the point cloud areas corresponding to the multiple targets are placed in the background point cloud at the same time, the point cloud areas corresponding to the various targets are prone to overlap, thereby affecting the quality of the enhanced point cloud data. Therefore, when multiple targets need to be placed in the background point cloud, the point cloud areas corresponding to the various targets can be placed in the background point cloud in order, that is, after the point cloud area corresponding to one target is merged with the background point cloud, the point cloud area corresponding to the next target is placed. Therefore, when placing the point cloud area corresponding to the target, the position of the point cloud area corresponding to the previous target in the background point cloud can be referred to, so that the point cloud areas corresponding to the various targets do not overlap in the background point cloud, so as to ensure the quality of the generated enhanced point cloud data.

[0116] Furthermore, the number of targets placed in the background point cloud corresponding to the scene to be enhanced, that is, the number of targets included in the enhanced point cloud data, can be set according to the scene type of the scene to be enhanced, so as to further ensure that the generated enhanced point cloud data matches the actual operating state of the scene to be enhanced. That is, in a possible implementation of the embodiment of the present application, the above step 104 may include:

[0117] According to the scene type corresponding to the scene to be enhanced, determine the target insertion number N corresponding to the background point cloud, where N is a positive integer;

[0118] The point cloud areas corresponding to the N targets are placed in the background point cloud to generate multiple enhanced point cloud data.

[0119] As a possible implementation method, the number of target insertions N corresponding to the background point cloud, that is, the number of targets placed in the background point cloud, can be determined according to the scene type corresponding to the scene to be enhanced; then, point cloud areas corresponding to N targets can be randomly selected from the point cloud areas corresponding to each target that has been extracted and placed in the background point cloud to generate an enhanced point cloud data corresponding to the scene to be enhanced; and so on, point cloud areas corresponding to N targets are randomly selected from the point cloud areas corresponding to each target and placed in the background point cloud to generate multiple enhanced point cloud data corresponding to the scene to be enhanced.

[0120] For example, if the scene to be enhanced is a traffic scene with less pedestrian flow, the target insertion number N can be determined as a smaller value, such as 1, 2, 3, etc.; if the scene to be enhanced is a traffic scene with more pedestrian flow, the target insertion number N can be determined as a larger value, such as 5, 10, etc.

[0121] It should be noted that the above examples are only exemplary and cannot be regarded as limiting the present application. In actual use, the setting rule of the target insertion number N can be determined according to actual needs and specific application scenarios, and the present application embodiment does not limit this.

[0122] Furthermore, since in the scene to be enhanced, a specific type of target usually appears in a specific area, the placement area of ​​each type of target in the background point cloud can also be predetermined so that the generated enhanced point cloud data matches the actual traffic state of the scene to be enhanced. That is, in a possible implementation of the embodiment of the present application, before the above step 104, it can also include:

[0123] Determine the area to be enhanced in the background point cloud and the type of target to be enhanced corresponding to the area to be enhanced;

[0124] Accordingly, the above step 104 may include:

[0125] According to the types of targets to be enhanced and the annotation types corresponding to the targets, a point cloud region corresponding to at least one target is placed in the region to be enhanced in the background point cloud to generate a plurality of enhanced point cloud data.

[0126] The area to be enhanced may refer to the area where the target needs to be placed in the background point cloud corresponding to the scene to be enhanced. For example, when the scene to be enhanced is an intersection, the area to be enhanced may include a zebra crossing area, an area behind a parking line, and so on. In actual use, the area to be enhanced is related to the actual traffic status of the scene to be enhanced and the detection purpose of the target detection algorithm, and the embodiments of the present application do not limit this.

[0127] The target type to be enhanced may refer to the target type to be placed in the area to be enhanced. For example, when the area to be enhanced is a zebra crossing area, the target type to be enhanced corresponding to the area to be enhanced may be a pedestrian; when the area to be enhanced is an area behind a parking line, the target type to be enhanced corresponding to the area to be enhanced may be a motor vehicle.

[0128] In an embodiment of the present application, the area to be enhanced in the background point cloud and the target type to be enhanced corresponding to each area to be enhanced can be determined according to the actual traffic status of the scene to be enhanced and the data enhancement requirements. Afterwards, for an area to be enhanced, one or more targets whose annotation types match the target type to be enhanced can be randomly selected and placed in the area to be enhanced according to the target type to be enhanced corresponding to the area to be enhanced and the annotation type corresponding to each target extracted in the aforementioned steps; and by analogy, when there are multiple areas to be enhanced, the same method as above can be used to determine to place one or more targets in each area to be enhanced to generate an enhanced point cloud data. Moreover, by repeating the above process, multiple enhanced point cloud data corresponding to the scene to be enhanced can be generated.

[0129] For example, if the type of target to be enhanced corresponding to a region to be enhanced in the background point cloud is a pedestrian, one or more point cloud regions corresponding to pedestrians may be randomly selected from various targets and placed in the region to be enhanced.

[0130] It should be noted that when generating enhanced point cloud data by placing targets whose annotation types match the target types to be enhanced in the area to be enhanced, each target can also be placed in sequence in the manner disclosed in the aforementioned embodiment to prevent overlapping of targets in the enhanced point cloud data; and, the number of target insertions corresponding to each area to be enhanced can also be determined according to the method for determining the number of target insertions disclosed in the aforementioned embodiment, and a corresponding number of targets can be placed in each area to be enhanced according to the number of target insertions corresponding to each area to be enhanced.

[0131] Step 105 , adding each enhanced point cloud data to the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced.

[0132] In the embodiment of the present application, after generating multiple enhanced point cloud data corresponding to the scene to be enhanced, you can add each enhanced point cloud data to the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced. If the enhanced point cloud data set is used for model training later, the model generated after training can obtain better perception ability in the scene to be enhanced, thereby realizing the generalization of the data set to the scene to be enhanced without field collection and manual annotation.

[0133] The point cloud data enhancement method provided in the embodiment of the present application generates enhanced point cloud data of the scene to be enhanced by fusing the point cloud data of the existing targets in the initial point cloud data set with the background point cloud of the scene to be enhanced, thereby obtaining training data of the new scene without the need for field collection and manual labeling, thereby not only reducing the difficulty of training data collection, saving time and labor costs, but also improving the generalization ability of the training data set.

[0134] In a possible implementation form of the present application, since the frequency and position of the target appearing in the actual traffic scene are random, and the installation position of the radar equipment is fixed, it will cause the point cloud data set collected in the field to have uneven sample distribution, thereby affecting the performance of the trained model. Therefore, the point cloud data enhancement method of the embodiment of the present application can also perform data enhancement on the initial point cloud data set according to the data distribution of the initial point cloud data set, so as to perform data expansion on the scenes with a small number of samples in the initial point cloud data set, so as to improve the uneven sample distribution of the data set collected in the field.

[0135] Combine the following Figure 2 , the point cloud data enhancement method provided in the embodiment of the present application is further explained.

[0136] Figure 2 A flow chart of another point cloud data enhancement method provided in an embodiment of the present application is shown.

[0137] like Figure 2 As shown, the point cloud data enhancement method includes the following steps:

[0138] Step 201, obtaining an initial point cloud data set, wherein the initial point cloud data set includes a plurality of initial point cloud data and annotation data corresponding to each initial point cloud data, and the annotation data includes annotation types and point cloud areas corresponding to each target included in the initial point cloud data.

[0139] The specific implementation process and principle of the above step 201 can refer to the detailed description of the above embodiment, which will not be repeated here.

[0140] Step 202 : performing background extraction on each initial point cloud data set to determine each background point cloud included in the initial point cloud data set.

[0141] Step 203: determine each background point cloud included in the initial point cloud data set as the background point cloud corresponding to the scene to be enhanced.

[0142] In the embodiment of the present application, when data enhancement is performed on the scenes already included in the initial point cloud data set to solve the problem of uneven data distribution of the initial point cloud data set, the background point clouds corresponding to the various scenes already included in the initial point cloud data set can be used as the background point clouds corresponding to the scenes to be enhanced. Therefore, background extraction can be performed on each initial point cloud data in the initial point cloud data set to generate background point clouds corresponding to each initial point cloud data; then, only one background point cloud corresponding to the same scene is retained to determine the background point clouds of each scene already included in the initial point cloud data set, and the extracted background point clouds of each scene are determined as the background point clouds of the scenes to be enhanced.

[0143] For example, assuming that the initial point cloud dataset contains point cloud data collected at three scenes: intersection A, intersection B, and intersection C, then background extraction is performed on each initial point cloud data in the initial point cloud dataset to obtain background point clouds corresponding to the three scenes of intersection A, intersection B, and intersection C, and these three background point clouds can be used as background point clouds corresponding to the scenes to be enhanced.

[0144] Step 204 , performing data analysis on the initial point cloud dataset according to preset indicators to determine the data distribution ratio of the first type of targets in the initial point cloud dataset under the preset indicators, wherein the first type of targets are targets of any annotated type in the initial point cloud dataset.

[0145] The preset index may be determined according to actual data enhancement requirements. For example, the preset index may include at least one of the following indicators: the distance between the target and the radar device, the number of point clouds on the target surface, and the angle of the target.

[0146] It should be noted that the distance between the target and the radar device may refer to the distance between the target and the radar device that collects the initial point cloud data; the angle of the target may refer to the heading angle of the target in the initial point cloud data.

[0147] As a possible implementation method, when the initial point cloud data set contains multiple types of targets, each type of target can be taken as the first type of target in turn to determine the data distribution ratio of each type of target under the preset index. For a type of target in the initial point cloud data set, the target of this type can be determined as the first type of target, and the first type of target can be divided into multiple subclasses according to the preset index, and the ratio of the number of targets contained in each subclass can be determined as the data distribution ratio of the first type of target under the preset index; and so on, the data distribution ratio of each type of target in the initial point cloud data set under the preset index can be determined in the same way.

[0148] For example, assuming that the initial point cloud data set contains three types of targets, namely, motor vehicles, non-motor vehicles, and pedestrians, the three types of targets, namely, motor vehicles, non-motor vehicles, and pedestrians, can be analyzed in turn to determine the data distribution ratios of the three types of targets under the preset indicators. For example, for motor vehicle targets, the targets of each motor vehicle type can be divided into multiple subcategories according to the preset indicators, and then the number of targets contained in each subcategory is determined, and then the ratio of the number of targets contained in each subcategory is determined as the data distribution ratio of motor vehicle targets under the preset indicators. For example, according to the preset indicators, the motor vehicle type targets are divided into three subcategories A, B, and C, and subcategory A contains 100 motor vehicle targets, subcategory B contains 50 motor vehicle targets, and subcategory C contains 50 motor vehicle targets. Then, the data distribution ratio of motor vehicle type targets under the preset indicators can be determined to be 2:1:1. Similarly, the same method can be used to determine the data distribution ratios of non-motor vehicle and pedestrian type targets under the preset indicators.

[0149] Furthermore, since the farther the target is from the radar device, the smaller the target is in the point cloud data, and the smaller the corresponding point cloud number is, the more difficult it is to identify targets that are farther away from the radar device. Therefore, the initial point cloud data set can be enhanced according to the distance between the target and the radar device to balance the algorithm's recognition ability for targets at different distances. That is, in a possible implementation of the embodiment of the present application, the above-mentioned preset indicator includes the distance between the target and the radar device; accordingly, the above-mentioned step 204 may include:

[0150] Determining the distance range to which each first-type target belongs according to the distance between each first-type target and the radar device;

[0151] The data distribution ratio is determined according to the number of first type targets corresponding to each distance range.

[0152] As a possible implementation method, when the preset indicator is the distance between the target and the radar device, multiple distance ranges can be preset, and then the distance range to which each first-type target belongs can be determined according to the distance between each first-type target and the radar device, so as to divide each first-type target into multiple subclasses, and then the ratio of the number of first-type targets corresponding to each distance range can be determined as the data distribution ratio of the first-type target under the preset indicator (i.e., the distance between the target and the radar device).

[0153] For example, assuming that the preset distance ranges include [0,50], (50,100], (100,150], (150,200], according to the distance between each first type target and the radar device, it is determined that the number of targets in the distance range [0,50] is 100, the number of targets in the distance range (50,100] is 150, the number of targets in the distance range (100,150] is 100, and the number of targets in the distance range (150,200] is 200. Then, the data distribution ratio of the first type target can be determined to be 1:1.5:1:2.

[0154] Furthermore, since the number of point clouds on the target surface will also affect the accuracy of target recognition, and since the installation position of the radar device is fixed, the point clouds on the surface of the same target may be unevenly distributed, thereby affecting the accuracy of target recognition. For example, for motor vehicle-type targets, due to the positional relationship between the target and the radar device, the radar device may only be able to collect point cloud data on one side of the target, that is, half of the point clouds of the target may be densely distributed and the other half may be sparsely distributed, thereby affecting the recognition accuracy of motor vehicle-type targets. That is, in a possible implementation method of the embodiment of the present application, the above-mentioned preset indicators may include the number of point clouds on the target surface; accordingly, the above-mentioned step 204 may include:

[0155] Taking the straight line where the heading angle of each first-type target is located as the axis, each first-type target is divided into a first part and a second part;

[0156] Determine the distribution ratio of the number of surface point clouds corresponding to each first type target according to the number of point clouds included in the first part and the number of point clouds included in the second part of each first type target;

[0157] Determine the proportion range to which each first-type target belongs according to the distribution proportion of the number of surface point clouds corresponding to each first-type target;

[0158] The data distribution ratio is determined according to the number of first type targets corresponding to each ratio range.

[0159] As a possible implementation method, for a first type target, the straight line where the heading angle of the first type target is located can be used as the axis to divide the first type target into two parts, namely the first part and the second part, and the number of point clouds contained in the first part and the number of point clouds contained in the second part are determined. Then, the ratio of the number of point clouds contained in the first part to the number of point clouds contained in the second part is determined as the surface point cloud quantity distribution ratio corresponding to the first type target; and so on, in the same way, the surface point cloud quantity distribution ratio corresponding to each first type target can be determined in turn.

[0160] Afterwards, the proportion range to which each first-type target belongs can be determined based on the preset proportion range and the distribution ratio of the number of surface point clouds corresponding to each first-type target; then the number of first-type targets contained in each proportion range is determined, and then the ratio of the number of first-type targets contained in each proportion range is determined as the data distribution ratio of the first-type target under the preset indicator (i.e. the number of point clouds encoded by the target).

[0161] For example, assuming that the preset scale ranges include [0, 0.5], (0.5, 1], (1, +∞], according to the distribution ratio of the number of surface point clouds corresponding to each first-type target, it is determined that the number of targets in the scale range [0, 0.5] is 100, the number of targets in the scale range (0.5, 1] ​​is 200, and the number of targets in the scale range (0.5, 1] ​​is 300. Then, the data distribution ratio of the first-type target can be determined to be 1:2:3.

[0162] It should be noted that the distribution ratio of the number of point clouds on the target surface has a greater impact on the recognition of motor vehicle targets. Therefore, this indicator can be used to determine the data distribution ratio of motor vehicle targets. The distribution of the number of surface point clouds of motor vehicle targets in the data set can be used to reduce the impact of uneven distribution of point clouds on the target surface on motor vehicle target recognition.

[0163] Furthermore, since the installation position of the radar device is fixed, the orientation of the target in the point cloud data may be relatively single, thereby affecting the accuracy of the algorithm in identifying targets at other angles. Therefore, the initial point cloud data set may also be enhanced according to the angles of each target in the initial point cloud data set, so that targets at various angles are evenly distributed in the data set. That is, in a possible implementation of the embodiment of the present application, the above-mentioned preset indicator includes the angle of the target; accordingly, the above-mentioned step 204 may include:

[0164] According to the angle of each first-type target, determine the angle range to which each first-type target belongs;

[0165] The data distribution ratio is determined according to the number of first type targets corresponding to each angle range.

[0166] As a possible implementation method, the angle of the target may refer to the heading angle of the target in the initial point cloud data. When analyzing the data distribution ratio of the first type of target by the angle of the target, multiple angle ranges may be preset, and according to the angle of each first type of target and the preset angle range, the angle range to which each first type of target belongs may be determined, and then the number of first type of targets corresponding to each angle range may be determined, and the ratio of the number of first type of targets corresponding to each angle range may be determined as the data distribution ratio of the first type of target under the indicator of the angle of the target.

[0167] For example, assuming that the preset angle ranges include [-180°, -120°], (-120°, -60°], (-60°, 0°], (0°, 60°], (60°, 120°], (120°, 180°], according to the angles of the first type targets, it is determined that the number of targets in the angle range [-180°, -120°] is 100, and the number of targets in the angle range (-120°, -60°] is 2 00, the number of targets in the angle range (-60°, 0°] is 50, the number of targets in the angle range (0°, 60°] is 150, the number of targets in the angle range (60°, 120°] is 150, and the number of targets in the angle range (120°, 180°] is 300. It can be determined that the data distribution ratio of the first type of targets under the indicator of target angle is 1:2:0.5:1.5:1.5:3.

[0168] Step 205: Determine the extraction ratio of the first type of target under the preset index according to the data distribution ratio.

[0169] As a possible implementation method, the extraction ratio of the first type of target under the preset indicator and the data distribution ratio of the first type of target under the preset indicator can be negatively correlated. It can be understood that for a first type of target, after the first type of target is divided into multiple subclasses according to the preset indicator and the data distribution ratio of each subclass under the preset indicator is determined, if the data distribution ratio corresponding to a subclass is higher, it means that the number of targets of this subclass in the initial point cloud data set is more; if the data distribution ratio corresponding to a subclass is lower, it means that the number of targets of this subclass in the initial point cloud data set is smaller. Therefore, in order to balance the data distribution of various types of samples in the data set after data enhancement, for subclasses with higher data distribution ratios, the corresponding extraction ratio can be lower; for subclasses with lower data distribution ratios, the corresponding extraction ratio can be higher. This allows more targets with a smaller number in the initial data set to be extracted, and further allows more targets with a smaller number in the initial point cloud data set to be selected randomly from the extracted targets and placed in the background point cloud to generate enhanced point cloud data. This allows the generated enhanced point cloud data to contain more targets with a smaller number in the initial point cloud data set, so as to balance the sample distribution in the enhanced data set.

[0170] As an example, the extraction ratio of the first type of target under the preset indicator may be inversely proportional to the data distribution ratio of the first type of target under the preset indicator. For example, assuming that the first type of target is divided into three subclasses according to the preset indicator, and the data distribution ratio of each subclass under the preset indicator is 1:2:3, then it can be determined that the extraction ratio of the first type of target under the preset indicator is 3:2:1.

[0171] Furthermore, since the farther the target is from the radar device, the fewer point clouds corresponding to the target are when the radar device collects the point cloud of the target, the more difficult it is to identify the target that is farther from the radar device. Therefore, when the initial point cloud data set is enhanced, more point cloud data of targets that are farther from the radar device can be added to the enhanced point cloud data to improve the recognition accuracy of distant targets. That is, in a possible implementation of the embodiment of the present application, when the preset indicator includes the distance between the target and the radar device, the extraction ratio of the first type of target under the preset indicator can be positively correlated with the distance between the first type of target and the radar device.

[0172] It can be understood that when the preset indicator includes the distance between the target and the radar device, if the extraction ratio of the first type of target under the preset indicator is positively correlated with the distance between the first type of target and the radar device, it means that the greater the distance from the radar device, the greater the extraction ratio of the first type of target, so that the targets farther away from the radar device in the final extraction account for a larger proportion of all the extracted targets, and then when targets are randomly selected from the extracted targets and placed in the background point cloud to generate enhanced point cloud data, more targets farther away from the radar device can be extracted, so that more enhanced point cloud data containing targets farther away from the radar device can be generated, so that the final generated enhanced data set can contain more targets farther away from the radar device, thereby further improving the recognition accuracy of long-distance targets.

[0173] Step 206 , extracting point cloud areas corresponding to the first type of target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data and the extraction ratio of the first type of target under a preset indicator.

[0174] In an embodiment of the present application, after determining the extraction ratio of each type of target in the initial point cloud data set under the preset indicator, the point cloud area corresponding to each type of target can be extracted from the initial point cloud data set according to the extraction ratio of each type of target under the preset indicator.

[0175] Step 207: placing a point cloud area corresponding to at least one target in the background point cloud to generate a plurality of enhanced point cloud data.

[0176] Step 208 : adding each enhanced point cloud data to the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced.

[0177] The specific implementation process and principle of the above steps 207-208 can refer to the detailed description of the above embodiment, which will not be repeated here.

[0178] The point cloud data enhancement method provided in the embodiment of the present application performs data enhancement on the initial point cloud data set according to the data distribution of various targets in the initial point cloud data set, so as to expand the data for scenes with a small number of samples in the initial point cloud data set, so as to improve the uneven distribution of samples in the data set collected in the field, thereby not only reducing the difficulty of training data collection, saving time and labor costs, and improving the generalization ability of the training data set, but also further improving the recognition accuracy of various targets.

[0179] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0180] Corresponding to the point cloud data enhancement method described in the above embodiment, Figure 3 A schematic diagram of the structure of a point cloud data enhancement device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0181] Reference Figure 3 The device 30 comprises:

[0182] A first acquisition module 31 is used to acquire an initial point cloud data set, wherein the initial point cloud data set includes a plurality of initial point cloud data and annotation data corresponding to each initial point cloud data, wherein the annotation data includes an annotation type and a point cloud area corresponding to each target included in the initial point cloud data;

[0183] The second acquisition module 32 is used to acquire the background point cloud corresponding to the scene to be enhanced;

[0184] A first extraction module 33 is used to extract a point cloud area corresponding to each target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data;

[0185] A first placement module 34 is used to place a point cloud area corresponding to at least one target in a background point cloud to generate a plurality of enhanced point cloud data;

[0186] The first generating module 35 is used to add each enhanced point cloud data to the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced.

[0187] The point cloud data enhancement device provided in the embodiment of the present application generates enhanced point cloud data of the scene to be enhanced by fusing the point cloud data of the existing targets in the initial point cloud data set with the background point cloud of the scene to be enhanced, thereby obtaining training data of the new scene without the need for field collection and manual labeling, thereby not only reducing the difficulty of training data collection, saving time and labor costs, but also improving the generalization ability of the training data set.

[0188] In a possible implementation form of the present application, the above-mentioned device 30 further includes:

[0189] The first determination module is used to determine the area to be enhanced in the background point cloud and the type of the target to be enhanced corresponding to the area to be enhanced;

[0190] Accordingly, the first placement module 34 includes:

[0191] The first placement unit is used to place a point cloud area corresponding to at least one target in the area to be enhanced in the background point cloud according to the type of the target to be enhanced and the annotation type corresponding to each target, so as to generate multiple enhanced point cloud data.

[0192] Furthermore, in another possible implementation of the present application, the second acquisition module 32 includes:

[0193] A first determining unit is used to perform background extraction on each initial point cloud data set to determine each background point cloud included in the initial point cloud data set;

[0194] The second determining unit is used to determine each background point cloud included in the initial point cloud data set as a background point cloud corresponding to the scene to be enhanced.

[0195] Furthermore, in another possible implementation of the present application, the device 30 further includes:

[0196] A second determination module is used to perform data analysis on the initial point cloud data set according to a preset indicator to determine a data distribution ratio of a first type of target in the initial point cloud data set under the preset indicator, wherein the first type of target is a target of any annotated type in the initial point cloud data set;

[0197] A third determination module is used to determine the extraction ratio of the first type of target under a preset indicator according to the data distribution ratio;

[0198] Accordingly, the first extraction module 33 includes:

[0199] The first extraction unit is used to extract a point cloud area corresponding to the first type of target from each initial point cloud data according to the annotation data corresponding to each initial point cloud data and the extraction ratio of the first type of target under a preset indicator.

[0200] Furthermore, in another possible implementation of the present application, the extraction ratio of the above-mentioned first type of target under the preset indicator is negatively correlated with the data distribution ratio of the first type of target under the preset indicator.

[0201] Furthermore, in another possible implementation of the present application, the above-mentioned initial point cloud data is collected by a radar device; accordingly, the above-mentioned preset indicators include at least one of the following indicators: the distance between the target and the radar device, the number of point clouds on the target surface, and the angle of the target.

[0202] Furthermore, in another possible implementation of the present application, when the above-mentioned preset indicator includes the distance between the target and the radar device, the extraction ratio of the above-mentioned first type of target under the preset indicator is positively correlated with the distance between the first type of target and the radar device.

[0203] Furthermore, in another possible implementation of the present application, the preset indicator includes a distance between the target and the radar device; accordingly, the second determination module includes:

[0204] A third determining unit, configured to determine a distance range to which each first-type target belongs according to a distance between each first-type target and the radar device;

[0205] The fourth determining unit is used to determine the data distribution ratio according to the number of first type targets corresponding to each distance range.

[0206] Furthermore, in another possible implementation of the present application, the preset indicator includes the number of point clouds of the target surface; accordingly, the second determination module includes:

[0207] A first division unit is used to divide each first type target into a first part and a second part respectively, with the straight line where the heading angle of each first type target is located as an axis;

[0208] a fifth determining unit, configured to determine a distribution ratio of the number of surface point clouds corresponding to each first type target, according to the number of point clouds included in the first part and the number of point clouds included in the second part of each first type target;

[0209] A sixth determining unit, configured to determine a proportion range to which each first-type target belongs according to a distribution ratio of the number of surface point clouds corresponding to each first-type target;

[0210] The seventh determining unit is used to determine the data distribution ratio according to the number of first type targets corresponding to each ratio range.

[0211] Furthermore, in another possible implementation of the present application, the preset indicator includes an angle of the target; accordingly, the second determination module includes:

[0212] An eighth determining unit, configured to determine, according to the angle of each first-type target, an angle range to which each first-type target belongs;

[0213] The ninth determining unit is used to determine the data distribution ratio according to the number of first type targets corresponding to each angle range.

[0214] Furthermore, in another possible implementation of the present application, the first placement module 34 includes:

[0215] a tenth determining unit, configured to determine, according to a scene type corresponding to the scene to be enhanced, a target insertion number N corresponding to the background point cloud, where N is a positive integer;

[0216] The second placement unit is used to place the point cloud areas corresponding to the N targets in the background point cloud to generate a plurality of enhanced point cloud data.

[0217] Furthermore, in another possible implementation of the present application, the first placement module 34 includes:

[0218] The third placement unit is used to place the point cloud areas corresponding to the respective targets in the background point cloud in sequence, so that the point cloud areas corresponding to the respective targets do not overlap.

[0219] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0220] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0221] In order to implement the above embodiments, the present application also proposes a terminal device.

[0222] Figure 4 A schematic diagram of the structure of a terminal device according to an embodiment of the present application.

[0223] like Figure 4 As shown, the terminal device 200 includes:

[0224] A memory 210 and at least one processor 220, a bus 230 connecting different components (including the memory 210 and the processor 220), the memory 210 stores a computer program, and when the processor 220 executes the program, the point cloud data enhancement method described in the embodiment of the present application is implemented.

[0225] Bus 230 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0226] The terminal device 200 typically includes a variety of electronic device readable media, which can be any available media that can be accessed by the terminal device 200, including volatile and non-volatile media, removable and non-removable media.

[0227] The memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. The terminal device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 260 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to bus 230 via one or more data medium interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0228] A program / utility 280 having a set (at least one) of program modules 270 may be stored, for example, in the memory 210, such program modules 270 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 270 generally perform the functions and / or methods of the embodiments described herein.

[0229] The terminal device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and can also communicate with one or more devices that enable users to interact with the terminal device 200, and / or communicate with any device that enables the terminal device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be carried out through an input / output (I / O) interface 292. In addition, the terminal device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 293. As shown in the figure, the network adapter 293 communicates with other modules of the terminal device 200 through a bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the terminal device 200, including but not limited to: microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive, and data backup storage system, etc.

[0230] The processor 220 executes various functional applications and data processing by running the programs stored in the memory 210 .

[0231] It should be noted that the implementation process and technical principles of the terminal device of this embodiment refer to the aforementioned explanation of the point cloud data enhancement method of the embodiment of the present application, and will not be repeated here.

[0232] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0233] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0234] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0235] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0236] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0237] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0238] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A point cloud data enhancement method, It is characterized in that include: Acquire an initial point cloud data set, wherein the initial point cloud data set includes a plurality of initial point cloud data and annotation data corresponding to each of the initial point cloud data, wherein the annotation data includes an annotation type and a point cloud area corresponding to each target included in the initial point cloud data; Obtain the background point cloud corresponding to the scene to be enhanced; Extracting point cloud regions corresponding to each of the targets from each of the initial point cloud data according to the annotation data corresponding to each of the initial point cloud data; placing at least one point cloud area corresponding to the target in the background point cloud to generate a plurality of enhanced point cloud data; Each of the enhanced point cloud data is added to the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced.

2. The method according to claim 1, It is characterized in that Before placing the point cloud area corresponding to at least one of the targets in the background point cloud to generate a plurality of enhanced point cloud data, the method further includes: Determine the area to be enhanced in the background point cloud and the type of the target to be enhanced corresponding to the area to be enhanced; Placing a point cloud area corresponding to at least one of the targets in the background point cloud to generate a plurality of enhanced point cloud data includes: According to the type of the target to be enhanced and the annotation type corresponding to each of the targets, a point cloud area corresponding to at least one of the targets is placed in the area to be enhanced in the background point cloud to generate a plurality of enhanced point cloud data.

3. The method according to claim 1, It is characterized in that The step of obtaining a background point cloud corresponding to the scene to be enhanced includes: Performing background extraction on each of the initial point cloud data to determine each of the background point clouds included in the initial point cloud data set; Each of the background point clouds included in the initial point cloud data set is determined as the background point cloud corresponding to the scene to be enhanced.

4. The method according to claim 3, It is characterized in that Before extracting the point cloud area corresponding to each of the targets from each of the initial point cloud data according to the annotation data corresponding to each of the initial point cloud data, the method further includes: Performing data analysis on the initial point cloud data set according to a preset indicator to determine a data distribution ratio of a first type of target in the initial point cloud data set under the preset indicator, wherein the first type of target is a target of any of the annotated types in the initial point cloud data set; Determining, according to the data distribution ratio, an extraction ratio of the first type of target under the preset indicator; The step of extracting a point cloud region corresponding to each of the targets from each of the initial point cloud data according to the annotation data corresponding to each of the initial point cloud data comprises: According to the annotation data corresponding to each of the initial point cloud data and the extraction ratio of the first type of target under the preset index, the point cloud area corresponding to the first type of target is extracted from each of the initial point cloud data.

5. The method according to claim 4, It is characterized in that There is a negative correlation between the extraction ratio of the first type of target under the preset indicator and the data distribution ratio of the first type of target under the preset indicator.

6. The method according to claim 4, It is characterized in that The initial point cloud data is collected by a radar device, and the preset index includes at least one of the following indexes: the distance between the target and the radar device, the number of point clouds on the target surface, and the angle of the target.

7. The method according to claim 6, It is characterized in that When the preset indicator includes the distance between the target and the radar device, the extraction ratio of the first type of target under the preset indicator is positively correlated with the distance between the first type of target and the radar device.

8. The method according to claim 6, It is characterized in that The preset indicator includes a distance between the target and the radar device, and the performing data analysis on the initial point cloud data set according to the preset indicator to determine a data distribution ratio of the first type of target in the initial point cloud data set under the preset indicator includes: Determining, according to the distance between each of the first-type targets and the radar device, a distance range to which each of the first-type targets belongs; The data distribution ratio is determined according to the number of the first type of targets corresponding to each of the distance ranges.

9. The method according to claim 6, It is characterized in that The preset indicator includes the number of point clouds on the target surface, and the data analysis of the initial point cloud data set according to the preset indicator to determine the data distribution ratio of the first type of target in the initial point cloud data set under the preset indicator includes: Taking the straight line where the heading angle of each first-type target lies as the axis, each first-type target is divided into a first part and a second part; Determining the distribution ratio of the number of surface point clouds corresponding to each first-type target respectively according to the number of point clouds included in the first part and the number of point clouds included in the second part of each first-type target; Determining a proportion range to which each of the first-type targets belongs according to a distribution ratio of the number of surface point clouds corresponding to each of the first-type targets; The data distribution ratio is determined according to the number of the first type of targets corresponding to each of the ratio ranges.

10. The method according to claim 6, It is characterized in that The preset indicator includes the angle of the target, and the data analysis of the initial point cloud data set according to the preset indicator to determine the data distribution ratio of the first type of target in the initial point cloud data set under the preset indicator includes: Determining, according to the angle of each of the first-type targets, an angle range to which each of the first-type targets belongs; The data distribution ratio is determined according to the number of the first type of targets corresponding to each of the angle ranges.

11. The method according to any one of claims 1 to 10, It is characterized in that Placing a point cloud area corresponding to at least one of the targets in the background point cloud to generate a plurality of enhanced point cloud data includes: Determine, according to the scene type corresponding to the scene to be enhanced, the target insertion number N corresponding to the background point cloud, where N is a positive integer; The point cloud areas corresponding to the N targets are placed in the background point cloud to generate a plurality of enhanced point cloud data.

12. The method according to any one of claims 1 to 10, It is characterized in that Placing a point cloud area corresponding to at least one of the targets in the background point cloud to generate a plurality of enhanced point cloud data includes: The point cloud regions corresponding to the respective targets are sequentially placed in the background point cloud so that the point cloud regions corresponding to the respective targets do not overlap.

13. A point cloud data enhancement device, It is characterized in that include: A first acquisition module is used to acquire an initial point cloud data set, wherein the initial point cloud data set includes a plurality of initial point cloud data and annotation data corresponding to each of the initial point cloud data, wherein the annotation data includes an annotation type and a point cloud area corresponding to each target included in the initial point cloud data; The second acquisition module is used to acquire the background point cloud corresponding to the scene to be enhanced; A first extraction module, configured to extract a point cloud region corresponding to each of the targets from each of the initial point cloud data according to the annotation data corresponding to each of the initial point cloud data; A first placement module, used for placing a point cloud area corresponding to at least one of the targets in the background point cloud to generate a plurality of enhanced point cloud data; The first generating module is used to add each of the enhanced point cloud data to the initial point cloud data set to generate an enhanced point cloud data set corresponding to the scene to be enhanced.

14. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images