Point cloud data equalization resampling method and device, target detection equipment and terminal

Abstract

The application discloses a point cloud data equalization resampling method and device, a target detection equipment and a terminal. By calculating the weight value of a target class and the weight value of a data frame, when randomly sampling the data frame of the target class, the sampling probability of important classes and important frames is improved, the long-tail distribution phenomenon is reduced, the sample quantity of the target class is reduced, the situation that the sample data quantity of resampling sharply increases is avoided, and the data distribution before and after resampling is compared according to the equalization index specially designed for resampling data, so that the equalization degree of the sample data of resampling is ensured, and the training effect of a target detection perception model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud data resampling, and particularly relates to a point cloud data balanced resampling method and device, a target detection equipment and a terminal. BACKGROUND

[0002] Target detection perception models are widely used in automatic driving, and their accuracy directly affects the effect of subsequent algorithms such as tracking and control. The training effect of the target detection perception model depends on the quality of the point cloud data training set.

[0003] Due to the limitations of data collection, there is a serious long-tail phenomenon in directly collected point cloud data. Therefore, it is necessary to resample the point cloud data.

[0004] Although the existing resampling method can increase the number of tail samples to some extent through random sampling, the total number of samples increases more, and cannot effectively reduce the long-tail phenomenon, and the distribution of resampled data is more uneven. SUMMARY

[0005] The main purpose of the present application is to provide a point cloud data balanced resampling method and device, a target detection equipment, an intelligent terminal and a computer readable storage medium, which can solve the problem that the long-tail phenomenon cannot be effectively reduced, and the distribution of resampled data is more uneven.

[0006] The first aspect of the present application provides a point cloud data balanced resampling method, which comprises:

[0007] calculating a first balance degree of the point cloud data for representing the degree of uniform distribution of data;

[0008] based on the point cloud data, calculating a first weight value of each target class in a preset target class set;

[0009] traversing the preset target class set in the order of descending first weight value to obtain a first current target class, sampling in the point cloud data according to the first current target class to obtain a class sample pool, calculating a second weight value of each data frame in the class sample pool, sampling in the class sample pool with the second weight value as a sampling probability, obtaining resampled data of the first current target class and saving to a sample set, the sample set including resampled data of each target class in the preset target class set;

[0010] calculating a second balance degree of the sample set for representing the degree of uniform distribution of data;

[0011] when the second balance degree is greater than the first balance degree, re-executing the above traversal operation to reacquire the sample set;

[0012] When the second balance degree is less than or equal to the first balance degree, a resampling result is obtained according to the sample set.

[0013] According to the above technical means, the first weight value of the target class and the second weight value of the data frame are calculated, the sampling probability of the important class and the important frame is improved when the data frame of the target class is randomly sampled, and the long-tail distribution phenomenon is reduced; and the data distribution before and after resampling is compared according to the balance degree, the balance degree of the resampling result is ensured, and the training effect of the target detection perception model is improved.

[0014] Optionally, the balance degree of the target data is calculated, comprising:

[0015] traversing the preset target class set to obtain a second current target class, searching for the second current target class in each frame of the target data, accumulating the number of frames in which the second current target class is found, obtaining the class frame number of the second current target class, the target data being the point cloud data or the sample set;

[0016] According to the class frame number of all target classes in the preset target class set, the probability of each target class in the preset target class set is calculated;

[0017] Based on all the probabilities, the probability distribution of the preset target class set is obtained;

[0018] The variance of the probability distribution is calculated, the first balance degree is obtained when the target data is the point cloud data, and the second balance degree is obtained when the target data is the sample set.

[0019] According to the above technical means, by calculating the probability of the target class and the balance degree according to all the probabilities, compared with directly using the variance, the data distribution degree before and after resampling can be compared under the same standard, and is not affected by the sample data size before and after resampling, and the comparison result is more reliable.

[0020] Optionally, based on the point cloud data, the first weight value of the target class is calculated, comprising:

[0021] traversing the preset target class set to obtain a third current target class, searching for the third current target class in each frame of the point cloud data, accumulating the number of frames in which the third current target class is found, and obtaining the class frame number of the third current target class;

[0022] According to the class frame number of all target classes in the preset target class set, the importance evaluation value of each target class in the preset target class set is calculated, and the importance evaluation value is inversely related to the class frame number.

[0023] According to the importance evaluation value of the target class, the first weight value is obtained.

[0024] According to the above technical means, the first weight value of each target class is calculated by using the class frame number of each target class, which is suitable for point cloud data and can accurately evaluate the weight of each target class in the point cloud data.

[0025] Optionally, the first weight value is obtained according to the importance evaluation value of the target class, comprising:

[0026] According to the importance evaluation value of all target classes in the preset target class set, the importance evaluation value of the target class is normalized to obtain the first weight value.

[0027] According to the above technical means, after normalization processing, the first weight value of the target class is more standard, which is conducive to comparison.

[0028] Optionally, the second weight value of the data frame in the class sample pool is calculated, comprising:

[0029] Enumerating the target classes appearing in the data frame, a class target set is obtained;

[0030] The first current target class is eliminated from the class target set;

[0031] The first weight value of each target class in the class target set is accumulated to obtain the second weight value of the data frame and perform normalization processing.

[0032] According to the above technical means, the second weight value of the data frame is calculated by using the first weight value of other target classes in the data frame, which can accurately and objectively measure the importance of the data frame.

[0033] Optionally, the resampling data of the first current target class is obtained by sampling in the class sample pool, comprising:

[0034] According to the number of frames containing the first current target class in the sample set, the sampling number is determined;

[0035] Based on the sampling number, the resampling data of the first current target class is obtained by sampling in the class sample pool.

[0036] According to the above technical means, by dynamically determining the sampling number, the sample number of the target class can be reduced, and the situation of explosive increase of sample data after resampling can be avoided.

[0037] Optionally, the sampling number is determined according to the number of frames containing the first current target class in the sample set, comprising:

[0038] when the total number of data frames in the sample set is less than the preset average frame number, setting the preset average frame number as the sampling number;

[0039] Otherwise, accumulating the number of frames in the sample set containing the first current target class to obtain a sampled frame number, when the sampled frame number is less than the preset average frame number, setting the difference between the preset average frame number and the sampled frame number as the sampling number.

[0040] According to the above technical means, the sampling number is dynamically determined according to the preset average frame number and the number of frames in the sample set containing the first current target class, which can not only ensure that the number of frames of each target class in the sample set meets the requirement of resampling, but also avoid the situation that the amount of sample data after resampling increases sharply.

[0041] Optionally, the obtaining of the resampling result according to the sample set comprises:

[0042] saving the sample set to a sampling pool;

[0043] re-executing the above traversal operation to obtain a next sample set, calculating the balance degree of the next sample set to obtain a third balance degree, when the third balance degree is less than the first balance degree, saving the next sample set to the sampling pool, until the number of sample sets in the sampling pool is equal to a preset scheme number;

[0044] setting the sample set with the minimum balance degree in the sampling pool as the resampling result.

[0045] According to the above technical means, a plurality of sample sets are resampled, and then the sample set with the minimum balance degree is selected, which further ensures that the resampling result is distributed evenly and improves the training effect of the target detection perception model.

[0046] The second aspect of the present application provides a point cloud data balanced resampling device, wherein the device comprises:

[0047] a balance degree calculation module configured to calculate a first balance degree of point cloud data and a second balance degree of the sample set, wherein the first balance degree and the second balance degree are both used to represent the degree of uniform distribution of data;

[0048] a class importance module configured to calculate a first weight value of each target class in a preset target class set based on the point cloud data;

[0049] a sample set module configured to traverse the preset target class set in a descending order of the first weight values, obtain a first current target class, sample in the point cloud data according to the first current target class, obtain a class sample pool, calculate a second weight value of each data frame in the class sample pool, sample in the class sample pool with the second weight value as a sampling probability, obtain resampling data of the first current target class, and save the resampling data to a sample set, the sample set including resampling data of each target class in the preset target class set;

[0050] a result module configured to re-execute the above traversal operation to re-obtain the sample set when the second balance degree is greater than the first balance degree, and obtain resampling results according to the sample set when the second balance degree is less than or equal to the first balance degree.

[0051] Optionally, the balance degree calculation module includes a traversal unit and a balance degree unit, the traversal unit is configured to traverse the preset target class set, obtain a second current target class, find the second current target class in each frame of the point cloud data or the sample set, accumulate a frame number of the second current target class, and obtain a class frame number of the second current target class; and the balance degree unit is configured to calculate a probability of each target class in the preset target class set according to the class frame numbers of all target classes in the preset target class set, obtain a probability distribution of the preset target class set based on all the probabilities, calculate a variance of the probability distribution, and obtain the balance degree.

[0052] Optionally, the class importance module includes a traversal unit and a weight value unit, the traversal unit is configured to traverse the preset target class set, obtain a third current target class, find the third current target class in each frame of the point cloud data, accumulate a frame number of the third current target class, and obtain a class frame number of the third current target class; and the weight value unit is configured to calculate an importance evaluation value of each target class in the preset target class set according to the class frame numbers of all target classes in the preset target class set, the importance evaluation value and the class frame number are inversely related, and obtain the first weight value according to the importance evaluation value of the target class.

[0053] Optionally, the sample set module includes a frame importance unit, the frame importance unit is configured to enumerate a target class appearing in a data frame in the class sample pool, obtain a class target set, eliminate the first current target class from the class target set, accumulate the first weight value of each target class in the class target set, obtain the second weight value of the data frame, and perform normalization processing.

[0054] Optionally, the sample set module comprises a sample number unit, which is configured to: when the total number of data frames in the sample set is less than the preset average frame number, set the preset average frame number as the sample number; otherwise, accumulate the number of frames in the sample set containing the first current target class to obtain a sampled frame number, and when the sampled frame number is less than the preset average frame number, set the difference between the preset average frame number and the sampled frame number as the sample number.

[0055] Optionally, the method further comprises a sample pool module, which is configured to save the sample set, and when the number of sample sets in the sample pool is less than a preset scheme number, re-perform the above-mentioned traversal operation to obtain a next sample set, calculate the balance degree of the next sample set to obtain a third balance degree, and when the third balance degree is less than the first balance degree, save the next sample set to the sample pool, until the number of sample sets in the sample pool is equal to the preset scheme number, and set the sample set with the smallest balance degree in the sample pool as the resampling result.

[0056] The third aspect of the present application provides a target detection device for automatic driving, which comprises a memory, a processor, and a target detection perception model stored in the memory and executable on the processor, wherein the training sample used when the target detection perception model is trained is the sample data obtained by performing any one of the point cloud data balanced resampling methods.

[0057] The fourth aspect of the present application provides an intelligent terminal, which comprises a memory, a processor, and a point cloud data balanced resampling program stored in the memory and executable on the processor, wherein the point cloud data balanced resampling program implements the steps of any one of the point cloud data balanced resampling methods when executed by the processor.

[0058] The fifth aspect of the present application provides a computer readable storage medium, which stores a point cloud data balanced resampling program, wherein the point cloud data balanced resampling program implements the steps of any one of the point cloud data balanced resampling methods when executed by a processor.

[0059] As can be seen from the above, the point cloud data balanced resampling method, device, target detection device, intelligent terminal and computer readable storage medium of the present application calculate the weight value of the target class and the weight value of the data frame, improve the sampling probability of important classes and important frames when randomly sampling the data frames of the target class, reduce the long-tail distribution phenomenon, and can reduce the sample number of the target class to avoid the situation of explosive increase in the sample data quantity of resampling; the data distribution before and after resampling is compared according to the balance degree index specially designed for resampling data, the balance degree of the sample data of resampling is ensured, and the training effect of the target detection perception model is improved. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating the point cloud data equalization and resampling method provided in an embodiment of the present invention;

[0062] Figure 2 yes Figure 1 A flowchart illustrating the calculation of the balance degree in the embodiment;

[0063] Figure 3 yes Figure 1 A flowchart illustrating the calculation of the first weight value of the target class in the embodiment;

[0064] Figure 4 yes Figure 1 A schematic diagram of the process for calculating the second weight value of a data frame in the embodiment;

[0065] Figure 5 This is a schematic diagram of the point cloud data equalization and resampling device provided in an embodiment of the present invention;

[0066] Figure 6 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0067] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0068] There can be multiple targets in each frame of the collected point cloud data, such as large trucks, cars, obstacles, water marks, isolation belts, green belts, etc. In the automatic driving process, a target detection perception model is required to accurately detect the targets existing in each frame of the point cloud data and determine the category (i.e., target class) to which the target belongs. According to the category of the target, subsequent tracking and control algorithms are executed. Therefore, the point cloud data samples used to train the target detection perception model need to include samples of each target class, and ensure that each target class has a certain number of samples, so that the training effect of the target detection perception model is better, and the accuracy of the target detection perception model is higher. Common target classes include vehicle class, obstacle class, and pedestrian class, etc. All target classes form a target class set. It should be noted that the specific content of the target class is not limited and can be pre-set according to the needs of the automatic driving scene.

[0069] However, due to the limitations of data collection, the directly collected point cloud data has a serious long-tail phenomenon, such as large trucks, cars, etc. accounting for a large proportion, and obstacle, water mark, etc. target class only accounts for a small proportion. This long-tail phenomenon often leads to low accuracy of the tail class during training.

[0070] To alleviate this long-tail distribution phenomenon, the collected point cloud data is often resampled (i.e., resampled). By counting the number of frames containing each target class, the average number of frames of all target classes is calculated. Then, random sampling is performed to enable each target class to reach the average number of frames. However, the data volume of the resampled samples is larger, increasing the training cost, and cannot effectively reduce the long-tail distribution effect, and also makes the data distribution of the resampled samples more uneven.

[0071] The present application collects point cloud data by class according to the importance of the target class, and measures the importance of the data frame when collecting point cloud data of each target class, so as to improve the sampling probability of important class data frames when randomly sampling target class data frames, thereby reducing the long-tail distribution phenomenon. And according to the characteristics of point cloud data resampling, an index for quantifying the balance degree of point cloud data distribution is designed to compare the data distribution before and after resampling, to ensure that the sample data distribution after resampling is balanced.

[0072] Specifically, the present embodiment provides a point cloud data balanced resampling method running on an electronic device, for resampling the collected laser point cloud data, and using the resampled sample data as training data set to train the target detection perception model. It can alleviate the long-tail distribution phenomenon of the current resampled laser point cloud training data, and at the same time reduce the problem of training data explosion and training time increase caused by resampling.

[0073] The resampling step is as shown in Figure 1 , specifically including:

[0074] Step S100: Calculate the first uniformity of the point cloud data to characterize the degree of uniformity of the data distribution;

[0075] Balance is a quantitative indicator of the degree of data dispersion, used to characterize the uniformity of data distribution. During deep learning model training, the more balanced the distribution of sample data, the better the model training performance. In other words, the more balanced the distribution of resampled sample data, the better the model training performance. However, resampling can lead to a surge in data volume, potentially worsening the balance of the resampled sample data. To ensure that the resampled sample data distribution remains balanced, we first calculate the balance of the point cloud data to obtain the initial balance, and then calculate the balance of the resampled data. By comparing the balance before and after resampling, we can determine the change in data distribution and ensure that the resampled sample data distribution is balanced.

[0076] Variance reflects the degree to which a set of data deviates from its mean. It is often used as an indicator of the balance of point cloud data.

[0077] Considering that resampling the collected point cloud data will increase the size of the resampled sample data by an order of magnitude, directly calculating the variance of the resampled sample data and comparing it directly with the variance of the original point cloud data does not compare the degree of balance before and after resampling under the same standard, resulting in inaccurate comparison results. Therefore, this embodiment uses a method of calculating normalized variance to measure the balance of point cloud data.

[0078] The steps for calculating the balance are as follows: Figure 2 As shown, it specifically includes:

[0079] Step S110: Traverse the preset target class set to obtain the current target class. Search for the current target class in each frame of the point cloud data, accumulate the number of frames in which the current target class is found, and obtain the class frame number of the current target class.

[0080] Each data frame of point cloud data may contain multiple target classes. For example, a single data frame may contain vehicles, pedestrians, and obstacles. Therefore, each target class will appear in multiple data frames, and multiple target classes can be found in each data frame. By traversing each target class in a preset target class set, the number of times the currently traversed target class (i.e., the current target class) appears in data frames is accumulated to obtain the class frame count of the current target class. Assuming the target class is pedestrians, and pedestrians appear in 2000 frames, then the class frame count for pedestrians is 2000. Finally, the class frame count for each target class in the preset target class set is obtained. This can be represented as: (s1, s2, ..., s n ),in s i This represents the class frame number of the i-th target class.

[0081] Step S120: According to the class frame number of all target classes in the preset target class set, the probability of each target class in the preset target class set is calculated.

[0082] Step S130: Based on all the above probabilities, the probability distribution of the preset target class set is obtained.

[0083] Step S140: The variance of the probability distribution is calculated to obtain the balance degree.

[0084] According to the class frame number of all target classes in the preset target class set, the probability of each target class in the preset target class set is calculated, and the specific formula is: , wherein P i Pi represents the probability of the i-th target class, s i ni represents the class frame number of the i-th target class.

[0085] The probabilities of all target classes in the preset target class set form the probability distribution of the preset target class set: p 1 , p 2 ,..., p n Then the variance of the probability distribution is calculated to obtain the balance degree: .

[0086] By calculating the normalized variance, the dispersion degree of the point cloud data before and after resampling can be compared under the same standard, and the comparison result is more reliable.

[0087] Step S200: Based on the above point cloud data, the first weight value of each target class in the preset target class set is calculated.

[0088] The occurrence probability of different target classes in the point cloud data is different, for example, the occurrence probability of the road repair sign and other fault warning classes is small. Therefore, relatively speaking, these target classes should be focused on during resampling, and the average frame number of these target classes should be prioritized, so as to both reduce the data volume of the resampled data and improve the long-tail distribution effect. However, at present, the importance of target classes is not distinguished during resampling, but random sampling is continuously performed in the point cloud data, and the data volume is increased to make each class reach the average frame number, resulting in a sharp increase in the data volume of the resampled data and a larger dispersion degree of the data distribution.

[0089] The embodiment determines the importance of the class by analyzing the point cloud data and calculating the first weight value of each target class in the preset target class set, and first samples the point cloud data corresponding to the important class during resampling.

[0090] Specifically, the step of calculating the first weight value of the target class comprises Figure 3 as shown, comprising:

[0091] Step S210: traversing the preset target class set to obtain a current target class, searching for the current target class in each frame of the point cloud data, accumulating the frame number of the current target class, and obtaining the class frame number of the current target class;

[0092] First, the preset target class set is traversed to count the class frame number of each target class. The specific process of traversal can refer to the description in step S110, which will not be repeated here.

[0093] Step S220: calculating the importance evaluation value of each target class in the preset target class set according to the class frame number of all target classes in the preset target class set, wherein the importance evaluation value is inversely related to the class frame number;

[0094] Step S230: obtaining the first weight value according to the importance evaluation value of the target class.

[0095] When the class frame number of the target class in the point cloud data is s i smaller, it should be focused on in the resampling process, and its importance is higher. Based on this criterion, the present application designs a quantitative index for evaluating the importance of the target class.

[0096] First, the importance evaluation value of each target class in the preset target class set is calculated according to the class frame number of all target classes in the preset target class set k i The specific calculation formula is: It can be seen from the above formula that the importance evaluation value of the target class is inversely related to the class frame number: the smaller the class frame number of the target class, the greater the importance evaluation value of the target class.

[0097] After obtaining the importance evaluation value of the target class, the importance evaluation value of the target class can be directly used as the first weight value of the target class.

[0098] The sum of the importance evaluation values of all target classes in this embodiment is: which is not equal to 1, so the importance evaluation value of the target class is normalized: k i The normalized importance evaluation value is used as the first weight value of the target class, so that the first weight value is more accurate and intuitive. The specific expression of normalization is: .

[0099] Step S300: traverse the preset target class set in descending order of the first weight value, obtain a current target class, sample in the point cloud data according to the current target class, obtain a class sample pool, calculate the second weight value of each data frame in the class sample pool, sample in the class sample pool by taking the second weight value as a sampling probability, obtain the resampling data of the current target class and save to the sample set;

[0100] After obtaining the first weight value of each target class in the preset target class set, the preset target class set is traversed in descending order of the first weight value, and each target class in the preset target class set is sequentially sampled to obtain the resampling data of each target class in the preset target class set, and the resampling data of each target class is saved to the sample set. That is, the sample set includes the resampling data of each target class in the preset target class set. The final indicator of resampling is that the total number of each target class appearing in the data frames of the sample set is greater than or equal to the average frame number. For example, assuming that the average frame number is 10, the preset target class set includes target class A, target class B and target class C, and after resampling is completed, at least 10 frames of data frames containing target class A can be found in all data frames in the sample set, and the same is true for target class B and target class C. By specifying the sampling order of the target class, the important target class is sampled first, which can improve the sampling probability of the important class and reduce the long-tail distribution phenomenon.

[0101] Specifically, when sampling the resampling data of the target class, data frames containing the target class are first sampled in the point cloud data, and the set of these data frames forms a class sample pool. It should be noted that each data frame in the class sample pool contains not only the target class being sampled, but also other target classes, and the number of other target classes contained in each data frame is also indefinite.

[0102] Therefore, in order to control the data amount of the resampling data, the second weight value of each data frame in the class sample pool is also calculated, the importance between different frames is measured by the second weight value, the second weight value is taken as the sampling probability of each data frame in the class sample pool, and then sampling is performed in the class sample pool to obtain the resampling data of the current target class. For example, the total number of other target classes included in each data frame in the class sample pool except the current target class is accumulated as the second weight value, or the first weight value of other target classes included in each data frame in the class sample pool except the current target class is accumulated as the second weight value. That is, under the premise that the target class with a high importance evaluation value is preferentially collected, the sampling probability of the data frame containing more target classes in the class sample pool of the target class is also improved, the number of class frames containing each target class in the sample set is improved, and the number of class frames is quickly made greater than or equal to the average frame number.

[0103] Based on the above, the number of targets required to be sampled in the class sample pool of different target classes in the sampling process is different, and the sampling number needs to be determined according to the number of frames containing the current target class in the sample set. The specific process of determining the sampling number in this embodiment is as follows: when the total number of data frames in the sample set is less than the preset average frame number, the preset average frame number is set as the sampling number. For example, when the current target class is the target class with the largest first weight value in the preset target class set, the total number of data frames in the sample set is 0, and then the point cloud data frames of the preset average frame number can be sampled at one time and saved to the sample set. Therefore, the class frame number of the current target class meets the class frame number requirement of the resampling. When the total number of data frames in the sample set is greater than or equal to the preset average frame number, it means that the current target class is not the target class with the largest first weight value in the preset target class set. For example: the target classes in the preset target class set are target class A, target class B and target class C according to the first weight value sorting, and if the current target class is target class B, the resampling data of target class A has been saved in the sample set, and the total number of data frames in the sample set is equal to the preset average frame number. Since each data frame in the resampling data of target class A can also include target class B, in order to make the class frame number corresponding to target class B also greater than or equal to the preset average frame number, the sampling number of target class B can be dynamically determined according to the sampled frame number of target class B in the sample set. Specifically, first, the frame number containing the current target class in the sample set is accumulated to obtain the sampled frame number of the current target class. If the sampled frame number of the current target class is less than the preset average frame number, the difference between the preset average frame number and the sampled frame number is obtained, and the difference is the sampling number of the current target class. If the sampled frame number of the current target class is greater than or equal to the preset average frame number, it means that the class frame number of the current target class meets the requirement, and then the next target class in the preset target class set can be directly sampled to obtain the resampling data of the next target class.

[0104] By analyzing the sample set and counting the frame number containing the current target class in the sample set, the sampling number can be dynamically determined, which can reduce the number of target classes and the total number of training samples, and can also reduce the long-tail distribution phenomenon to a certain extent.

[0105] Step S400: Calculate the balance degree of the sample set to obtain a second balance degree;

[0106] Step S500: When the second balance degree is less than the first balance degree, output the sample set, otherwise re-execute the above iteration operation to reacquire the sample set.

[0107] In order to keep the balance degree of the data in the sample set obtained by resampling, after obtaining the sample set, the balance degree of the sample set is calculated to obtain a second balance degree, the first balance degree and the second balance degree are compared, and only when the second balance degree is less than the first balance degree, it is considered that the balance degree of the sample set obtained by resampling is qualified, otherwise, the step S300 is returned to re-execute the traversal operation to obtain a new sample set, until the balance degree of the sample set is qualified, and the sample set is output as a training sample for training the target detection perception model.

[0108] As can be seen from the above, the weight value of the target class and the weight value of the data frame are calculated in the embodiment, the sampling probability of the important class and the important frame is improved when randomly sampling the data frame of the target class, the long-tail distribution phenomenon is reduced, and the number of samples and training samples of the target class is reduced, avoiding the situation that the amount of sample data increases sharply after resampling; the data distribution before and after resampling is compared according to the balance degree index specially designed for resampling data, the balance degree of the sample data after resampling is ensured, and the training effect of the target detection perception model is improved.

[0109] In one embodiment, the specific steps of calculating the second weight value of the data frame in the class sample pool are as shown in Figure 4 , including:

[0110] Step S310: enumerating the target classes appearing in the data frame to obtain a class target set;

[0111] Step S320: eliminating the current target class from the class target set;

[0112] Step S330: accumulating the first weight value of each target class in the class target set to obtain the second weight value of the data frame and performing normalization processing.

[0113] Different targets exist on each data frame, how to measure the importance of the frame, and determine the sampling probability of the data frame in the sampling process through the importance of the frame, so as to sample according to the importance during sampling, reduce the long-tail effect and reduce the number of samples is crucial.

[0114] The embodiment enumerates each target class appearing in the data frame to obtain a target class set composed of all target classes. Since the class sample pool is sampled in the point cloud data for the current target class, each data frame in the class sample pool contains the current target class. Therefore, the importance of the data frame should exclude the current target class and be measured by other target classes, that is, the current target class needs to be excluded from the target class set, the target class set is updated, and then the importance of the data frame is measured according to the target class set. Specifically, according to the first weight value of each target class calculated in step S200, the first weight value of each target class in the target class set is accumulated, and the accumulated result is taken as the second weight value of the data frame; after obtaining the second weight value of each data frame in the class sample pool, the second weight value of each data frame is normalized.

[0115] The specific process can be represented as follows: assuming that the class sample pool is u{ f 1, f 2,…, f n}, f i is the data frame, which contains the current target class c i and other target classes c 1 ,…, c m , (it should be noted that: f 1, f 2 each contains the current target class c i , but the other target classes contained are different, and the number of m is also different). The second weight value of the data frame is calculated as follows: , wherein k c1 represents the first weight value of the target class c1, k c2 represents the first weight value of the target class c2, and the second weight value of each data frame in the current class sample pool is{ o 1, o 2,…, o n}, and the expression for normalizing the second weight value is: , wherein o i represents the second weight value of the i-th data frame.

[0116] The second weight value is calculated to reflect the importance of the data frame, the importance of other target classes in the data frame is considered, and more balanced resampling data can be obtained.

[0117] In one embodiment, in order to make the data distribution of the sample set obtained by resampling better, multiple rounds of collection are performed, each round obtains a sample set, and finally the sample set with the smallest balance degree is output to ensure the balance of the point cloud data after resampling.

[0118] Specifically, in the step S500, when the second balance degree is smaller than the first balance degree, the sample set obtained in this round is not directly output, but is saved to the sampling pool; the step S300 is re-executed, that is, the traversal operation is re-executed to obtain the next sample set, the balance degree of the next sample set is calculated, the third balance degree is obtained, and when the third balance degree is smaller than the first balance degree, the next sample set is saved to the sampling pool, and the cycle is continued until the number of sample sets in the sampling pool is equal to the preset number of schemes (10 in this embodiment). Then, the balance degrees of each sample set in the sampling pool are compared, and the sample set with the smallest balance degree is taken as the final sampling result as the training sample for training the target detection perception model.

[0119] In summary, the implementation process of the embodiment is as follows: first, the sampling pool is initialized and the sample set is initialized. The importance of each target class is calculated, and sampling is performed in the point cloud data in the order from high to low according to the importance of the target class, to obtain the class sample pool of each target class. It is assumed that the current target class is i Before sampling in the current class sample pool, the importance of each data frame in the class sample pool is calculated as the sampling probability, and then the total number of frames containing the current target class i in the sample set is calculated. If the total number of frames is greater than the preset average number of frames, no sampling is performed; if the total number of frames is less than the preset average number of frames y, sampling is performed. After sampling of all target classes is completed, the balance degree of the sample set is calculated. If the balance degree is less than the balance degree of the point cloud data, the sample set is added to the sampling pool; if the balance degree is greater than the balance degree of the point cloud data, the next round of random sampling is performed, until the total number of sample sets in the sampling pool is 10. Finally, the sample set with the smallest balance degree in the sampling pool is selected as the training data set.

[0120] As shown in Figure 5 corresponding to the above point cloud data balance resampling method, the embodiment of the present application also provides a point cloud data balance resampling device, and the device comprises:

[0121] The balance degree calculation module 600 is configured to calculate the first balance degree of the point cloud data and the second balance degree of the sample set, and the first balance degree and the second balance degree are both used to represent the degree of uniform distribution of data.

[0122] The class importance module 610 is configured to calculate the first weight value of each target class in the preset target class set based on the point cloud data.

[0123] The sample set module 620 is configured to traverse the preset target class set in a descending order of the first weight values, obtain a first current target class, sample the point cloud data according to the first current target class to obtain a class sample pool, calculate a second weight value of each data frame in the class sample pool, sample the class sample pool by taking the second weight value as a sampling probability, obtain resampling data of the first current target class, and save the resampling data to a sample set, wherein the sample set includes resampling data of each target class in the preset target class set.

[0124] The result module 630 is configured to re-perform the above traversal operation to reacquire the sample set when the second balance degree is greater than the first balance degree, and obtain resampling results according to the sample set when the second balance degree is less than or equal to the first balance degree.

[0125] Optionally, the balance degree calculation module includes a traversal unit and a balance degree unit. The traversal unit is configured to traverse the preset target class set, obtain a second current target class, find the second current target class in each frame of the point cloud data or the sample set, accumulate a frame number of the second current target class, and obtain a class frame number of the second current target class. The balance degree unit is configured to calculate a probability of each target class in the preset target class set according to the class frame numbers of all target classes in the preset target class set, obtain a probability distribution of the preset target class set based on all the probabilities, calculate a variance of the probability distribution, and obtain the balance degree.

[0126] Optionally, the class importance module includes a traversal unit and a weight value unit. The traversal unit is configured to traverse the preset target class set, obtain a third current target class, find the third current target class in each frame of the point cloud data, accumulate a frame number of the third current target class, and obtain a class frame number of the third current target class. The weight value unit is configured to calculate an importance evaluation value of each target class in the preset target class set according to the class frame numbers of all target classes in the preset target class set, the importance evaluation value and the class frame number are inversely related, and obtain the first weight value according to the importance evaluation value of the target class.

[0127] Optionally, the sample set module includes a frame importance unit. The frame importance unit is configured to enumerate a target class appearing in a data frame in the class sample pool, obtain a class target set, eliminate the first current target class from the class target set, accumulate the first weight value of each target class in the class target set, obtain the second weight value of the data frame, and perform normalization processing.

[0128] Optionally, the sample set module comprises a sample number unit, which is configured to: when the total number of data frames in the sample set is less than the preset average frame number, set the preset average frame number as the sample number; otherwise, accumulate the number of frames in the sample set containing the first current target class to obtain a sampled frame number, and when the sampled frame number is less than the preset average frame number, set the difference between the preset average frame number and the sampled frame number as the sample number.

[0129] Optionally, the method further comprises a sampling pool module, which is configured to save the sample set, and when the number of sample sets in the sampling pool is less than a preset scheme number, re-perform the above-mentioned traversal operation to obtain a next sample set, calculate the balance degree of the next sample set to obtain a third balance degree, and when the third balance degree is less than the first balance degree, save the next sample set to the sampling pool, until the number of sample sets in the sampling pool is equal to the preset scheme number, and set the sample set with the smallest balance degree in the sampling pool as the resampling result.

[0130] Specifically, the specific functions of the modules of the point cloud data balance resampling device in the embodiment can be referred to the corresponding description in the point cloud data balance resampling method, which will not be described here.

[0131] Based on the above-mentioned embodiments, the application further provides a target detection device for automatic driving, which comprises a memory and a processor, and the memory stores a target detection perception model executable on the processor. The training sample used in the training of the target detection perception model is the sample data obtained by executing any one of the point cloud data balance resampling methods.

[0132] Based on the above-mentioned embodiments, the application further provides an intelligent terminal, and its principle block diagram can be as shown in Figure 6 The above-mentioned intelligent terminal comprises a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the electric vehicle is configured to provide calculation and control capabilities. The memory of the electric vehicle comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a point cloud data balance resampling program. The internal memory provides an environment for the operation of the operating system and the point cloud data balance resampling program in the non-volatile storage medium. The network interface of the electric vehicle is configured to communicate with external terminals through network connection. The point cloud data balance resampling program, when executed by the processor, implements the steps of any one of the point cloud data balance resampling methods. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0133] Those skilled in the art can understand, Figure 6The principle block diagram shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the intelligent terminal to which the scheme of the present application is applied; a specific intelligent terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0134] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a point cloud data equalization resampling program, and the point cloud data equalization resampling program realizes the steps of any point cloud data equalization resampling method provided by the embodiment of the present application when executed by a processor.

[0135] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0136] In addition, the terms "first", "second" are only for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0137] Any process or method descriptions in the flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and the preferred embodiments of the present application include additional implementations in which the order of the steps can differ from those shown or described, including a different order of the steps, or including additional steps, or including some steps in parallel, or including some steps in reverse order, or including some steps in a different manner, all of which are within the scope of the present application.

[0138] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer- readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (a magnetic storage medium), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Note that the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for example, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0139] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0140] Those of skill in the art could readily implement the above described example methods with all or a subset of the recited steps carried out with a program of instructions executed by associated hardware, the program of instructions being stored in a computer readable storage medium that, when executed, includes one or a combination of the steps of the method embodiments.

[0141] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0142] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A point cloud data equalization resampling method, characterized in that, The method includes: Calculate the first uniformity of point cloud data to characterize the degree of uniformity in the data distribution; Based on the point cloud data, calculate the first weight value of each target class in the preset target class set; The preset target class set is traversed in descending order of the first weight value to obtain the first current target class. Based on the first current target class, the point cloud data is sampled to obtain a class sample pool. The second weight value of each data frame in the class sample pool is calculated. The second weight value is used as the sampling probability to sample in the class sample pool to obtain the resampled data of the first current target class and save it to the sample set. The sample set includes the resampled data of each target class in the preset target class set. Calculate the second uniformity of the sample set, which characterizes the degree of uniformity of the data distribution; When the second balance is greater than the first balance, the above traversal operation is re-executed to obtain the sample set again; When the second equalization degree is less than or equal to the first equalization degree, the resampling result is obtained based on the sample set.

2. The point cloud data equalization and resampling method as described in claim 1, characterized in that, Calculate the balance of the target data, including: Traverse the preset target class set to obtain the second current target class. Search for the second current target class in each frame of the target data. Accumulate the number of frames in which the second current target class is found to obtain the class frame number of the second current target class. The target data is the point cloud data or the sample set. Calculate the probability of each target class in the preset target class set based on the number of class frames of all target classes in the preset target class set; Based on all the probabilities, the probability distribution of the preset target class set is obtained; The variance of the probability distribution is calculated. When the target data is the point cloud data, the first balance is obtained. When the target data is the sample set, the second balance is obtained.

3. The point cloud data equalization and resampling method as described in claim 1, characterized in that, Based on the point cloud data, the first weight value of the target class is calculated, including: Traverse the preset target class set to obtain the third current target class. Search for the third current target class in each frame of the point cloud data. Accumulate the number of frames in which the third current target class is found to obtain the class frame number of the third current target class. Based on the number of class frames of all target classes in the preset target class set, calculate the importance evaluation value of each target class in the preset target class set, and the importance evaluation value is inversely correlated with the number of class frames; The first weight value is obtained based on the importance assessment value of the target class.

4. The point cloud data equalization and resampling method as described in claim 3, characterized in that, Obtaining the first weight value based on the importance evaluation value of the target class includes: Based on the importance evaluation values ​​of all target classes in the preset target class set, the importance evaluation values ​​of the target classes are normalized to obtain the first weight value.

5. The point cloud data equalization and resampling method as described in claim 1, characterized in that, Calculating the second weight value of the data frames in the sample pool includes: Enumerate the target classes appearing in the data frame to obtain the class target set; Remove the first current target class from the set of target classes; The first weight value of each target class in the target class set is accumulated to obtain the second weight value of the data frame and then normalized.

6. The point cloud data equalization and resampling method as described in claim 1, characterized in that, Sampling is performed in the sample pool of the class to obtain resampled data of the first current target class, including: The number of samples is determined based on the number of frames in the sample set that contain the first current target class; Based on the stated number of samples, samples are taken from the class sample pool to obtain resampled data for the first current target class.

7. The point cloud data equalization and resampling method as described in claim 6, characterized in that, The step of determining the sampling quantity based on the number of frames in the sample set containing the first current target class includes: When the total number of data frames in the sample set is less than the preset average number of frames, the preset average number of frames is set as the sampling number; Otherwise, the number of frames in the sample set containing the first current target class is accumulated to obtain the number of frames sampled. When the number of frames sampled is less than the preset average number of frames, the difference between the preset average number of frames and the number of frames sampled is set as the sampling quantity.

8. The point cloud data equalization and resampling method as described in claim 1, characterized in that, The step of obtaining the resampling result based on the sample set includes: Save the sample set to the sampling pool; Repeat the above traversal operation to obtain the next sample set, calculate the balance of the next sample set, obtain the third balance, and when the third balance is less than the first balance, save the next sample set to the sampling pool until the number of sample sets in the sampling pool is equal to the preset number of schemes. The sample set with the lowest balance in the sampling pool is set as the resampling result.

9. A point cloud data equalization and resampling device, characterized in that, The device includes: The balance calculation module is used to calculate the first balance of point cloud data and the second balance of sample set. Both the first balance and the second balance are used to characterize the degree of uniform distribution of data. The class importance module is used to calculate the first weight value of each target class in the preset target class set based on the point cloud data; The sample set module is used to traverse the preset target class set in descending order of the first weight value to obtain the first current target class, sample the point cloud data according to the first current target class to obtain the class sample pool, calculate the second weight value of each data frame in the class sample pool, use the second weight value as the sampling probability to sample the class sample pool, obtain the resampled data of the first current target class and save it to the sample set, wherein the sample set includes the resampled data of each target class in the preset target class set; The result module is used to re-execute the above traversal operation to re-obtain the sample set when the second balance degree is greater than the first balance degree; and to obtain the resampling result based on the sample set when the second balance degree is less than or equal to the first balance degree.

10. The point cloud data equalization and resampling device as described in claim 9, characterized in that, The balance calculation module includes a traversal unit and a balance unit. The traversal unit is used to traverse the preset target class set, obtain the second current target class, search for the second current target class in each frame of the point cloud data or the sample set, accumulate the number of frames in which the second current target class is found, and obtain the class frame number of the second current target class. The balance unit is used to calculate the probability of each target class in the preset target class set based on the class frame number of all target classes in the preset target class set, obtain the probability distribution of the preset target class set based on all the probabilities, calculate the variance of the probability distribution, and obtain the balance.

11. The point cloud data equalization and resampling device as described in claim 9, characterized in that, The class importance module includes a traversal unit and a weight value unit. The traversal unit is used to traverse the preset target class set, obtain the third current target class, search for the third current target class in each frame of the point cloud data, accumulate the number of frames in which the third current target class is found, and obtain the class frame number of the third current target class. The weight value unit is used to calculate the importance evaluation value of each target class in the preset target class set based on the class frame number of all target classes in the preset target class set. The importance evaluation value is inversely correlated with the class frame number. Based on the importance evaluation value of the target class, the first weight value is obtained.

12. The point cloud data equalization and resampling device as described in claim 9, characterized in that, The sample set module includes a frame importance unit, which is used to enumerate the target classes appearing in the data frames in the class sample pool, obtain a class target set, remove the first current target class from the class target set, accumulate the first weight value of each target class in the class target set, obtain the second weight value of the data frame, and perform normalization processing.

13. The point cloud data equalization and resampling device as described in claim 9, characterized in that, The sample set module includes a sampling quantity unit, which is used to set the preset average frame number as the sampling quantity when the total number of data frames in the sample set is less than the preset average frame number. Otherwise, the number of frames in the sample set containing the first current target class is accumulated to obtain the number of frames sampled. When the number of frames sampled is less than the preset average number of frames, the difference between the preset average number of frames and the number of frames sampled is set as the sampling quantity.

14. The point cloud data equalization and resampling device as described in claim 9, characterized in that, It also includes a sampling pool module, which is used to store the sample set. When the number of sample sets in the sampling pool is less than the preset number of schemes, the above traversal operation is re-executed to obtain the next sample set, calculate the balance degree of the next sample set, and obtain the third balance degree. When the third balance degree is less than the first balance degree, the next sample set is saved to the sampling pool until the number of sample sets in the sampling pool is equal to the preset number of schemes. The sample set with the smallest balance degree in the sampling pool is set as the resampling result.

15. A target detection device for autonomous driving, characterized in that, The target detection device includes a memory, a processor, and a target detection perception model stored in the memory and capable of running on the processor. The training samples used during the training of the target detection perception model are sample data obtained by performing the point cloud data equalization and resampling method as described in any one of claims 1-8.

16. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a point cloud data equalization and resampling program stored in the memory and executable on the processor. When the point cloud data equalization and resampling program is executed by the processor, it implements the steps of the point cloud data equalization and resampling method as described in any one of claims 1-8.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a point cloud data equalization and resampling program, which, when executed by a processor, implements the steps of the point cloud data equalization and resampling method as described in any one of claims 1-8.

Images