Sample data enhancement method and device, target classification method and device and electronic equipment

Abstract

The invention relates to a sample data enhancement method and device, a target classification method and device, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: fusing sample point cloud data belonging to the same target in multiple frames of sample point cloud data to obtain fused point cloud data; randomly sampling the fused point cloud data to generate at least one sampled point cloud data; according to the point cloud attribute information of each piece of sampled point cloud data, determining a label confidence coefficient corresponding to each piece of sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of the point cloud data; according to the sampled point cloud data and the label confidence corresponding to the sampled point cloud data, obtaining sample training data to train a target classification model; the target classification model is used for performing target classification on the acquired point cloud data. The method can improve the classification recognition precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a sample data enhancement method, a target classification method, an apparatus, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] With the development of computer technology, target classification detection is a research hotspot in the field of computer technology. The main purpose is to distinguish different categories of targets according to different characteristics reflected in each data, and correct identification of the target is a key task to realize machine intelligence.

[0003] In related technologies, a classification model is usually used for target classification detection. However, in the process of model training in related technologies, different categories of target data information are often translated by coordinates, mirrored, copied, and enhanced by generating virtual data such as synthetic data to expand and enrich different categories of data sets. Although this data enhancement method can enhance the sample size of different categories, the diversity and reliability of the generated data are not high, and it cannot effectively improve the classification recognition accuracy of the classification model.

[0004] Therefore, there is a problem of low classification recognition accuracy in related technologies. SUMMARY

[0005] Therefore, it is necessary to provide a sample data enhancement method, a target classification method, an apparatus, an electronic device, a storage medium and a computer program product to solve the above technical problems.

[0006] In a first aspect, the present application provides a sample data enhancement method, comprising:

[0007] Fusing sample point cloud data belonging to the same target in multiple frames of sample point cloud data to obtain fused point cloud data;

[0008] Sampling processing the fused point cloud data to generate at least one sampled point cloud data;

[0009] Determining the label confidence of each sampled point cloud data according to the point cloud attribute information of each sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of the point cloud data;

[0010] Obtaining sample training data according to each sampled point cloud data and the label confidence corresponding to each sampled point cloud data to train a target classification model; the target classification model is used for target classification of the collected point cloud data.

[0011] In a second aspect, the present application provides a target classification method, comprising:

[0012] acquire a plurality of frames of point cloud data collected in a target space;

[0013] input the plurality of frames of point cloud data into a trained target classification model, perform target classification processing on the plurality of frames of point cloud data by using the trained target classification model, and obtain the category of each target in the target space; the trained target classification model is obtained by training sample training data; the sample training data is obtained by using the sample data enhancement method in any one of claims 1 to 7.

[0014] In a third aspect, the present application also provides a sample data enhancement device, comprising:

[0015] a fusion module configured to fuse sample point cloud data belonging to the same target in the plurality of frames of sample point cloud data, and obtain fused point cloud data;

[0016] a sampling module configured to perform sampling processing on the fused point cloud data, and generate at least one sampled point cloud data;

[0017] a confidence determination module configured to determine the label confidence corresponding to each sampled point cloud data according to the point cloud attribute information of each sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of point cloud data;

[0018] a sample construction module configured to obtain sample training data according to each sampled point cloud data and the label confidence corresponding to each sampled point cloud data, and train a target classification model; the target classification model is used for target classification of the collected point cloud data.

[0019] In one embodiment, the point cloud attribute information includes point cloud attribute values; the confidence determination module is specifically configured to obtain the point cloud attribute values corresponding to the fused point cloud data, and obtain the fused point cloud attribute values; for any one of the sampled point cloud data, the label confidence corresponding to the any one of the sampled point cloud data is obtained according to the ratio between the point cloud attribute values corresponding to the any one of the sampled point cloud data and the fused point cloud attribute values.

[0020] In one embodiment, when the point cloud attribute information includes point cloud attribute values corresponding to at least two point cloud attribute types, the confidence determination module is specifically configured to obtain the fused point cloud attribute values corresponding to each point cloud attribute type of the fused point cloud data; and the label confidence corresponding to the any one of the sampled point cloud data is obtained according to the ratio between the point cloud attribute values corresponding to each point cloud attribute type of the any one of the sampled point cloud data and the fused point cloud attribute values corresponding to the corresponding point cloud attribute type.

[0021] In one of the embodiments, the sample training data includes sampled point cloud data and corresponding label confidence; the sample data enhancement device further includes a training module configured to input the sampled point cloud data into a target classification model to be trained, perform target classification on the sampled point cloud data by the target classification model to be trained, obtain a category prediction result corresponding to the sampled point cloud data, obtain a loss value according to the category prediction result and the label confidence corresponding to the sampled point cloud data, adjust model parameters of the target classification model to be trained according to the loss value and continue iterative training until a training end condition is met to stop training, and obtain a trained target classification model.

[0022] In one of the embodiments, the sampling module is specifically configured to perform random sampling processing on the point cloud in the fused point cloud data according to a preset sampling rate to obtain the at least one sampled point cloud data.

[0023] In one of the embodiments, the sampling module is specifically configured to perform random frame loss processing on the point cloud in the fused point cloud data according to a preset frame number ratio to obtain frame loss point cloud data, and take the frame loss point cloud data as the sampled point cloud data.

[0024] In one of the embodiments, the fusion module is specifically configured to fuse sample point cloud data belonging to the same target according to a preset frame number and a preset sliding step to obtain a plurality of the fused point cloud data belonging to the same target.

[0025] In a fourth aspect, the present application further provides a target classification device, including:

[0026] an acquisition module configured to acquire a plurality of frames of point cloud data collected in a target space;

[0027] a classification module configured to input the plurality of frames of point cloud data into a trained target classification model, perform target classification processing on the plurality of frames of point cloud data by the target classification model, and obtain categories of targets in the target space; the trained target classification model is obtained by training according to sample training data; the sample training data is obtained according to the sample data enhancement method.

[0028] In one of the embodiments, the categories of targets include human bodies; the target classification device further includes a control module configured to, if the categories of targets are human bodies and positions of the targets meet a trigger condition in an automatic control scheme, instruct a corresponding target device to perform a target action according to the automatic control scheme.

[0029] In a fifth aspect, the present application also provides an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to implement the steps of the method described above.

[0030] In a sixth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method described above.

[0031] In a seventh aspect, the present application also provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the steps of the method described above.

[0032] The sample data enhancement method, device, electronic device, storage medium and computer program product described above, by fusing sample point cloud data belonging to the same target in multiple frames of sample point cloud data, obtaining fused point cloud data; sampling the fused point cloud data to generate at least one sampled point cloud data; determining the label confidence corresponding to each sampled point cloud data according to the point cloud attribute information of each sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of the point cloud data; obtaining sample training data according to each sampled point cloud data and the label confidence corresponding to each sampled point cloud data, to train a target classification model; the target classification model is used for target classification of the collected point cloud data.

[0033] In this way, by fusing sample point cloud data belonging to the same target, and by randomly sampling the fused point cloud data, at least one sampled point cloud data can be generated for the same target, so as to enhance the sample data amount of different categories of targets, improve the balance of the sample quantity, generate more training samples, avoid overfitting of the target classification model, and effectively improve the classification accuracy and generalization of the target classification model. Moreover, by using attribute information related to the reliability of the point cloud data, the label confidence corresponding to each sampled point cloud data can be more accurately determined, so that the label confidence can be added in the model training process, the overfitting problem in the training model can be reduced, and the classification performance of the model can be further improved, effectively improving the classification recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0035] Figure 1 An application environment diagram of a sample data enhancement method in an embodiment;

[0036] Figure 2 A flowchart of a sample data enhancement method in an embodiment;

[0037] Figure 3 An overall training architecture diagram of a target classification model in an embodiment;

[0038] Figure 4 An architecture diagram of a random sampling processing method in an embodiment;

[0039] Figure 5 A flowchart of a sample data enhancement method in another embodiment;

[0040] Figure 6 A flowchart of a target classification method in an embodiment;

[0041] Figure 7 A structural block diagram of a sample data enhancement device in an embodiment;

[0042] Figure 8 A structural block diagram of a target classification device in an embodiment;

[0043] Figure 9 An internal structure diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0045] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or a chronological sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0046] The sample data enhancement method provided in the application can be applied to an electronic device, for example, the electronic device can be a user terminal, a server, and the like. The sample data enhancement method in the embodiments of the application can be applied to various Internet of Things scenarios, for example, the Internet of Things scenarios include but are not limited to smart home scenarios, office scenarios, and the like. An application environment related to the application will be introduced below.

[0047] In one embodiment, Figure 1 A schematic diagram of an implementation environment related to the sample data enhancement method and the target classification method can be provided. The implementation environment at least includes a user terminal 110, a smart device 130, a server 170, and a network device. In the implementation environment, Figure 1 The network device includes a gateway 150 and a router 190, which are not specifically limited herein.

[0048] The user terminal 110, which can also be referred to as a user end or a terminal, can be used to deploy (or install) a client associated with the smart device 130. The user terminal 110 can be an electronic device such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart control panel, or another device with display and control functions, which are not specifically limited herein.

[0049] The client is associated with the smart device 130. In other words, a user registers an account in the client and configures the smart device 130 in the client. For example, the configuration includes adding a device identifier to the smart device 130, so that when the client runs in the user terminal 110, the user terminal 110 can provide device display and device control functions of the smart device 130 for the user. The client can be in the form of an application program or a web page. Correspondingly, the interface for device display of the client can be in the form of a program window or a web page, which are not specifically limited herein.

[0050] The smart device 130 is deployed in the gateway 150 and communicates with the gateway 150 through a communication module configured by itself, and is controlled by the gateway 150. It should be understood that the smart device 130 generally refers to one of a plurality of smart devices 130, and the embodiments of the present application are only exemplified by the smart device 130, that is, the number and type of smart devices deployed in the gateway 150 are not limited in the embodiments of the present application. In one application scenario, the smart device 130 accesses the gateway 150 through a local area network, so as to be deployed in the gateway 150. The process of the smart device 130 accessing the gateway 150 through the local area network includes: first establishing a local area network by the gateway 150, and the smart device 130 connects the gateway 150 to join the local area network established by the gateway 150. The local area network includes but is not limited to ZIGBEE or Bluetooth. The smart device 130 can be a smart printer, a smart fax machine, a smart camera, a smart air conditioner, a smart door lock, a smart lamp, or a human body sensor, a door and window sensor, a temperature and humidity sensor, a water immersion sensor, a natural gas alarm, a smoke alarm, a wall switch, a wall socket, a wireless switch, a wireless wall switch, a magic cube controller, a curtain motor, a millimeter wave radar, etc. configured with a communication module.

[0051] The interaction between the user terminal 110 and the smart device 130 can be realized through a local area network, and can also be realized through a wide area network. In one application scenario, the user terminal 110 establishes a wired or wireless communication connection between the router 190 and the gateway 150, for example, the wired or wireless communication connection includes but is not limited to WIFI, etc., so that the user terminal 110 and the gateway 150 are deployed in the same local area network, and then the user terminal 110 can realize the interaction with the smart device 130 through a local area network path. In another application scenario, the user terminal 110 establishes a wired or wireless communication connection between the server 170 and the gateway 150, for example, the wired or wireless communication connection includes but is not limited to 2G, 3G, 4G, 5G, WIFI, etc., so that the user terminal 110 and the gateway 150 are deployed in the same wide area network, and then the user terminal 110 can realize the interaction with the smart device 130 through a wide area network path.

[0052] The server 170 can also be referred to as a cloud, a cloud platform, a platform, a server, etc. The server 170 can be a server or a server cluster composed of multiple servers, or a cloud computing center composed of multiple servers, so as to better provide background services to a large number of user terminals 110. For example, the background services include target classification services.

[0053] In one application scenario, the server end 170 can fuse sample point cloud data belonging to the same target in multiple frames of sample point cloud data to obtain fused point cloud data; perform sampling processing on the fused point cloud data to generate at least one sampled point cloud data; determine the label confidence corresponding to each sampled point cloud data according to the point cloud attribute information of each sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of the point cloud data; obtain sample training data according to each sampled point cloud data and the label confidence corresponding to each sampled point cloud data, to train a target classification model; the target classification model is used for target classification on the collected point cloud data.

[0054] Of course, in other application scenarios, the above-mentioned sample data enhancement process completed by the server end 170 can also be implemented by the user terminal 110.

[0055] In one embodiment, as shown in Figure 2 , a sample data enhancement method is provided, and the embodiment takes the method applied to an electronic device as an example. As described above, the electronic device can be a user terminal, a server, etc., for example, the electronic device can specifically be the user terminal 110 or the server end 170 in Figure 1 . In the embodiment, the method includes the following steps:

[0056] Step S210, fuse sample point cloud data belonging to the same target in multiple frames of sample point cloud data to obtain fused point cloud data.

[0057] Point cloud data is a data structure used to represent a large number of points in three-dimensional space. Each point contains its coordinate information in three-dimensional space, and usually also includes other attributes such as color, normal, etc. Point cloud data can usually be collected by devices such as laser radar, millimeter wave radar, or camera, and is widely used in target detection, target recognition, computer vision, robotics, etc. Point cloud data can also be used to identify targets in space, to achieve target positioning and classification, etc.

[0058] Sample point cloud data refers to a small set of points extracted from the entire point cloud data set. When dealing with large-scale point cloud data, the entire data set is usually first sampled to select a representative part of the points as sample point cloud data to reduce the amount of calculation and improve processing efficiency. Sample point cloud data usually retains the main features and structure of the original point cloud data and can be used for various point cloud processing tasks such as classification, segmentation, registration, etc. In the field of machine learning, sample point cloud data is also commonly used to train models.

[0059] The sample point cloud data in the embodiment can be point cloud data corresponding to each sample target extracted from the collected point cloud data.

[0060] The scene where the sample target is located is installed with a target detection device, which refers to an instrument for acquiring the geometric information of the surface of an object in the environment. In some embodiments, the target detection device can be a point cloud acquisition device, such as a radar device and a camera device, etc. The radar device can include a laser radar and a millimeter wave radar device, etc. The millimeter wave radar device is a radar system that uses a millimeter wave frequency band for detection and measurement.

[0061] Exemplarily, the scene where the sample target is located can be an indoor scene, and the sample target can be an adult, a child, a floor sweeping machine, a pet, a green plant, or the like in the indoor scene. It should be noted that the type of the scene where the sample target is located and the type of the sample target are not specifically limited herein.

[0062] The sample point cloud data can be acquired by the target detection device, and the electronic device can acquire the multiple frames of sample point cloud data acquired by the target detection device.

[0063] The sample point cloud data can be pre-stored in the memory of the electronic device, and the electronic device can directly acquire the sample point cloud data from the memory when needed. The specific method of acquiring the sample point cloud data is not limited in the present application, as long as the function can be realized.

[0064] In specific implementation, the electronic device can acquire multiple frames of sample point cloud data based on the target detection device, and the multiple frames of sample point cloud data can include sample point cloud data corresponding to at least one sample target, and each sample target can correspond to at least one frame of sample point cloud data. The electronic device can fuse the sample point cloud data belonging to the same target (i.e., the same sample target) in the multiple frames of sample point cloud data to obtain the fused point cloud data corresponding to each sample target.

[0065] In step S220, the fused point cloud data is subjected to sampling processing to generate at least one sampled point cloud data.

[0066] The sampling processing of the point cloud data can refer to a sampling operation on the point cloud data for the purpose of reducing the data amount, reducing the calculation complexity, or extracting key information, etc. In processing the point cloud data, the sampling processing can help to simplify the data structure, improve the processing efficiency, or extract key information therefrom.

[0067] The number of the sampled point cloud data can match the number of times of the sampling processing.

[0068] In specific implementation, the electronic device can perform sampling processing on the fused point cloud data to generate at least one sampled point cloud data.

[0069] Specifically, for the fused point cloud data corresponding to each sample target, the electronic device can perform sampling processing on the fused point cloud data corresponding to the sample target, and generate at least one sampled point cloud data based on the fused point cloud data corresponding to the sample target.

[0070] In this way, the electronic device can randomly sample the fused point cloud data corresponding to each sample target, so that each fused point cloud data corresponds to at least one sampled point cloud data, thereby increasing the amount of sample data of different categories and increasing the diversity of samples.

[0071] In step S230, the label confidence corresponding to each sampled point cloud data is determined according to the point cloud attribute information of each sampled point cloud data.

[0072] The point cloud attribute information generally refers to the attribute information contained in each point of the point cloud data. The point cloud is a three-dimensional data set composed of a large number of points, and each point can contain different attribute information, which can be used for analysis, processing and visualization of point cloud data. Specifically, the point cloud attribute information can be attribute information related to the reliability of the point cloud data. For example, the point cloud attribute information can include point cloud quantity information, point cloud signal-to-noise ratio information, etc.

[0073] The label confidence refers to a measure of the confidence or reliability of the predicted label result in the field of machine learning. When performing a classification task, the model will predict the input data and output a label as the classification result, and also give a confidence score of the prediction result. The label confidence is usually a probability value or a score value, indicating the confidence of the model for the prediction result. Generally, the higher the confidence value, the more confident the model is about the prediction result; on the contrary, the lower the confidence value, the lower the confidence of the model for the prediction result.

[0074] In practical applications, the label confidence can assist in evaluating the performance and stability of the model, as well as the degree of trust in the model prediction result. In some scenarios, the prediction results with high confidence can be selected according to the label confidence, thereby improving the accuracy and reliability of the model.

[0075] The label confidence corresponding to the sampled point cloud data is positively correlated with the reliability of the sampled point cloud data.

[0076] In a specific implementation, in the process of randomly sampling the fused point cloud data to generate the sampled point cloud data, since the real points may be deleted during random sampling, resulting in the target sample becoming a negative sample, therefore, a confidence can be assigned to the sample category label corresponding to each sampled point cloud data. That is, a corresponding label confidence is set for each sampled point cloud data, and the label confidence can be used to represent the probability that the sampled point cloud data belongs to the corresponding sample category label, so as to guarantee the reliability of the sampled point cloud data. For example, the label confidence corresponding to the sampled point cloud data of the target person (i.e., the sample category label is a person) is 0.8, which means that there is an 80% probability that the sampled point cloud data is a person.

[0077] However, the point cloud attribute information corresponding to different sampled point cloud data may be different, resulting in different reliabilities of different sampled point cloud data, and thus different label confidences corresponding to each sampled point cloud data. Therefore, the electronic device can determine the label confidence corresponding to each sampled point cloud data that is positively correlated with the reliability of the point cloud data according to the point cloud attribute information of each sampled point cloud data that is related to the reliability of the point cloud data. In this way, the higher the reliability of the sampled point cloud data, the higher the label confidence corresponding to it, so that the label confidence corresponding to each sampled point cloud data can be more accurately determined based on the point cloud attribute information related to the reliability of the point cloud data.

[0078] Further, for a certain sampled point cloud data corresponding to any fused point cloud data, the electronic device can determine the label confidence corresponding to the sampled point cloud data according to the point cloud attribute information of the sampled point cloud data and the point cloud attribute information of the any fused point cloud data.

[0079] Step S240, obtaining sample training data according to each sampled point cloud data and the label confidence corresponding to each sampled point cloud data.

[0080] The sample training data is used to train the target classification model.

[0081] The sample training data refers to a data set used to train a model in the field of machine learning. When training a machine learning model, a large amount of sample data needs to be provided as input, and the model learns the characteristics and patterns of these data to perform prediction or classification tasks.

[0082] The sample training data can include two parts: feature data and label data. The feature data describes the attributes or characteristics of the sample and can be in the form of numbers, text, images, audio, etc. The feature data is the basis for model input, and the model learns the relationship between the feature data to make predictions or classifications. Label data is the target output corresponding to feature data, used to guide the model to learn the correct prediction or classification. In supervised learning, each sample has a corresponding label, and the model adjusts its parameters by comparing the predicted results with the label data.

[0083] The quality and quantity of sample training data have a significant impact on the performance and generalization ability of the model. Generally, the more representative the training data is, the better the performance of the model will be. At the same time, the accuracy, diversity and balance of sample training data are also important factors affecting the training effect of the model. In machine learning tasks, the selection, processing and preparation of sample training data are very crucial steps that determine the training effect and generalization ability of the model. By reasonably selecting and processing sample training data, a high-performance machine learning model can be trained.

[0084] In actual applications, the classification algorithm used by the target classification model can be, but is not limited to, SVM (Support Vector Machine), convolutional neural network, and logistic regression algorithm.

[0085] In specific implementations, the electronic device can obtain sample training data according to the sampled point cloud data and the label confidence corresponding to each sampled point cloud data, so as to train the target classification model using the sample training data, thereby adding label confidence in the model training process to reduce the overfitting problem in the trained model. Moreover, the trained target classification model can be used for target classification of the collected point cloud data.

[0086] In the above sample data enhancement method, the sample point cloud data belonging to the same target in the multiple frames of sample point cloud data is fused to obtain fused point cloud data; the fused point cloud data is sampled to generate at least one sampled point cloud data; the label confidence corresponding to each sampled point cloud data is determined according to the point cloud attribute information of each sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of the point cloud data; sample training data is obtained according to the sampled point cloud data and the label confidence corresponding to each sampled point cloud data to train a target classification model; the target classification model is used for target classification of the collected point cloud data.

[0087] Thus, by fusing sample point cloud data belonging to the same target and by randomly sampling the fused point cloud data, at least one sampled point cloud data for the same target can be generated, so as to increase the sample data amount of different categories of targets, improve the balance of the sample quantity, generate more training samples, avoid overfitting of the target classification model, and effectively improve the classification accuracy and generalization of the target classification model. Moreover, by using the attribute information related to the reliability of the point cloud data, the label confidence corresponding to each sampled point cloud data can be more accurately determined, so that the label confidence can be added in the model training process, the overfitting problem in the training model can be reduced, and the classification performance of the model can be further improved, thereby effectively improving the classification recognition accuracy.

[0088] In one embodiment, the point cloud attribute information includes a point cloud attribute value; and the label confidence corresponding to each sampled point cloud data is determined according to the point cloud attribute information of each sampled point cloud data, including: obtaining a point cloud attribute value corresponding to the fused point cloud data to obtain a fused point cloud attribute value; and for any one of the sampled point cloud data, the label confidence corresponding to the any one of the sampled point cloud data is obtained according to a ratio between the point cloud attribute value corresponding to the any one of the sampled point cloud data and the fused point cloud attribute value.

[0089] In one embodiment, the point cloud attribute information includes a point cloud attribute value; and the label confidence corresponding to each sampled point cloud data is determined according to the point cloud attribute information of each sampled point cloud data, including: obtaining a point cloud attribute value corresponding to the fused point cloud data to obtain a fused point cloud attribute value; and for any one of the sampled point cloud data, the label confidence corresponding to the any one of the sampled point cloud data is obtained according to a ratio between the point cloud attribute value corresponding to the any one of the sampled point cloud data and the fused point cloud attribute value.

[0090] In one embodiment, the point cloud attribute information includes a point cloud attribute value; and the label confidence corresponding to each sampled point cloud data is determined according to the point cloud attribute information of each sampled point cloud data, including: obtaining a point cloud attribute value corresponding to the fused point cloud data to obtain a fused point cloud attribute value; and for any one of the sampled point cloud data, the label confidence corresponding to the any one of the sampled point cloud data is obtained according to a ratio between the point cloud attribute value corresponding to the any one of the sampled point cloud data and the fused point cloud attribute value.

[0091] For example, taking the point cloud attribute information as the point cloud signal-to-noise ratio information, that is, the point cloud attribute value is the point cloud signal-to-noise ratio value (SNR value) for description. Because of the noise points and different point cloud signal-to-noise ratios, the ratio of the generated sampled point cloud data to the corresponding fused point cloud data can be calculated by the relationship between the signal-to-noise ratios, and the value is taken as the label confidence of the generated sampled point cloud data.

[0092] It is assumed that the fused point cloud data is obtained by fusing sample point cloud data of continuous T frames (for example, T = 10, and other values are also possible, and the specific value is not specifically limited here) of the same target (combining all sample point cloud data of the same target of T frames). Wherein, the fusion is to combine the sample point cloud data of the same target of continuous T frames, for example, the sample point cloud data of the first frame of the same target + the sample point cloud data of the second frame + … + the sample point cloud data of the tenth frame to obtain the fused point cloud data.

[0093] Wherein, the fused point cloud data can be represented as:

[0094] P = {p0, p1, …, p N},

[0095] Wherein, p0, p1,..., p N respectively represent different points in the fused point cloud data; N is a positive integer, representing the number of point clouds in the fused point cloud data, and the information contained in a point in the fused point cloud data is as follows:

[0096] p i = (x, y, z, snr, v x , v y , v z ), 0≤i≤N,

[0097] Wherein, p i represents a point in the fused point cloud data; x, y, z respectively represent three-dimensional coordinate information of the point in the coordinate system, v x , v y , v z respectively represent the velocity information of the point in the x, y, z directions of the coordinate system, and snr represents the signal-to-noise ratio value of the point.

[0098] For any sampled point cloud data corresponding to the fused point cloud data, the electronic device can sum the signal-to-noise ratio values corresponding to all point clouds in the sampled point cloud data to obtain the point cloud signal-to-noise ratio value corresponding to the sampled point cloud data. Specifically, the following formula is used:

[0099]

[0100] wherein S sum represents the point cloud signal-to-noise ratio value corresponding to the sampled point cloud data, N' represents the number of point clouds in the sampled point cloud data, and snr i represents the point cloud signal-to-noise ratio value corresponding to the i-th point in the sampled point cloud data, 0≤i≤N'.

[0101] wherein the point cloud signal-to-noise ratio value corresponding to the fused point cloud data can be obtained by summing the point cloud signal-to-noise ratio values corresponding to all point clouds in the fused point cloud data. Specifically, the following formula is used:

[0102]

[0103] wherein S org_sum represents the point cloud signal-to-noise ratio value corresponding to the fused point cloud data, snr i ' represents the point cloud signal-to-noise ratio value corresponding to the i-th point in the fused point cloud data.

[0104] In this way, the electronic device can calculate the ratio between the point cloud signal-to-noise ratio value corresponding to the sampled point cloud data and the point cloud signal-to-noise ratio value corresponding to the fused point cloud data, obtain the confidence of the sampled point cloud data belonging to the corresponding sample category label based on the ratio, and obtain the label confidence corresponding to the sampled point cloud data. Therefore, in the case where the point cloud attribute information is point cloud signal-to-noise ratio information, the label confidence (denoted by a) is calculated according to the following formula:

[0105]

[0106] wherein β is the sampling rate. For example, in the present solution, it can be set to 90%. The specific sampling rate value is not specifically limited here.

[0107] The technical solution of the present embodiment is that the point cloud attribute information includes a point cloud attribute value; the fused point cloud attribute value is obtained by obtaining the point cloud attribute value corresponding to the fused point cloud data; for any sampled point cloud data in the sampled point cloud data, the label confidence corresponding to the any sampled point cloud data is obtained according to the ratio between the point cloud attribute value corresponding to the any sampled point cloud data and the fused point cloud attribute value. In this way, by obtaining the label confidence corresponding to the sampled point cloud data according to the ratio between the point cloud attribute value corresponding to the sampled point cloud data and the point cloud attribute value corresponding to the corresponding fused point cloud data, the label confidence corresponding to the sampled point cloud data with a larger point cloud attribute value, i.e., a higher data reliability, is larger, and the label confidence corresponding to each sampled point cloud data is more accurately determined based on the point cloud attribute value.

[0108] In an embodiment, in a case where the point cloud attribute information includes point cloud attribute values corresponding to at least two point cloud attribute types, the electronic device obtains a point cloud attribute value of the fused point cloud data to obtain a fused point cloud attribute value, including: obtaining a point cloud attribute value corresponding to each point cloud attribute type of the fused point cloud data.

[0109] According to a ratio between the point cloud attribute value corresponding to any sampled point cloud data and the fused point cloud attribute value, the electronic device obtains a label confidence corresponding to any sampled point cloud data, including: according to a ratio between the point cloud attribute value corresponding to any sampled point cloud data in each point cloud attribute type and the fused point cloud attribute value corresponding to the corresponding point cloud attribute type, the electronic device obtains a label confidence corresponding to any sampled point cloud data.

[0110] In an embodiment, the point cloud attribute value can be a specific value of a point cloud attribute related to point cloud data reliability. The point cloud data can have corresponding values under different types of point cloud attributes. As an optional embodiment of the present application, the point cloud attribute value of the point cloud data can include point cloud attribute values corresponding to at least two point cloud attribute types. For example, the point cloud quantity value and the point cloud signal-to-noise ratio value of the point cloud data are point cloud attribute values corresponding to two point cloud attribute types (point cloud quantity and point cloud signal-to-noise ratio, respectively). The point cloud quantity can include at least one of dynamic point cloud quantity and static point cloud quantity. It can be understood that other point cloud attributes related to point cloud data reliability can also be used for label confidence calculation, which is not limited in the present application.

[0111] In an embodiment, the point cloud attribute value can be a specific value of a point cloud attribute related to point cloud data reliability. The point cloud data can have corresponding values under different types of point cloud attributes. As an optional embodiment of the present application, the point cloud attribute value of the point cloud data can include point cloud attribute values corresponding to at least two point cloud attribute types. For example, the point cloud quantity value and the point cloud signal-to-noise ratio value of the point cloud data are point cloud attribute values corresponding to two point cloud attribute types (point cloud quantity and point cloud signal-to-noise ratio, respectively). The point cloud quantity can include at least one of dynamic point cloud quantity and static point cloud quantity. It can be understood that other point cloud attributes related to point cloud data reliability can also be used for label confidence calculation, which is not limited in the present application.

[0112] Further, in a process of obtaining a label confidence corresponding to any sampled point cloud data according to a ratio between the point cloud attribute value corresponding to any sampled point cloud data and the fused point cloud attribute value, the electronic device can obtain a label confidence corresponding to the any sampled point cloud data according to a ratio between the point cloud attribute value corresponding to the any sampled point cloud data in each point cloud attribute type and the fused point cloud attribute value corresponding to the corresponding point cloud attribute type. In this way, for each of the at least one sampled point cloud data corresponding to the fused point cloud data, the electronic device can obtain a label confidence corresponding to each sampled point cloud data based on the same method.

[0113] Further, in one embodiment, the label confidence corresponding to any sampled point cloud data is obtained according to a ratio of the point cloud attribute value corresponding to any sampled point cloud data at each point cloud attribute type to the fusion point cloud attribute value corresponding to the corresponding point cloud attribute type, including: obtaining the label confidence corresponding to any sampled point cloud data at each point cloud attribute type according to a ratio of the point cloud attribute value corresponding to any sampled point cloud data at each point cloud attribute type to the fusion point cloud attribute value corresponding to the corresponding point cloud attribute type, and obtaining the label confidence corresponding to any sampled point cloud data according to an average value of the label confidence corresponding to any sampled point cloud data at all point cloud attribute types.

[0114] In a specific implementation, the electronic device can use a simple average method or a weighted average method in calculating the average value of the label confidence corresponding to any sampled point cloud data at all point cloud attribute types, which is not limited here. In the case of using a weighted average, each point cloud attribute type can have a corresponding attribute weight, and the electronic device can determine the weighted average value of the label confidence corresponding to any sampled point cloud data at all point cloud attribute types according to the attribute weight corresponding to each point cloud attribute type, and finally obtain the label confidence corresponding to the sampled point cloud data.

[0115] The technical solution of the embodiment, in the case that the point cloud attribute information includes point cloud attribute values corresponding to at least two point cloud attribute types, obtains the fusion point cloud attribute value corresponding to each point cloud attribute type of the fused point cloud data, and obtains the label confidence corresponding to any sampled point cloud data according to a ratio of the point cloud attribute value corresponding to any sampled point cloud data at each point cloud attribute type to the fusion point cloud attribute value corresponding to the corresponding point cloud attribute type. In this way, the ratio of the point cloud attribute value of the sampled point cloud data at each point cloud attribute type to the fusion point cloud attribute value corresponding to the corresponding point cloud attribute type is calculated, so that the point cloud attribute values of the sampled point cloud data at multiple point cloud attribute types can be comprehensively evaluated, the data reliability of the sampled point cloud data can be accurately evaluated, and the label confidence corresponding to the sampled point cloud data can be more accurately determined.

[0116] In an embodiment, the sample training data comprises sampled point cloud data and corresponding label confidence; the method further comprises: inputting the sampled point cloud data into the target classification model to be trained, performing target classification on the sampled point cloud data by the target classification model to be trained to obtain a category prediction result corresponding to the sampled point cloud data; obtaining a loss value according to the category prediction result and the label confidence corresponding to the sampled point cloud data; adjusting the model parameters of the target classification model to be trained according to the loss value, and continuing to perform iterative training until the training is stopped when a training end condition is met, to obtain the trained target classification model.

[0117] The training end condition is a training end condition for the target classification model. In some embodiments, the training end condition can be that the loss value reaches a preset threshold or the number of training iterations reaches a preset number, and the training end condition is not specifically limited here.

[0118] In a specific implementation, each sample training data can comprise sampled point cloud data and label confidence corresponding to the sampled point cloud data. During the training of the target classification model, the electronic device can input the sampled point cloud data into the target classification model to be trained to obtain a category prediction result corresponding to the sampled point cloud data, and the electronic device can obtain a loss value based on the category prediction result corresponding to the sampled point cloud data and the corresponding label confidence. In this way, the electronic device can adjust the model parameters of the target classification model to be trained based on the loss value, and continue to perform iterative training until the training is stopped when a training end condition is met, to obtain the trained target classification model.

[0119] In actual application, after the electronic device adjusts the model parameters of the target classification model to be trained to obtain an adjusted target classification model, the electronic device can return to the step of inputting the sampled point cloud data into the target classification model to be trained, retrain the model parameters of the adjusted target classification model, until the trained target classification model meets the training end condition, to obtain the trained target classification model, in the case that the adjusted target classification model does not meet the training end condition.

[0120] In actual application, the target classification model can adopt a PointNet network structure (a deep network architecture suitable for processing three-dimensional unordered point sets). When the PointNet network structure is adopted, the overall training architecture diagram of the target classification model is as follows: Figure 3As shown, it includes a first multi-layer perceptron (MLP) unit 310, a feature transform unit 320, a second multi-layer perceptron unit 330, a global pooling unit 340, and a fully connected (FC) unit 350. The information contained in each point in the sampled point cloud data can include (x, y, z, snr, v x , v y , v z . In this way, the sampled point cloud data as sample training data can be expressed in the form of an n x 7 vector (n represents the number of point clouds in one sampled point cloud data, and 7 represents (x, y, z, snr, v x , v y , v z )), then undergo dimensionality elevation processing by the first multi-layer perceptron unit 310 to obtain dimensionally elevated sampled point cloud data expressed in the form of an n x 64 vector, and input to the feature transform unit 320; the same affine transformation is applied to each point in the dimensionally elevated sampled point cloud data by the feature transform unit 320 to obtain transformed sampled point cloud data expressed in the form of an n x 64 vector, and input to the second multi-layer perceptron unit 330; the dimensionally elevated sampled point cloud data is processed by the second multi-layer perceptron unit 330 to obtain dimensionally elevated sampled point cloud data expressed in the form of an n x 1024 vector, and input to the global pooling unit 340; then a one-dimensional feature vector is obtained by the global pooling unit 340 and the fully connected unit 350, and finally the output one-dimensional feature vector is taken as the category prediction result corresponding to the input sampled point cloud data, combined with the corresponding sample category label and the corresponding confidence ratio, and input into a classification loss function (Lcls). According to the output loss function value, the target classification model is continuously trained and iterated to generate a target classification model that meets the training end condition. The label confidence corresponding to each sampled point cloud data includes the sample category label corresponding to the sampled point cloud data and the corresponding confidence value.

[0121] The loss function can use a cross-entropy loss function, and the model parameters are adjusted by learning rate adjustment and gradient descent optimization in each iteration.

[0122] In the process of calculating the loss function using label confidence, samples with low confidence can be assigned smaller weights, thereby reducing the model's overlearning of noisy samples and reducing the risk of overfitting. This allows the model to focus more on the classification of samples with high confidence, thus better learning and understanding these key samples and improving the classification accuracy of key samples. This enables the model to learn the distribution patterns of the data more accurately and improve the overall classification performance of the model.

[0123] Therefore, by combining the label confidence scores of the sampled point cloud data to calculate the loss function, the model can better learn and understand key samples, reduce the model's overlearning of noisy samples, and thus improve the model's performance and generalization ability.

[0124] In one embodiment, sampling processing is performed on the fused point cloud data to generate at least one sampled point cloud data, including: randomly sampling the point cloud in the fused point cloud data according to a preset sampling rate to obtain at least one sampled point cloud data.

[0125] In practice, during the process of sampling the fused point cloud data to generate at least one sampled point cloud data, the electronic device can randomly sample the point cloud in the fused point cloud data according to a preset sampling rate to obtain at least one sampled point cloud data.

[0126] Specifically, such as Figure 4 As shown, an architecture diagram of a random sampling method using a preset sampling rate is provided. Figure 4 Taking the fusion of 10 frames of sample point cloud data belonging to the same target as an example, the electronic device can randomly extract points β (sampling rate, which can be set to 90%) from the point cloud set of the fused point cloud data to reconstruct a new sample, which is then used as the sampled point cloud data. A new sample is generated each time a sample is taken from the fused point cloud data. Multiple new samples can be generated by setting the number of random samplings; that is, a new sample is generated each time a sample is extracted from the point cloud set of the fused point cloud data, resulting in a data-enhanced sample set.

[0127] In practical applications, electronic devices can perform the same operation on the fused point cloud data corresponding to each sample target to generate a large number of sampled point cloud data corresponding to different categories, thereby increasing the amount of data for different categories, increasing the diversity of samples, and improving the accuracy of target classification.

[0128] In addition, the electronic device can sample one fused point cloud data each time or multiple fused point cloud data at the same time in the random sampling process. For example, the electronic device can perform random sampling processing on the fused point cloud data corresponding to multiple sample targets at the same time.

[0129] The technical solution of the embodiment can obtain at least one sampled point cloud data by performing random sampling processing on the point cloud in the fused point cloud data according to the preset sampling rate. In this way, the fused point cloud data corresponding to different targets is randomly sampled according to the preset sampling rate to generate the sampled point cloud data, which can effectively enhance the data quantity of different categories, increase the diversity of samples, especially for the categories with less quantity and difficult to collect, generate more sample data, avoid the problem of overfitting of the classification model, and improve the target classification accuracy.

[0130] In one embodiment, the sampling processing on the fused point cloud data to generate at least one sampled point cloud data includes: performing random frame dropping processing on the point cloud in the fused point cloud data according to a preset frame number ratio to obtain frame-dropped point cloud data; and taking the frame-dropped point cloud data as the sampled point cloud data.

[0131] In the specific implementation, in the process of performing random sampling on the fused point cloud data to generate at least one sampled point cloud data, the electronic device can perform random frame dropping processing on the point cloud in the fused point cloud data according to a preset frame number ratio to obtain frame-dropped point cloud data. In this way, the frame-dropped point cloud data can be taken as the sampled point cloud data.

[0132] In the specific implementation, in the process of performing random sampling on the fused point cloud data to generate at least one sampled point cloud data, the electronic device can perform random frame dropping processing on the point cloud in the fused point cloud data according to a preset frame number ratio to obtain frame-dropped point cloud data. In this way, the frame-dropped point cloud data can be taken as the sampled point cloud data.

[0133] It should be noted that in the process of the electronic device performing random frame loss processing on the point cloud in the fused point cloud data according to the preset frame number ratio to obtain the point cloud data after frame loss, the electronic device can specifically perform random frame loss processing on the multiple frames of sample point cloud data that meet the fusion frame number according to the preset frame number ratio, fuse the sample point cloud data of the remaining frames, and obtain the point cloud data after frame loss as the sampled point cloud data. For example, continuing the above example, the electronic device can randomly discard 2 frames from the 10 frames of sample point cloud data to be fused, and fuse the remaining 8 frames of sample point cloud data to obtain the point cloud data after frame loss.

[0134] It can be understood that the electronic device can also discard sample point cloud data that meets the preset frame number ratio from the fused point cloud data obtained by fusion according to the fusion frame number to obtain the point cloud data after frame loss as the sampled point cloud data. Specifically, continuing the above example, for the 10 frames of fused point cloud data obtained by fusion, the electronic device can randomly discard 2 frames of sample point cloud data from the fused point cloud data to obtain the point cloud data after frame loss.

[0135] In the fused point cloud data, each point corresponds to frame index information, and the sample point cloud data to which each point in the fused point cloud data belongs can be determined based on the frame index information, so that 2 frames of sample point cloud data can be randomly discarded from the fused point cloud data.

[0136] In addition, in the process of random frame loss, each time of frame loss can be for one fused point cloud data, or for multiple fused point cloud data at the same time, for example, the fused point cloud data corresponding to multiple sample targets can be simultaneously subjected to random frame loss processing. The number of fused point cloud data selected each time of random frame loss is not specifically limited herein.

[0137] The technical scheme of the embodiment, by performing random frame loss processing on the point cloud in the fused point cloud data according to the preset frame number ratio, obtains the fused point cloud data after frame loss; and taking the fused point cloud data after frame loss as the sampled point cloud data. In this way, by performing random frame loss processing on the point cloud in the fused point cloud data according to the preset frame number ratio to generate the sampled point cloud data, the data amount of different categories can be effectively enhanced, the diversity of samples can be increased, and the target classification accuracy can be improved.

[0138] In one embodiment, the sample point cloud data belonging to the same target in the multiple frames of sample point cloud data is fused to obtain the fused point cloud data, including: fusing the sample point cloud data belonging to the same target according to a preset frame number and a preset sliding step to obtain multiple fused point cloud data belonging to the same target.

[0139] The preset frame number is the fusion frame number.

[0140] In specific implementation, in the process of fusing sample point cloud data belonging to the same target in multiple frames of sample point cloud data to obtain fused point cloud data, the electronic device can fuse sample point cloud data belonging to the same target according to the preset frame number and the preset sliding step to obtain multiple fused point cloud data belonging to the same target.

[0141] In the fusion process, a sliding window fusion can be specifically used, the number of sample point cloud data in the sliding window is the preset frame number, and the step of the sliding window is the preset sliding step. For multiple frames of sample point cloud data belonging to the same target, the sliding window starts from the start position of the multiple frames of sample point cloud data, takes sample point cloud data satisfying the preset frame number in the current window for fusion, after the data in the current window is fused, the sliding window moves backward by the preset sliding step, and then takes sample point cloud data satisfying the preset frame number for fusion, until the sliding window moves to the end of the multiple frames of sample point cloud data.

[0142] Taking 10 frames as the preset frame number and 1 as the preset sliding step as an example, for 12 frames of sample point cloud data of the same target, the electronic device can fuse sample point cloud data of 1-10 frames, 2-11 frames and 3-12 frames of the same target to obtain three fused point cloud data belonging to the same target.

[0143] It can be understood that each target can correspond to multiple fused point cloud data, and each fused point cloud data can be subjected to sampling processing, so that each fused point cloud data can correspond to multiple sampled point cloud data, so as to perform target classification model training based on the sampled point cloud data corresponding to each target.

[0144] The technical scheme of the embodiment fuses sample point cloud data belonging to the same target according to the preset frame number and the preset sliding step to obtain multiple fused point cloud data belonging to the same target. In this way, each target can correspond to multiple fused point cloud data, so that random sampling can be performed based on the fused point cloud data corresponding to each target to generate a large amount of sampled point cloud data obtained based on the fused point cloud data corresponding to each target, so that each target can correspond to a large amount of sampled point cloud data, thereby effectively enhancing the data amount of different categories and increasing the diversity of samples.

[0145] In another embodiment, as shown in Figure 5 , a sample data enhancement method is provided, which is applied to Figure 1 an electronic device as an example for illustration, including the following steps:

[0146] In step S502, the sample point cloud data belonging to the same target is fused according to the preset frame number and the preset sliding step in the multi-frame sample point cloud data, and a plurality of fused point cloud data belonging to the same target is obtained.

[0147] In step S504, the fused point cloud data is sampled to generate at least one sampled point cloud data.

[0148] In step S506, the point cloud attribute value corresponding to the fused point cloud data is obtained to obtain a fused point cloud attribute value.

[0149] In step S508, for any one of the sampled point cloud data, a label confidence corresponding to the any one of the sampled point cloud data is obtained according to a ratio between a point cloud attribute value corresponding to the any one of the sampled point cloud data and the fused point cloud attribute value.

[0150] In step S510, sample training data is obtained according to the sampled point cloud data and the label confidence corresponding to the sampled point cloud data.

[0151] In step S512, the sampled point cloud data is input into a target classification model to be trained, and the target classification model to be trained is used to classify the target of the sampled point cloud data to obtain a category prediction result corresponding to the sampled point cloud data.

[0152] In step S514, a loss value is obtained according to the category prediction result and the label confidence.

[0153] In step S516, the model parameters of the target classification model to be trained are adjusted according to the loss value, and iterative training is continued until the training is stopped when a training end condition is met, and a trained target classification model is obtained.

[0154] It should be noted that the specific definition of the above steps can refer to the specific definition of the sample data enhancement method in the foregoing.

[0155] In one embodiment, as shown in Figure 6 , a target classification method is provided, and the embodiment takes the method applied to an electronic device as an example for description. The electronic device can be the user terminal 110 or the server 170 in Figure 1 . The method includes the following steps:

[0156] In step S610, a plurality of point cloud data collected in a target space is obtained.

[0157] In step S620, the plurality of point cloud data is input into a trained target classification model, and the target classification model is used to classify the plurality of point cloud data to obtain the category of each target in the target space.

[0158] The trained target classification model is trained according to sample training data.

[0159] The sample training data is obtained according to the sample data enhancement method.

[0160] It can be understood that the target space can be a space that needs to be classified, which can be a relatively large open space, such as an office, a hall, a cinema, and the like, or a relatively small closed space, such as a living room, a bedroom, and the like, and the size and shape of the target space are not limited herein.

[0161] The multi-frame point cloud data can be point cloud data collected by the target detection device in the target space.

[0162] In actual application, the trained target classification model can be deployed in an electronic device, and the electronic device can use the trained target classification model for classification detection. Specifically, the electronic device can obtain multi-frame point cloud data collected by the target detection device in the target space, and input the multi-frame point cloud data into the trained target classification model, so as to obtain the category of each target in the target space through target classification processing of the multi-frame point cloud data by the target classification model.

[0163] In some embodiments, the electronic device can also be a radar device, and the target detection device can be the radar device itself.

[0164] In some embodiments, in order to further improve the classification detection accuracy, the multi-frame fused point cloud data belonging to the same target can be input into the trained target classification model, and the category of each target in the target space can be obtained.

[0165] Specifically, after the electronic device obtains the multi-frame point cloud data collected in the target space, the electronic device can cluster the multi-frame point cloud data, determine the multi-frame point cloud data belonging to the same target, and fuse the multi-frame point cloud data belonging to the same target, to obtain the multi-frame fused point cloud data belonging to the same target.

[0166] For example, for the point cloud data belonging to the same target, the electronic device can fuse the multi-frame point cloud data belonging to the same target that meets the preset frame number (for example, 10 frames, and the specific frame number is not limited herein) according to the fusion method in the above embodiments after the point cloud data accumulates to the preset frame number, to obtain the fused point cloud data belonging to the same target. Then, the target detection device can input the fused point cloud data corresponding to different targets into the trained target classification model in sequence, and the trained target classification model can output the categories corresponding to different targets.

[0167] The technical scheme of the embodiment comprises the following steps: acquiring a plurality of frames of point cloud data collected in a target space; inputting the plurality of frames of point cloud data into a trained target classification model, performing target classification processing on the plurality of frames of point cloud data by the target classification model, and obtaining the categories of each target in the target space; the trained target classification model is obtained according to sample training data; the sample training data is obtained according to the sample data enhancement method.

[0168] In this way, the sample training data is obtained according to the sample data enhancement method. The sample data enhancement method can generate at least one sampled point cloud data for the same target by fusing sample point cloud data belonging to the same target and randomly sampling the fused point cloud data, thereby increasing the sample data amount of different categories of targets, improving the balance of the sample quantity, generating more training samples, avoiding overfitting of the target classification model, and effectively improving the classification accuracy and generalization of the target classification model. Moreover, by using the attribute information related to the reliability of the point cloud data, the label confidence of each sampled point cloud data can be more accurately determined. In this way, the label confidence can be added in the model training process, which can reduce the overfitting problem in the training model and further improve the classification performance of the model. Thus, after obtaining the trained target classification model, the point cloud data of the target to be detected is input into the trained target classification model, and the category information corresponding to the target to be detected can be more accurately obtained.

[0169] In another embodiment, the category of the target comprises a human body; the method further comprises: if the category of the target is a human body and the position of the target satisfies a trigger condition in an automatic control scheme, instructing a corresponding target device to perform a target action according to the automatic control scheme.

[0170] The automatic control scheme refers to a scheme that can realize automatic control of each intelligent device bound in the target space. The automatic control scheme is configured with a trigger condition and an execution action, and the device implementing the automatic control scheme comprises a trigger device and a controlled device (or an execution device), which are communicatively connected through a gateway. When the trigger device satisfies the trigger condition, the gateway controls the controlled device to perform a corresponding execution action. The trigger device can be various sensors such as radar sensors, pressure sensors, etc. The controlled device can be various switches, televisions, sockets, lamps, etc. intelligent devices. Assuming that the user sets an automatic control scheme as controlling the light when the door is opened, the trigger condition in this automation is the door opening, and the execution action is the intelligent switch controlling the light to turn on. Based on this application scenario, the door magnetic sensor, the intelligent door lock, etc. can be set as the trigger device, and the intelligent switch connected with the lamp can be set as the controlled device.

[0171] In this way, the sample training data is obtained according to the sample data enhancement method. The sample data enhancement method can generate at least one sampled point cloud data for the same target by fusing sample point cloud data belonging to the same target and randomly sampling the fused point cloud data, thereby increasing the sample data amount of different categories of targets, improving the balance of the sample quantity, generating more training samples, avoiding overfitting of the target classification model, and effectively improving the classification accuracy and generalization of the target classification model. Moreover, by using the attribute information related to the reliability of the point cloud data, the label confidence of each sampled point cloud data can be more accurately determined. In this way, the label confidence can be added in the model training process, which can reduce the overfitting problem in the training model and further improve the classification performance of the model. Thus, after obtaining the trained target classification model, the point cloud data of the target to be detected is input into the trained target classification model, and the category information corresponding to the target to be detected can be more accurately obtained.

[0172] The correspondence between the category of the target and the automatic control scheme can be established in advance, for example, an adult corresponds to an automatic control intelligent air conditioner, an intelligent lamp, etc., a child corresponds to an automatic control intelligent sound, an intelligent lamp, etc., and a pet corresponds to an automatic control intelligent feeder, etc.

[0173] In a specific implementation, after determining the category of each target in the target space, the electronic device can determine the automatic control scheme adapted to the target according to the correspondence and the category of the target. Specifically, the electronic device can determine the automatic control scheme corresponding to the same category as the recognized category in the correspondence as the automatic control scheme adapted to the target. If the state information of the target meets the trigger condition in the adapted automatic control scheme, the electronic device can instruct the corresponding target device to perform the target action according to the automatic control scheme through the gateway.

[0174] The target device is the controlled device in the automatic control scheme adapted to the target, and the target action is the execution action of the controlled device in the automatic control scheme adapted to the target.

[0175] The state information of the target can include the position information of the target in the target space.

[0176] For example, the automatic control scheme can be to automatically start the automatic feeding function of the intelligent feeder when the pet is recognized and the distance between the position of the pet and the position of the intelligent feeder meets the first distance condition. The distance between the position of the pet and the position of the intelligent feeder meeting the first distance is the trigger condition, the intelligent feeder is the target device, and starting the automatic feeding function of the intelligent feeder is the target action.

[0177] In some embodiments, the category of the target includes a human body. If the category of the target is recognized as a human body and the position of the target meets the trigger condition in the automatic control scheme adapted to the human body, the electronic device can instruct the corresponding target device to perform the target action according to the automatic control scheme through the gateway. For example, the automatic control scheme can be to automatically turn on the light when a person is recognized and the distance between the person and the light meets the second distance condition. The distance between the person and the light meeting the second distance condition is the trigger condition, the intelligent switch connected to the light is the target device, and the target action is that the intelligent switch controls the light to turn on.

[0178] In other embodiments, the automatic control scheme can be further set according to the current brightness of the target space. For example, the automatic control scheme can be to automatically turn on the light closest to the person when the current brightness does not meet the preset brightness condition.

[0179] Therefore, by acquiring the category of the target, and controlling the target device to perform the target action according to whether the position of the target meets the trigger condition in the corresponding automatic control scheme, the automatic control of the corresponding device is realized according to different categories, thereby avoiding the problem of false triggering of the target device, and improving the accuracy and flexibility of device control.

[0180] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0181] Based on the same inventive concept, the embodiments of the present application also provide a sample data enhancement device for implementing the above-mentioned sample data enhancement method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more sample data enhancement device embodiments provided below can refer to the limitations of the sample data enhancement method in the above text, which will not be repeated here.

[0182] In one exemplary embodiment, as shown in Figure 7 A sample data enhancement device is provided, including a fusion module 710, a sampling module 720, a confidence determination module 730, and a sample construction module 740, wherein:

[0183] The fusion module 710 is configured to fuse sample point cloud data belonging to the same target in multiple frames of sample point cloud data to obtain fused point cloud data.

[0184] The sampling module 720 is configured to perform sampling processing on the fused point cloud data to generate at least one sampled point cloud data.

[0185] The confidence determination module 730 is configured to determine the label confidence corresponding to each of the sampled point cloud data according to point cloud attribute information of each of the sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of the point cloud data.

[0186] The sample construction module 740 is configured to obtain sample training data according to the sampled point cloud data and the label confidence corresponding to the sampled point cloud data, so as to train a target classification model; and the target classification model is configured to perform target classification on the collected point cloud data.

[0187] In one of the embodiments, the point cloud attribute information includes a point cloud attribute value; the confidence determination module 730 is specifically configured to obtain the point cloud attribute value corresponding to the fused point cloud data, to obtain a fused point cloud attribute value; and for any one of the sampled point cloud data, the label confidence corresponding to the any one of the sampled point cloud data is obtained according to the ratio between the point cloud attribute value corresponding to the any one of the sampled point cloud data and the fused point cloud attribute value.

[0188] In one of the embodiments, in the case where the point cloud attribute information includes point cloud attribute values corresponding to at least two point cloud attribute types, the confidence determination module 730 is specifically configured to obtain the fused point cloud attribute value corresponding to each of the point cloud attribute types; and the label confidence corresponding to the any one of the sampled point cloud data is obtained according to the ratio between the point cloud attribute value corresponding to each of the point cloud attribute types and the fused point cloud attribute value corresponding to the corresponding point cloud attribute type.

[0189] In one of the embodiments, the sample training data includes the sampled point cloud data and the corresponding label confidence; and the sample data enhancement device further includes a training module configured to input the sampled point cloud data into a target classification model to be trained, perform target classification on the sampled point cloud data by using the target classification model to be trained, obtain a category prediction result corresponding to the sampled point cloud data, obtain a loss value according to the category prediction result and the label confidence corresponding to the sampled point cloud data, adjust the model parameters of the target classification model to be trained according to the loss value and continue iterative training until the training is stopped when a training end condition is met, and obtain a trained target classification model.

[0190] In one of the embodiments, the sampling module 720 is specifically configured to perform random sampling processing on the point cloud in the fused point cloud data according to a preset sampling rate, to obtain the at least one sampled point cloud data.

[0191] In one of the embodiments, the sampling module 720 is specifically configured to perform random frame loss processing on the point cloud in the fused point cloud data according to a preset frame number ratio, to obtain frame loss point cloud data; and the frame loss point cloud data is taken as the sampled point cloud data.

[0192] In one of the embodiments, the fusion module 710 is specifically configured to fuse sample point cloud data belonging to the same target according to a preset frame number and a preset sliding step, to obtain a plurality of fused point cloud data belonging to the same target.

[0193] The modules in the sample data enhancement apparatus can be implemented in whole or in part by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in the electronic device in hardware form, or stored in a memory in the electronic device in software form, so as to be invoked and executed by the processor to perform operations corresponding to the modules.

[0194] Based on the same inventive concept, the embodiments of the present application also provide a target classification apparatus for implementing the target classification method described above. The apparatus provides a solution to the implementation scheme as described in the above method, and therefore the specific limitations in one or more target classification apparatus embodiments provided below can be referred to the limitations of the target classification method described above, which will not be repeated here.

[0195] In one exemplary embodiment, as shown in Figure 8 a target classification apparatus is provided, comprising: an acquisition module 810 and a classification module 820, wherein:

[0196] The acquisition module 810 is configured to acquire a plurality of frames of point cloud data collected in a target space.

[0197] The classification module 820 is configured to input the plurality of frames of point cloud data into a trained target classification model, perform target classification processing on the plurality of frames of point cloud data by the target classification model, and obtain the category of each target in the target space; the trained target classification model is obtained by training sample training data; the sample training data is obtained according to the sample data enhancement method described above.

[0198] In one of the embodiments, the category of the target includes a human body; the target classification apparatus further comprises a control module configured to, if the category of the target is a human body and the position of the target satisfies a trigger condition in an automatic control scheme, instruct a corresponding target device to perform a target action according to the automatic control scheme.

[0199] The modules in the target classification apparatus can be implemented in whole or in part by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in the electronic device in hardware form, or stored in a memory in the electronic device in software form, so as to be invoked and executed by the processor to perform operations corresponding to the modules.

[0200] In an example embodiment, an electronic device, which can be a server, has an internal structure diagram as shown in Figure 9 The electronic device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is configured to exchange information between the processor and external devices. The communication interface of the electronic device is configured to communicate with external terminals through network connections. The computer program is executed by the processor to implement a sample data enhancement method and / or a target classification method.

[0201] Those skilled in the art can understand that Figure 9 The structure shown in the above embodiment is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. Specifically, the electronic device can include more or fewer components than those shown in the diagram, or combine certain components, or have a different arrangement of components.

[0202] In an example embodiment, an electronic device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0203] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0204] In an example embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0205] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant regulations.

[0206] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0207] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0208] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method of sample data augmentation, the method comprising: The method comprises: Fusing sample point cloud data belonging to the same target in multiple frames of sample point cloud data to obtain fused point cloud data; Sampling processing the fused point cloud data to generate at least one sampled point cloud data; Determining a label confidence corresponding to each of the sampled point cloud data according to point cloud attribute information of each of the sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of the point cloud data; Obtaining sample training data according to each of the sampled point cloud data and the label confidence corresponding to each of the sampled point cloud data to train a target classification model; the target classification model is used for target classification of the collected point cloud data.

2. The method of claim 1, wherein, The point cloud attribute information comprises point cloud attribute values; and the determining of the label confidence corresponding to each of the sampled point cloud data according to the point cloud attribute information of each of the sampled point cloud data comprises: Obtaining point cloud attribute values corresponding to the fused point cloud data to obtain fused point cloud attribute values; For any one of the sampled point cloud data, obtaining the label confidence corresponding to the any one of the sampled point cloud data according to a ratio between the point cloud attribute values corresponding to the any one of the sampled point cloud data and the fused point cloud attribute values.

3. The method of claim 2, wherein, In the case that the point cloud attribute information comprises point cloud attribute values corresponding to at least two point cloud attribute types, the obtaining of the point cloud attribute values of the fused point cloud data to obtain the fused point cloud attribute values comprises: Obtaining fused point cloud attribute values corresponding to each of the point cloud attribute types of the fused point cloud data; The obtaining of the label confidence corresponding to the any one of the sampled point cloud data according to the ratio between the point cloud attribute values corresponding to the any one of the sampled point cloud data and the fused point cloud attribute values comprises: Obtaining the label confidence corresponding to the any one of the sampled point cloud data according to a ratio between the point cloud attribute values corresponding to each of the point cloud attribute types of the any one of the sampled point cloud data and the fused point cloud attribute values corresponding to the corresponding point cloud attribute type.

4. The method of claim 1, wherein, The sample training data comprises the sampled point cloud data and the corresponding label confidence; and the method further comprises: Inputting the sampled point cloud data into a target classification model to be trained to perform target classification on the sampled point cloud data by the target classification model to be trained to obtain a category prediction result corresponding to the sampled point cloud data; Obtaining a loss value according to the category prediction result and the label confidence corresponding to the sampled point cloud data; Adjusting model parameters of the target classification model to be trained according to the loss value and continuing iteration training until the training is stopped when a training end condition is met to obtain a trained target classification model.

5. The method of claim 1, wherein, The sampling processing of the fused point cloud data to generate at least one sampled point cloud data comprises: Randomly sampling processing point clouds in the fused point cloud data according to a preset sampling rate to obtain at least one of the sampled point cloud data.

6. The method of claim 1, wherein, The sampling processing on the fused point cloud data generates at least one sampled point cloud data, including: According to a preset frame number ratio, the point cloud in the fused point cloud data is randomly frame-dropped to obtain frame-dropped point cloud data; The frame-dropped point cloud data is taken as the sampled point cloud data.

7. The method of claim 1, wherein, The method comprises: According to a preset frame number ratio, the point cloud in the fused point cloud data is randomly frame-dropped to obtain frame-dropped point cloud data; 8. A method of target classification, characterized by, The method comprises: According to a preset frame number ratio, the point cloud in the fused point cloud data is randomly frame-dropped to obtain frame-dropped point cloud data; The target classification model is trained according to sample training data obtained by the sample data enhancement method in any one of claims 1 to 7.

9. The method of claim 8, wherein, The target category includes a human body; and the method further comprises: If the target category is a human body and the position of the target meets a trigger condition in an automatic control scheme, a corresponding target device is instructed to perform a target action according to the automatic control scheme.

10. A sample data augmentation apparatus, characterized by, The device comprises: A fusion module is configured to fuse sample point cloud data belonging to the same target in multiple frames of sample point cloud data to obtain fused point cloud data; A sampling module is configured to sample the fused point cloud data to generate at least one sampled point cloud data; A confidence determination module is configured to determine a label confidence corresponding to each sampled point cloud data according to point cloud attribute information of each sampled point cloud data; the point cloud attribute information is attribute information related to the reliability of point cloud data; A sample construction module is configured to obtain sample training data according to each sampled point cloud data and the label confidence corresponding to each sampled point cloud data, to train a target classification model; the target classification model is used for target classification on collected point cloud data.

11. A target classification apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire multiple frames of point cloud data collected in a target space; A classification module is configured to input the multiple frames of point cloud data into a trained target classification model, to perform target classification processing on the multiple frames of point cloud data by using the target classification model, and to obtain the category of each target in the target space; the trained target classification model is trained according to sample training data; the sample training data is obtained according to the sample data enhancement method in any one of claims 1 to 7.

12. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method in any one of claims 1 to 9 when executing the computer program.

13. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, implements the steps of the method in any one of claims 1 to 9.

14. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method in any one of claims 1 to 9.

Images