Point cloud enhancement method and device

Abstract

The invention provides a point cloud enhancement method and device, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision and deep learning, and can be applied to automatic driving, unmanned aerial vehicle detection and obstacle avoidance, intelligent security and protection, industrial automation and other scenes. A specific embodiment of the method comprises the following steps: performing point cloud diffusion on an original point cloud point by using a surrounding point cloud point of the original point cloud point in original point cloud data to obtain a diffusion point cloud point; and generating enhanced point cloud data based on the original point cloud points and the diffusion point cloud points.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision and deep learning technology, and can be applied to scenarios such as autonomous driving, drone detection and obstacle avoidance, intelligent security, and industrial automation. Background Art

[0002] Radar is the transliteration of the English word Radar, which comes from the abbreviation of radio detection and ranging, meaning "radio detection and ranging", that is, using radio methods to discover targets and determine their spatial positions. Therefore, radar is also called "radio positioning". Radar is an electronic device that uses electromagnetic waves to detect targets. Radar emits electromagnetic waves to illuminate the target and receives its echo, thereby obtaining information such as the distance from the target to the electromagnetic wave emission point, the rate of change of distance (radial velocity), azimuth, altitude, etc.

[0003] Due to its inherent physical properties, such as low resolution and sparsity, the point cloud data generated by millimeter-wave radar often contains less information, which limits its application in fields such as autonomous driving and robotics. Traditional solutions usually rely on expensive lidar to obtain high-resolution point cloud data. However, the high cost of lidar limits its popularity. Summary of the invention

[0004] The embodiments of the present disclosure provide a point cloud enhancement method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes a point cloud enhancement method, including: utilizing the surrounding point cloud points of the original point cloud points in the original point cloud data to perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points; and generating enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0006] In the second aspect, an embodiment of the present disclosure proposes a point cloud enhancement device, including: a diffusion module, configured to use the surrounding point cloud points of the original point cloud points in the original point cloud data to perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points; a generation module, configured to generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0007] In a third aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute the method described in the first aspect.

[0009] In a fifth aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0010] The key or important features of the embodiments of the present disclosure are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, purposes and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:

[0012] Figure 1 is a flow chart of an embodiment of a point cloud enhancement method according to the present disclosure;

[0013] Figure 2 is a flow chart of another embodiment of the point cloud enhancement method according to the present disclosure;

[0014] Figure 3 is a flow chart of another embodiment of the point cloud enhancement method according to the present disclosure;

[0015] Figure 4 is a scene graph that can implement the point cloud enhancement method disclosed herein;

[0016] Figure 5 is a structural schematic diagram of an embodiment of a point cloud enhancement device according to the present disclosure;

[0017] Figure 6 It is a block diagram of an electronic device used to implement the point cloud enhancement method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] Figure 1 A process 100 of an embodiment of a point cloud enhancement method according to the present disclosure is shown. The point cloud enhancement method comprises the following steps:

[0021] Step 101, using the surrounding point cloud points of the original point cloud points in the original point cloud data, perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points.

[0022] In this embodiment, the execution subject of the point cloud enhancement method can use the surrounding point cloud points of the original point cloud points in the original point cloud data to perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points.

[0023] The execution subject of the point cloud enhancement method is usually a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0024] The original point cloud data may be point cloud data collected by millimeter wave radar. Due to the limitations of the physical characteristics and environmental factors of millimeter wave radar, the original point cloud data often has low resolution and sparsity, and it is difficult to accurately represent the detailed features of the object.

[0025] The original point cloud data can be composed of a series of discrete original point cloud points (x i ,y i ,z i ). Where i represents the index of the original point cloud point. The original point cloud data is unevenly distributed in the three-dimensional space and may contain noise and missing points.

[0026] Point cloud diffusion of the original point cloud data can make these point cloud data smoother and more continuous in three-dimensional space, thereby improving the resolution and density of the point cloud data. For the original point cloud points in the original point cloud data, the original point cloud points can be diffused using the surrounding point cloud points to generate diffused point cloud points. In this way, the diffused point cloud points are related to both the original point cloud points and the surrounding point cloud points. Among them, the surrounding point cloud points of the original point cloud points can be other original point cloud points around the original point cloud points, for example, other original point cloud points whose distance from the original point cloud points is less than a preset distance threshold.

[0027] In some embodiments, for an original point cloud point in the original point cloud data, the coordinate mean of the surrounding point cloud points of the original point cloud point can be calculated, and a new point cloud point, ie, a diffuse point cloud point, can be generated.

[0028] In some embodiments, a diffusion kernel function K(x, y, z) is defined. The diffusion kernel function K(x, y, z) can describe the degree of influence of each point cloud point on its surrounding point cloud points. For the original point cloud points in the original point cloud data, the diffusion kernel function can be used to calculate the weighted sum of the surrounding point cloud points of the original point cloud points to obtain the diffused point cloud points. The weighted sum formula can be as follows:

[0029]

[0030] Among them, (x i ,y i ,z i ) represents the original point cloud point, i represents the original point cloud point (x i ,y i ,z i ), (x j ,y j ,z j ) is the original point cloud point (x i ,y i ,z i ), j represents the surrounding point cloud point (x j ,y j ,z j ), (x' i ,y' i ,z' i ) represents diffuse point cloud points.

[0031] In three-dimensional space, the diffusion kernel function K(x, y, z) may include but is not limited to a Gaussian function, a uniform function or other suitable functions. Since the Gaussian function has smoothness and locality, it can better simulate the diffusion process of point cloud data, so the Gaussian function can be selected as the diffusion kernel function. At this time, the diffusion kernel function can be as follows:

[0032]

[0033] Among them, K(x,y,z) represents the diffusion kernel function, and σ represents the standard deviation of the diffusion kernel, which is used to determine the range and intensity of diffusion.

[0034] Step 102: generating enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0035] In this embodiment, the above-mentioned execution entity can generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0036] In some embodiments, the original point cloud points and the diffused point cloud points are fused to generate fused point cloud data. Based on the fused point cloud data, enhanced point cloud data can be generated. For example, the fused point cloud data obtained by fusing the original point cloud points and the diffused point cloud points is directly used as the enhanced point cloud data. At this time, the number of point cloud points in the enhanced point cloud data is greatly increased compared to the number of point cloud points in the original point cloud data, thereby improving the resolution and density of the point cloud data. For another example, the fused point cloud data is denoised to generate enhanced point cloud data. At this time, the diffusion process can smoothly propagate information and reduce the impact of noise, thereby improving the accuracy and reliability of the point cloud data.

[0037] The disclosed embodiment provides a point cloud enhancement method, which utilizes the point cloud data generated by relatively inexpensive millimeter wave radar, and generates high-resolution point cloud data equivalent to or better than that of laser radar through point cloud diffusion processing, so that it can more accurately represent the shape and detailed features of objects. This process not only improves the data quality of radar point clouds, but also makes it applicable to more downstream tasks, such as object detection, tracking, and map construction.

[0038] The point cloud enhancement method provided by the embodiments of the present disclosure can achieve the following goals:

[0039] 1. Improve the resolution of radar point clouds: Generate denser and more detailed point cloud data than the original radar point cloud, thereby improving the usability of the data.

[0040] 2. Enhance the information content of radar point cloud: Through resolution enhancement, more detailed information can be captured, such as the edge, shape and texture of the object, which is crucial for subsequent perception and decision-making tasks.

[0041] 3. Reduce costs: Compared with relying on expensive lidar, it provides a more economical and practical solution, making the acquisition of high-resolution point cloud data easier and more feasible.

[0042] 4. Improve the performance of downstream tasks: By enhancing the data quality of radar point clouds, the performance of downstream tasks that rely on point cloud data, such as object detection, tracking, and path planning, can be improved.

[0043] Compared with the existing technology, the differences are:

[0044] The prior art usually performs post-processing on the generated point cloud data, such as filtering, interpolation, etc. However, the embodiment of the present disclosure directly applies a 3D diffusion method to the radar echo signal to improve the resolution of the point cloud data from the source.

[0045] The disclosed embodiment adopts a 3D diffusion method, which can more comprehensively capture the three-dimensional features of point cloud data and achieve a better effect of improving resolution.

[0046] Continue to refer Figure 2 , which shows a process 200 of another embodiment of the point cloud enhancement method according to the present disclosure. The point cloud enhancement method comprises the following steps:

[0047] Step 201 , using a diffusion kernel function, calculates a weighted sum of surrounding point cloud points of an original point cloud point in the original point cloud data to obtain a diffuse point cloud point.

[0048] In this embodiment, the execution body of the point cloud enhancement method can use the diffusion kernel function to calculate the weighted sum of the surrounding point cloud points of the original point cloud points in the original point cloud data to obtain the diffused point cloud points.

[0049] The execution subject of the point cloud enhancement method is usually a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0050] The original point cloud data may be point cloud data collected by millimeter wave radar. Due to the limitations of the physical characteristics and environmental factors of millimeter wave radar, the original point cloud data often has low resolution and sparsity, and it is difficult to accurately represent the detailed features of the object.

[0051] The original point cloud data can be composed of a series of discrete original point cloud points (x i ,y i ,z i ). Where i represents the index of the original point cloud point. The original point cloud data is unevenly distributed in the three-dimensional space and may contain noise and missing points.

[0052] Point cloud diffusion of the original point cloud data can make these point cloud data smoother and more continuous in three-dimensional space, thereby improving the resolution and density of the point cloud data. For the original point cloud points in the original point cloud data, the original point cloud points can be diffused using the surrounding point cloud points to generate diffused point cloud points. In this way, the diffused point cloud points are related to both the original point cloud points and the surrounding point cloud points. Among them, the surrounding point cloud points of the original point cloud points can be other original point cloud points around the original point cloud points, for example, other original point cloud points whose distance from the original point cloud points is less than a preset distance threshold.

[0053] Define a diffusion kernel function K(x,y,z). The diffusion kernel function K(x,y,z) can describe the degree of influence of each point cloud point on its surrounding point cloud points. For the original point cloud points in the original point cloud data, the diffusion kernel function can be used to calculate the weighted sum of the surrounding point cloud points of the original point cloud points to obtain the diffuse point cloud points. The weighted sum formula can be as follows:

[0054]

[0055] Among them, (x i ,y i ,z i ) represents the original point cloud point, i represents the original point cloud point (x i ,y i ,z i ), (x j ,y j ,z j ) is the original point cloud point (x i ,y i ,z i ), j represents the surrounding point cloud point (x j ,y j ,z j ), (x' i ,y' i ,z' i ) represents diffuse point cloud points.

[0056] In three-dimensional space, the diffusion kernel function K(x, y, z) may include but is not limited to a Gaussian function, a uniform function or other suitable functions. Since the Gaussian function has smoothness and locality, it can better simulate the diffusion process of point cloud data, so the Gaussian function can be selected as the diffusion kernel function. At this time, the diffusion kernel function can be as follows:

[0057]

[0058] Among them, K(x,y,z) represents the diffusion kernel function, and σ represents the standard deviation of the diffusion kernel, which is used to determine the range and intensity of diffusion.

[0059] Step 202: Fusing the original point cloud points with the diffused point cloud points to generate fused point cloud data.

[0060] In this embodiment, the execution entity may fuse the original point cloud points with the diffused point cloud points to generate fused point cloud data.

[0061] Step 203: determine whether the number of point cloud diffusion times reaches a preset number threshold.

[0062] In this embodiment, the execution subject can determine whether the number of point cloud diffusion times reaches a preset number threshold. If the number of point cloud diffusion times does not reach the preset number threshold, step 204 is executed; if the number of point cloud diffusion times reaches the preset number threshold, step 205 is executed.

[0063] Step 204: Use the fused point cloud data as new original point cloud data.

[0064] In this embodiment, if the number of point cloud diffusion times does not reach the preset number threshold, the execution entity may use the fused point cloud data as new original point cloud data, return to step 201, and perform the next round of point cloud diffusion.

[0065] Among them, the iteration number threshold and the parameters of the diffusion kernel may need to be adjusted according to the specific application scenario and data characteristics.

[0066] For example, point cloud data is often used in target recognition application scenarios. Target recognition application scenarios require target detection on point cloud data. Therefore, for each type of target, millimeter-wave radar can be used to collect point cloud data and perform multiple rounds of point cloud diffusion. For each round of point cloud diffusion to obtain point cloud data, target detection can be performed to obtain target detection results. Based on the target detection results and the true target category, the accuracy of target detection can be calculated. If the accuracy of target detection reaches the preset accuracy threshold, the current number of iterations can be used as the iteration number threshold for the point cloud data of this type of target.

[0067] For example, the parameters of the diffusion kernel may include the diffusion steps and the noise intensity. According to the diffusion steps, the noise intensity function may be set so that the noise intensity gradually decreases as the diffusion steps increase.

[0068] Among them, the formula of the noise intensity function β(t) can be as follows:

[0069] β(t)=β0(1-t / T) 2 .

[0070] Among them, β0 is the initial noise, t is the number of diffusion steps, and T is the total number of diffusion steps.

[0071] In each diffusion process, monitor the variance change of the model output. If the variance is too large, increase the noise intensity of the current step appropriately to enhance stability; otherwise, reduce the noise intensity.

[0072] In addition, by using prior knowledge of environment geometry, material, etc., it is also possible to estimate the reasonable noise intensity of different regions. For example, for a flat area, a smaller noise intensity can be used; for an edge area, a larger noise intensity can be used. Step 205: Generate enhanced point cloud data based on the fused point cloud data.

[0073] In this embodiment, if the number of point cloud diffusion times reaches a preset number threshold, the execution entity may generate enhanced point cloud data based on the fused point cloud data.

[0074] For example, the fused point cloud data obtained by fusing the original point cloud points and the diffused point cloud points is directly used as the enhanced point cloud data. At this time, the number of point cloud points in the enhanced point cloud data is greatly increased compared to the number of point cloud points in the original point cloud data, thereby improving the resolution and density of the point cloud data.

[0075] For another example, the fused point cloud data is denoised to generate enhanced point cloud data. At this time, the diffusion process can smoothly propagate information and reduce the impact of noise, thereby improving the accuracy and reliability of the point cloud data.

[0076] from Figure 2 It can be seen that Figure 1 Compared with the corresponding embodiment, the process 200 of the point cloud enhancement method in this embodiment highlights the step of iterative diffusion. Therefore, the solution described in this embodiment performs multiple iterative diffusions on the point cloud data. After each iteration, the resolution of the point cloud data will be improved to a certain extent, thereby improving the diffusion effect.

[0077] Further references Figure 3 , which shows a process 300 of another embodiment of the point cloud enhancement method according to the present disclosure. The point cloud enhancement method comprises the following steps:

[0078] Step 301, using a diffusion kernel function, calculates the weighted sum of the surrounding point cloud points of the original point cloud points in the original point cloud data to obtain diffuse point cloud points.

[0079] In this embodiment, the execution body of the point cloud enhancement method can use the diffusion kernel function to calculate the weighted sum of the surrounding point cloud points of the original point cloud points in the original point cloud data to obtain the diffused point cloud points.

[0080] The execution subject of the point cloud enhancement method is usually a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0081] The original point cloud data may be point cloud data collected by millimeter wave radar. Due to the limitations of the physical characteristics and environmental factors of millimeter wave radar, the original point cloud data often has low resolution and sparsity, and it is difficult to accurately represent the detailed features of the object.

[0082] The original point cloud data can be composed of a series of discrete original point cloud points (xi ,y i ,z i ). Where i represents the index of the original point cloud point. The original point cloud data is unevenly distributed in the three-dimensional space and may contain noise and missing points.

[0083] Point cloud diffusion of the original point cloud data can make these point cloud data smoother and more continuous in three-dimensional space, thereby improving the resolution and density of the point cloud data. For the original point cloud points in the original point cloud data, the original point cloud points can be diffused using the surrounding point cloud points to generate diffused point cloud points. In this way, the diffused point cloud points are related to both the original point cloud points and the surrounding point cloud points. Among them, the surrounding point cloud points of the original point cloud points can be other original point cloud points around the original point cloud points, for example, other original point cloud points whose distance from the original point cloud points is less than a preset distance threshold.

[0084] Define a diffusion kernel function K(x,y,z). The diffusion kernel function K(x,y,z) can describe the degree of influence of each point cloud point on its surrounding point cloud points. For the original point cloud points in the original point cloud data, the diffusion kernel function can be used to calculate the weighted sum of the surrounding point cloud points of the original point cloud points to obtain the diffuse point cloud points. The weighted sum formula can be as follows:

[0085]

[0086] Among them, (x i ,y i ,z i ) represents the original point cloud point, i represents the original point cloud point (x i ,y i ,z i ), (x j ,y j ,z j ) is the original point cloud point (x i ,y i ,z i ), j represents the surrounding point cloud point (x j ,y j ,z j ), (x' i ,y' i ,z' i ) represents diffuse point cloud points.

[0087] In three-dimensional space, the diffusion kernel function K(x, y, z) may include but is not limited to a Gaussian function, a uniform function or other suitable functions. Since the Gaussian function has smoothness and locality, it can better simulate the diffusion process of point cloud data, so the Gaussian function can be selected as the diffusion kernel function. At this time, the diffusion kernel function can be as follows:

[0088]

[0089] Among them, K(x,y,z) represents the diffusion kernel function, and σ represents the standard deviation of the diffusion kernel, which is used to determine the range and intensity of diffusion.

[0090] Step 302: Fusing the original point cloud points and the diffused point cloud points to generate fused point cloud data.

[0091] In this embodiment, the execution entity may fuse the original point cloud points with the diffused point cloud points to generate fused point cloud data.

[0092] Step 303: determine whether the point cloud diffusion times reaches a preset times threshold.

[0093] In this embodiment, the execution subject can determine whether the number of point cloud diffusion times reaches a preset number threshold. If the number of point cloud diffusion times does not reach the preset number threshold, step 304 is executed; if the number of point cloud diffusion times reaches the preset number threshold, step 305 is executed.

[0094] Step 304: Use the fused point cloud data as new original point cloud data.

[0095] In this embodiment, if the number of point cloud diffusion times does not reach the preset number threshold, the above-mentioned execution entity can use the fused point cloud data as new original point cloud data, return to execute step 301, and perform the next round of point cloud diffusion.

[0096] Among them, the iteration number threshold and the parameters of the diffusion kernel may need to be adjusted according to the specific application scenario and data characteristics.

[0097] For example, point cloud data is often used in target recognition application scenarios. Target recognition application scenarios require target detection on point cloud data. Therefore, for each type of target, millimeter-wave radar can be used to collect point cloud data and perform multiple rounds of point cloud diffusion. For each round of point cloud diffusion to obtain point cloud data, target detection can be performed to obtain target detection results. Based on the target detection results and the true target category, the accuracy of target detection can be calculated. If the accuracy of target detection reaches the preset accuracy threshold, the current number of iterations can be used as the iteration number threshold for the point cloud data of this type of target.

[0098] For example, the parameters of the diffusion kernel may include the diffusion steps and the noise intensity. According to the diffusion steps, the noise intensity function may be set so that the noise intensity gradually decreases as the diffusion steps increase.

[0099] Among them, the formula of the noise intensity function β(t) can be as follows:

[0100] β(t)=β0(1-t / T) 2.

[0101] Among them, β0 is the initial noise, t is the number of diffusion steps, and T is the total number of diffusion steps.

[0102] In each diffusion process, monitor the variance change of the model output. If the variance is too large, increase the noise intensity of the current step appropriately to enhance stability; otherwise, reduce the noise intensity.

[0103] In addition, by using prior knowledge of environmental geometry, material, etc., it is also possible to estimate the reasonable noise intensity of different areas. For example, for flat areas, a smaller noise intensity can be used; for edge areas, a larger noise intensity can be used. Step 305, input the fused point cloud data into the U-Net network to obtain enhanced point cloud data.

[0104] In this embodiment, if the number of point cloud diffusion times reaches a preset threshold, the above-mentioned execution entity can input the fused point cloud data into the U-Net network to obtain enhanced point cloud data.

[0105] Among them, the U-Net network can be used for the reverse diffusion process. The U-Net network has a U-shaped structure and consists of a symmetrical encoder and decoder. The encoder can be used to extract high-level feature representations of the fused point cloud data. The decoder can be used to map these feature representations back to the size of the fused point cloud data. The fused point cloud data obtained by the 3D diffusion method is input into the U-Net network for denoising. The U-Net network further removes noise from the point cloud data through its powerful feature extraction and mapping capabilities, improving the clarity and accuracy of the point cloud data.

[0106] It should be noted that the point cloud enhancement method provided in the embodiment of the present disclosure can input only the fused point cloud data obtained by the last point cloud diffusion into the U-Net network for denoising, or can input the fused point cloud data obtained by each point cloud diffusion into the U-Net network for denoising, which is not limited here.

[0107] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiment, the process 300 of the point cloud enhancement method in this embodiment highlights the denoising step. Therefore, the solution described in this embodiment further removes noise in the point cloud data and improves the clarity and accuracy of the point cloud data through the powerful feature extraction and mapping capabilities of the U-Net network.

[0108] Figure 4 A scene diagram showing a point cloud enhancement method of the present disclosure can be implemented. Figure 4As shown in the figure, at the beginning, the millimeter wave radar is used to collect signals to obtain the original point cloud data. Then, for the original point cloud data, the diffusion kernel function is used to perform 3D diffusion, and then the U-Net network is used for inverse diffusion. After multiple iterations of diffusion, the enhanced point cloud data is obtained. Finally, the enhanced point cloud data is post-processed and can be used in application scenarios such as target recognition.

[0109] When used in target recognition and other application scenarios, it can enhance the point cloud resolution of millimeter wave radar, enabling the system to more accurately identify targets and improve the accuracy and reliability of target detection. In complex environments and interference conditions, it can optimize the processing of radar signals and improve the robustness and stability of the system.

[0110] Common target recognition application scenarios may include:

[0111] 1. Autonomous driving: Autonomous driving cars need to perceive the surrounding environment in real time, including roads, vehicles, pedestrians, etc. As one of the autonomous driving sensors, millimeter wave radar can provide information such as distance and speed. Through point cloud diffusion, the point cloud resolution of millimeter wave radar can be enhanced, the accuracy of target detection can be improved, and thus the safety and reliability of the autonomous driving system can be enhanced.

[0112] 2. Drone detection and obstacle avoidance: Drones need to detect and avoid obstacles during flight. Millimeter-wave radar can help drones perceive the environment ahead, and point cloud diffusion can further enhance the radar’s perception capabilities, enabling drones to more accurately identify obstacles and make obstacle avoidance decisions.

[0113] 3. Smart security: In the field of smart security, millimeter wave radar can be used to monitor and detect intruders. Point cloud diffusion can enhance the point cloud resolution of radar, improve the detection accuracy of intruders, and provide more accurate target information for the security system.

[0114] 4. Industrial Automation: In the field of industrial automation, millimeter wave radar can be used in scenarios such as material handling and robot navigation. Point cloud diffusion can enhance the radar’s perception capability, enabling robots to more accurately identify objects and paths, thereby improving production efficiency.

[0115] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a point cloud enhancement device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0116] like Figure 5As shown, the point cloud enhancement device 500 of this embodiment may include: a diffusion module 501 and a generation module 502. The diffusion module 501 is configured to perform point cloud diffusion on the original point cloud points using the surrounding point cloud points of the original point cloud points in the original point cloud data to obtain diffused point cloud points; the generation module 502 is configured to generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0117] In this embodiment, in the point cloud enhancement device 500, the specific processing of the diffusion module 501 and the generation module 502 and the technical effects thereof can be referred to in Figure 1 The relevant descriptions of steps 101 - 102 in the corresponding embodiment are not repeated here.

[0118] In some optional implementations of this embodiment, the diffusion module 501 is further configured to: use a diffusion kernel function to calculate the weighted sum of the surrounding point cloud points of the original point cloud point to obtain a diffuse point cloud point, wherein the diffusion kernel function describes the degree of influence of each point cloud point on the surrounding point cloud points.

[0119] In some optional implementations of this embodiment, the diffusion kernel function includes a Gaussian function or a uniform function.

[0120] In some optional implementations of this embodiment, the generation module 502 includes: a fusion submodule, configured to fuse the original point cloud points and the diffused point cloud points to generate fused point cloud data; and a generation submodule, configured to generate enhanced point cloud data based on the fused point cloud data.

[0121] In some optional implementations of the present embodiment, the generation submodule is further configured to: in response to determining that the number of point cloud diffusion times does not reach a preset number threshold, use the fused point cloud data as new original point cloud data, diffuse the point cloud again, and obtain new fused point cloud data; in response to determining that the number of point cloud diffusion times reaches a preset number threshold, generate enhanced point cloud data based on the fused point cloud data.

[0122] In some optional implementations of this embodiment, the generating submodule is further configured to: perform denoising processing on the fused point cloud data to generate enhanced point cloud data.

[0123] In some optional implementations of this embodiment, the generation submodule is further configured to: input the fused point cloud data into a U-Net network to obtain enhanced point cloud data.

[0124] In some optional implementations of this embodiment, the U-Net network includes an encoder and a decoder; and the generation submodule is further configured to: use the encoder to extract features from the fused point cloud data to obtain a feature representation of the fused point cloud data; use the decoder to map the feature representation back to the size of the fused point cloud data to obtain enhanced point cloud data.

[0125] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0126] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0127] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0128] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0129] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the point cloud enhancement method. For example, in some embodiments, the point cloud enhancement method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the point cloud enhancement method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the point cloud enhancement method in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A point cloud enhancement method, comprising: Using the surrounding point cloud points of the original point cloud points in the original point cloud data, performing point cloud diffusion on the original point cloud points to obtain diffused point cloud points; Enhanced point cloud data are generated based on the original point cloud points and the diffused point cloud points.

2. The method according to claim 1, wherein: The method of utilizing surrounding point cloud points of the original point cloud points in the original point cloud data to perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points includes: The diffusion kernel function is used to calculate the weighted sum of the surrounding point cloud points of the original point cloud point to obtain the diffuse point cloud point, wherein the diffusion kernel function describes the influence degree of each point cloud point on the surrounding point cloud points.

3. The method according to claim 2, wherein: The diffusion kernel function includes a Gaussian function or a uniform function.

4. The method according to claim 1, wherein: The step of generating enhanced point cloud data based on the original point cloud points and the diffused point cloud points comprises: Fusing the original point cloud points with the diffused point cloud points to generate fused point cloud data; Based on the fused point cloud data, enhanced point cloud data is generated.

5. The method according to claim 4, wherein: The step of generating enhanced point cloud data based on the fused point cloud data comprises: In response to determining that the number of point cloud diffusions does not reach a preset number threshold, taking the fused point cloud data as new original point cloud data, and performing point cloud diffusion again to obtain new fused point cloud data; In response to determining that the number of point cloud diffusion times reaches the preset number threshold, enhanced point cloud data is generated based on the fused point cloud data.

6. The method according to claim 4 or 5, wherein: The step of generating enhanced point cloud data based on the fused point cloud data comprises: The fused point cloud data is subjected to denoising processing to generate the enhanced point cloud data.

7. The method according to claim 6, wherein: The denoising process is performed on the fused point cloud data to generate the enhanced point cloud data, including: The fused point cloud data is input into the U-Net network to obtain the enhanced point cloud data.

8. The method according to claim 7, wherein: The U-Net network includes an encoder and a decoder; as well as The step of inputting the fused point cloud data into a U-Net network to obtain the enhanced point cloud data includes: Using the encoder to perform feature extraction on the fused point cloud data to obtain a feature representation of the fused point cloud data; The decoder is used to map the feature representation back to the size of the fused point cloud data to obtain the enhanced point cloud data.

9. A point cloud enhancement device, comprising: A diffusion module is configured to perform point cloud diffusion on the original point cloud points using surrounding point cloud points of the original point cloud points in the original point cloud data to obtain diffused point cloud points; The generating module is configured to generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

10. The device according to claim 9, wherein: The diffusion module is further configured to: The diffusion kernel function is used to calculate the weighted sum of the surrounding point cloud points of the original point cloud point to obtain the diffuse point cloud point, wherein the diffusion kernel function describes the influence degree of each point cloud point on the surrounding point cloud points.

11. The device according to claim 10, wherein: The diffusion kernel function includes a Gaussian function or a uniform function.

12. The device according to claim 9, wherein: The generation module comprises: A fusion submodule is configured to fuse the original point cloud points with the diffused point cloud points to generate fused point cloud data; The generating submodule is configured to generate enhanced point cloud data based on the fused point cloud data.

13. The device according to claim 12, wherein: The generation submodule is further configured to: In response to determining that the number of point cloud diffusions does not reach a preset number threshold, taking the fused point cloud data as new original point cloud data, and performing point cloud diffusion again to obtain new fused point cloud data; In response to determining that the number of point cloud diffusion times reaches the preset number threshold, enhanced point cloud data is generated based on the fused point cloud data.

14. The device according to claim 12 or 13, wherein: The generation submodule is further configured to: The fused point cloud data is subjected to denoising processing to generate the enhanced point cloud data.

15. The device according to claim 14, wherein: The generation submodule is further configured to: The fused point cloud data is input into the U-Net network to obtain the enhanced point cloud data.

16. The device according to claim 15, wherein: The U-Net network includes an encoder and a decoder; and The generation submodule is further configured to: Using the encoder to perform feature extraction on the fused point cloud data to obtain a feature representation of the fused point cloud data; The decoder is used to map the feature representation back to the size of the fused point cloud data to obtain the enhanced point cloud data.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images