Point cloud enhancement method and apparatus

Abstract

This disclosure provides a point cloud enhancement method and apparatus, relating to the field of artificial intelligence technology, particularly computer vision and deep learning technology, and applicable to scenarios such as autonomous driving, drone detection and obstacle avoidance, intelligent security, and industrial automation. One specific implementation of the method includes: using surrounding point cloud points in the original point cloud data to diffuse the original point cloud points, obtaining diffused point cloud points; and generating enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision and deep learning technology, and can be applied to scenarios such as autonomous driving, drone detection and obstacle avoidance, intelligent security and industrial automation. Background Technology

[0002] Radar, a transliteration of the English word "radar," is an abbreviation of "radio detection and ranging," meaning "radio detection and ranging," that is, using radio waves to detect targets and determine their spatial location. Therefore, radar is also known as "radio positioning." Radar is an electronic device that uses electromagnetic waves to detect targets. It emits electromagnetic waves to illuminate a target and receives its echo, thereby obtaining information such as the distance from the target to the electromagnetic wave emission point, the rate of change of distance (radial velocity), azimuth, and altitude.

[0003] Millimeter-wave radar, due to its inherent physical characteristics such as lower resolution and sparsity, often generates point cloud data that contains limited information, restricting its application in fields such as autonomous driving and robotics. Traditional solutions typically rely on expensive lidar to acquire high-resolution point cloud data. However, the high cost of lidar limits its widespread adoption. Summary of the Invention

[0004] This disclosure provides a point cloud enhancement method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, embodiments of this disclosure propose a point cloud enhancement method, comprising: using the surrounding point cloud points of the original point cloud points in the original point cloud data to perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points; and generating enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0006] Secondly, embodiments of this disclosure propose a point cloud enhancement device, comprising: a diffusion module configured to diffuse the original point cloud points using surrounding point cloud points in the original point cloud data to obtain diffused point cloud points; and a generation module configured to generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described in the first aspect.

[0009] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0010] The key or essential features of the embodiments disclosed herein are not intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein:

[0012] Figure 1 This is a flowchart of an embodiment of the point cloud enhancement method according to the present disclosure;

[0013] Figure 2 This is a flowchart of yet another embodiment of the point cloud enhancement method according to the present disclosure;

[0014] Figure 3 This is a flowchart of another embodiment of the point cloud enhancement method according to the present disclosure;

[0015] Figure 4 This is a scene diagram that can implement the point cloud enhancement method disclosed herein;

[0016] Figure 5 This is a schematic diagram of a structure of an embodiment of the point cloud enhancement device according to the present disclosure;

[0017] Figure 6 This is a block diagram of an electronic device used to implement the point cloud enhancement method of the embodiments of this disclosure. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flow 100 of an embodiment of a point cloud enhancement method according to the present disclosure is shown. The point cloud enhancement method includes the following steps:

[0021] Step 101: Using the surrounding point cloud points in the original point cloud data, perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points.

[0022] In this embodiment, the entity executing the point cloud enhancement method can use the surrounding point cloud points in the original point cloud data to diffuse the original point cloud points and obtain diffused point cloud points.

[0023] The execution entity of point cloud augmentation methods is typically a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0024] Raw point cloud data can be point cloud data acquired using millimeter-wave radar. Due to limitations in the physical characteristics and environmental factors of millimeter-wave radar, raw point cloud data often has low resolution and sparsity, making it difficult to accurately represent the detailed features of objects.

[0025] Raw point cloud data can consist of a series of discrete raw point cloud points (x... i ,y i ,z i It consists of ) where i represents the index of the original point cloud points. The original point cloud data is unevenly distributed in three-dimensional space and may contain noise and missing data.

[0026] Point cloud diffusion of raw point cloud data can make the data smoother and more continuous in three-dimensional space, thereby improving the resolution and density of the point cloud data. For a raw point cloud point, its surrounding point cloud points can be used to perform point cloud diffusion, generating diffused point cloud points. In this way, the diffused point cloud points are related to both the original point cloud point and its surrounding point cloud points. The surrounding point cloud points can be other raw point cloud points located near the original point cloud point, for example, those located at a distance less than a preset distance threshold.

[0027] In some embodiments, for an original point cloud point in the original point cloud data, the average coordinates of the surrounding point cloud points can be calculated to generate a new point cloud point, i.e., a diffused point cloud point.

[0028] In some embodiments, a diffusion kernel function K(x,y,z) is defined. The diffusion kernel function K(x,y,z) can describe the degree of influence of each point cloud point on its surrounding point cloud points. For the original point cloud points in the original point cloud data, the diffusion kernel function can be used to calculate the weighted sum of the surrounding point cloud points to obtain the diffused point cloud points. The weighted sum formula can be as follows:

[0029]

[0030] Among them, (x i ,y i ,z i ) represents the original point cloud points, and i represents the original point cloud points (x i ,y i ,z i The index of (x) j ,y j ,z j ) represents the original point cloud points (x) i ,y i ,z i The surrounding point cloud points of ), where j represents the surrounding point cloud points (x). j ,y j ,z j The index of (x') i ,y' i ,z' i ) represents the diffuse point cloud points.

[0031] In three-dimensional space, the diffusion kernel function K(x,y,z) can include, but is not limited to, Gaussian functions, uniform functions, or other suitable functions. Because the Gaussian function possesses smoothness and locality, it can effectively simulate the diffusion process of point cloud data; therefore, the Gaussian function can be chosen as the diffusion kernel function. In this case, the diffusion kernel function can be as follows:

[0032]

[0033] Where K(x,y,z) represents the diffusion kernel function, and σ represents the standard deviation of the diffusion kernel, which is used to determine the range and intensity of diffusion.

[0034] Step 102: Generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0035] In this embodiment, the aforementioned execution entity can generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0036] In some embodiments, fusing the original point cloud points and the diffused point cloud points can generate fused point cloud data. Enhanced point cloud data can then be generated based on the fused point cloud data. For example, the fused point cloud data obtained by fusing the original and diffused point cloud points can be directly used as enhanced point cloud data. In this case, the number of point clouds in the enhanced point cloud data is significantly increased compared to the number of point clouds in the original point cloud data, thereby improving the resolution and density of the point cloud data. Another example is denoising the fused point cloud data to generate enhanced point cloud data. In this case, the diffusion process can smoothly propagate information and reduce the impact of noise, thereby improving the accuracy and reliability of the point cloud data.

[0037] This disclosure provides a point cloud enhancement method that utilizes point cloud data generated by relatively inexpensive millimeter-wave radar. Through point cloud diffusion processing, it generates high-resolution point cloud data comparable to or better than that of lidar, enabling it to more accurately represent the shape and detailed features of objects. This process not only improves the data quality of radar point clouds but also makes them applicable to more downstream tasks, such as object detection, tracking, and map building.

[0038] The point cloud enhancement method provided in this disclosure can achieve the following objectives:

[0039] 1. Improve radar point cloud resolution: Generate denser and more detailed point cloud data than the original radar point cloud, thereby improving data availability.

[0040] 2. Enhance the information content of radar point clouds: By enhancing resolution, capture more detailed information, such as the edges, shapes and textures of objects, which are crucial for subsequent perception and decision-making tasks.

[0041] 3. Reduced costs: Compared with relying on expensive LiDAR, it provides a more economical and practical solution, making the acquisition of high-resolution point cloud data easier and more feasible.

[0042] 4. Improve the performance of downstream tasks: By enhancing the data quality of radar point clouds, improve the performance of downstream tasks that rely on point cloud data, such as object detection, tracking, and path planning.

[0043] The difference compared to existing technologies is:

[0044] Existing technologies typically involve post-processing of already generated point cloud data, such as filtering and interpolation. However, the embodiments disclosed in this disclosure directly apply a 3D diffusion method to the radar echo signal, improving the resolution of the point cloud data from the source.

[0045] The embodiments disclosed herein employ a 3D diffusion method, which can more comprehensively capture the three-dimensional features of point cloud data and improve the resolution effect.

[0046] Continue to refer to Figure 2 This illustrates a flow 200 of yet another embodiment of the point cloud enhancement method according to the present disclosure. The point cloud enhancement method includes the following steps:

[0047] Step 201: Using the diffusion kernel function, calculate the weighted sum of the surrounding point cloud points of the original point cloud points in the original point cloud data to obtain the diffused point cloud points.

[0048] In this embodiment, the execution entity of the point cloud enhancement method can use a diffusion kernel function to calculate the weighted sum of the surrounding point cloud points of the original point cloud points in the original point cloud data to obtain the diffused point cloud points.

[0049] The execution entity of point cloud augmentation methods is typically a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0050] Raw point cloud data can be point cloud data acquired using millimeter-wave radar. Due to limitations in the physical characteristics and environmental factors of millimeter-wave radar, raw point cloud data often has low resolution and sparsity, making it difficult to accurately represent the detailed features of objects.

[0051] Raw point cloud data can consist of a series of discrete raw point cloud points (x... i ,y i ,z i It consists of ) where i represents the index of the original point cloud points. The original point cloud data is unevenly distributed in three-dimensional space and may contain noise and missing data.

[0052] Point cloud diffusion of raw point cloud data can make the data smoother and more continuous in three-dimensional space, thereby improving the resolution and density of the point cloud data. For a raw point cloud point, its surrounding point cloud points can be used to perform point cloud diffusion, generating diffused point cloud points. In this way, the diffused point cloud points are related to both the original point cloud point and its surrounding point cloud points. The surrounding point cloud points can be other raw point cloud points located near the original point cloud point, for example, those located at a distance less than a preset distance threshold.

[0053] Define a diffusion kernel function K(x,y,z). The diffusion kernel function K(x,y,z) can describe the degree of influence of each point cloud point on its surrounding point cloud points. For the original point cloud points in the original point cloud data, the diffusion kernel function can be used to calculate the weighted sum of the surrounding point cloud points to obtain the diffused point cloud points. The weighted sum formula can be as follows:

[0054]

[0055] Among them, (x i ,y i ,z i ) represents the original point cloud points, and i represents the original point cloud points (x i ,y i ,z i The index of (x) j ,y j ,z j ) represents the original point cloud points (x) i ,y i ,z i The surrounding point cloud points of ), where j represents the surrounding point cloud points (x). j ,y j ,z j The index of (x') i ,y' i ,z' i ) represents the diffuse point cloud points.

[0056] In three-dimensional space, the diffusion kernel function K(x,y,z) can include, but is not limited to, Gaussian functions, uniform functions, or other suitable functions. Because the Gaussian function possesses smoothness and locality, it can effectively simulate the diffusion process of point cloud data; therefore, the Gaussian function can be chosen as the diffusion kernel function. In this case, the diffusion kernel function can be as follows:

[0057]

[0058] Where K(x,y,z) represents the diffusion kernel function, and σ represents the standard deviation of the diffusion kernel, which is used to determine the range and intensity of diffusion.

[0059] Step 202: Merge the original point cloud points and the diffused point cloud points to generate merged point cloud data.

[0060] In this embodiment, the aforementioned execution entity can fuse the original point cloud points and the diffused point cloud points to generate fused point cloud data.

[0061] Step 203: Determine whether the number of point cloud diffusions has reached the preset threshold.

[0062] In this embodiment, the execution entity can determine whether the number of point cloud diffusions has reached a preset threshold. If the number of point cloud diffusions has not reached the preset threshold, step 204 is executed; if the number of point cloud diffusions has reached the preset threshold, step 205 is executed.

[0063] Step 204: Use the fused point cloud data as the new original point cloud data.

[0064] In this embodiment, if the number of point cloud diffusions does not reach the preset threshold, the aforementioned execution entity can use the fused point cloud data as new original point cloud data and return to execution step 201 to perform the next round of point cloud diffusion.

[0065] The iteration number threshold and the parameters of the diffusion kernel may need to be adjusted according to the specific application scenario and data characteristics.

[0066] For example, point cloud data is commonly used in target recognition applications. These applications require target detection within the point cloud data. Therefore, for each target type, millimeter-wave radar can be used to collect point cloud data, which is then diffused multiple times. Target detection can be performed on the point cloud data obtained from each round of diffusion, yielding the detection results. Based on the detection results and the true target category, the accuracy of the target detection can be calculated. If the target detection accuracy reaches a preset accuracy threshold, the current iteration count can be used as the threshold for the number of iterations for that target type's point cloud data.

[0067] For example, the parameters of the diffusion kernel can include the number of diffusion steps and the noise intensity. Based on the number of diffusion steps, a noise intensity function can be set such that the noise intensity gradually decreases as the number of diffusion steps increases.

[0068] The formula for the noise intensity function β(t) can be as follows:

[0069] β(t)=β0(1-t / T) 2 .

[0070] Where β0 is the initial noise, t is the number of diffusion steps, and T is the total number of diffusion steps.

[0071] During each diffusion step, the variance of the model output is monitored. If the variance is too large, the noise intensity of the current step is increased appropriately to enhance stability; conversely, the noise intensity is decreased.

[0072] Furthermore, by utilizing prior knowledge of environmental geometry and material properties, reasonable noise intensities for different regions can be estimated. For example, a lower noise intensity can be used for flat areas, while a higher noise intensity can be used for edge areas. Step 205: Generate enhanced point cloud data based on the fused point cloud data.

[0073] In this embodiment, if the number of point cloud diffusions reaches a preset threshold, the aforementioned execution entity can generate enhanced point cloud data based on the fused point cloud data.

[0074] For example, the fused point cloud data obtained by fusing the original point cloud points and the diffused point cloud points can be directly used as the enhanced point cloud data. In this case, the number of point cloud points in the enhanced point cloud data is greatly increased compared to the number of point cloud points in the original point cloud data, thereby improving the resolution and density of the point cloud data.

[0075] For example, denoising the fused point cloud data can generate enhanced point cloud data. In this case, the diffusion process can smoothly propagate information and reduce the impact of noise, thereby improving the accuracy and reliability of the point cloud data.

[0076] from Figure 2 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the point cloud enhancement method in this embodiment emphasizes the iterative diffusion step in its process flow 200. Therefore, the scheme described in this embodiment performs multiple iterative diffusions on the point cloud data. After each iteration, the resolution of the point cloud data is improved to a certain extent, thereby enhancing the diffusion effect.

[0077] Further reference Figure 3 This illustrates a flow 300 of another embodiment of the point cloud enhancement method according to the present disclosure. The point cloud enhancement method includes the following steps:

[0078] Step 301: Using the diffusion kernel function, calculate the weighted sum of the surrounding point cloud points of the original point cloud points in the original point cloud data to obtain the diffused point cloud points.

[0079] In this embodiment, the execution entity of the point cloud enhancement method can use a diffusion kernel function to calculate the weighted sum of the surrounding point cloud points of the original point cloud points in the original point cloud data to obtain the diffused point cloud points.

[0080] The execution entity of point cloud augmentation methods is typically a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0081] Raw point cloud data can be point cloud data acquired using millimeter-wave radar. Due to limitations in the physical characteristics and environmental factors of millimeter-wave radar, raw point cloud data often has low resolution and sparsity, making it difficult to accurately represent the detailed features of objects.

[0082] Raw point cloud data can consist of a series of discrete raw point cloud points (x...i ,y i ,z i It consists of ) where i represents the index of the original point cloud points. The original point cloud data is unevenly distributed in three-dimensional space and may contain noise and missing data.

[0083] Point cloud diffusion of raw point cloud data can make the data smoother and more continuous in three-dimensional space, thereby improving the resolution and density of the point cloud data. For a raw point cloud point, its surrounding point cloud points can be used to perform point cloud diffusion, generating diffused point cloud points. In this way, the diffused point cloud points are related to both the original point cloud point and its surrounding point cloud points. The surrounding point cloud points can be other raw point cloud points located near the original point cloud point, for example, those located at a distance less than a preset distance threshold.

[0084] Define a diffusion kernel function K(x,y,z). The diffusion kernel function K(x,y,z) can describe the degree of influence of each point cloud point on its surrounding point cloud points. For the original point cloud points in the original point cloud data, the diffusion kernel function can be used to calculate the weighted sum of the surrounding point cloud points to obtain the diffused point cloud points. The weighted sum formula can be as follows:

[0085]

[0086] Among them, (x i ,y i ,z i ) represents the original point cloud points, and i represents the original point cloud points (x i ,y i ,z i The index of (x) j ,y j ,z j ) represents the original point cloud points (x) i ,y i ,z i The surrounding point cloud points of ), where j represents the surrounding point cloud points (x). j ,y j ,z j The index of (x') i ,y' i ,z' i ) represents the diffuse point cloud points.

[0087] In three-dimensional space, the diffusion kernel function K(x,y,z) can include, but is not limited to, Gaussian functions, uniform functions, or other suitable functions. Because the Gaussian function possesses smoothness and locality, it can effectively simulate the diffusion process of point cloud data; therefore, the Gaussian function can be chosen as the diffusion kernel function. In this case, the diffusion kernel function can be as follows:

[0088]

[0089] Where K(x,y,z) represents the diffusion kernel function, and σ represents the standard deviation of the diffusion kernel, which is used to determine the range and intensity of diffusion.

[0090] Step 302: Merge the original point cloud points and the diffused point cloud points to generate merged point cloud data.

[0091] In this embodiment, the aforementioned execution entity can fuse the original point cloud points and the diffused point cloud points to generate fused point cloud data.

[0092] Step 303: Determine whether the number of point cloud diffusions has reached the preset threshold.

[0093] In this embodiment, the execution entity can determine whether the number of point cloud diffusions has reached a preset threshold. If the number of point cloud diffusions has not reached the preset threshold, step 304 is executed; if the number of point cloud diffusions has reached the preset threshold, step 305 is executed.

[0094] Step 304: Use the fused point cloud data as the new original point cloud data.

[0095] In this embodiment, if the number of point cloud diffusion attempts does not reach the preset threshold, the aforementioned execution entity can use the fused point cloud data as new original point cloud data and return to execution step 301 to perform the next round of point cloud diffusion.

[0096] The iteration number threshold and the parameters of the diffusion kernel may need to be adjusted according to the specific application scenario and data characteristics.

[0097] For example, point cloud data is commonly used in target recognition applications. These applications require target detection within the point cloud data. Therefore, for each target type, millimeter-wave radar can be used to collect point cloud data, which is then diffused multiple times. Target detection can be performed on the point cloud data obtained from each round of diffusion, yielding the detection results. Based on the detection results and the true target category, the accuracy of the target detection can be calculated. If the target detection accuracy reaches a preset accuracy threshold, the current iteration count can be used as the threshold for the number of iterations for that target type's point cloud data.

[0098] For example, the parameters of the diffusion kernel can include the number of diffusion steps and the noise intensity. Based on the number of diffusion steps, a noise intensity function can be set such that the noise intensity gradually decreases as the number of diffusion steps increases.

[0099] The formula for the noise intensity function β(t) can be as follows:

[0100] β(t)=β0(1-t / T) 2.

[0101] Where β0 is the initial noise, t is the number of diffusion steps, and T is the total number of diffusion steps.

[0102] During each diffusion step, the variance of the model output is monitored. If the variance is too large, the noise intensity of the current step is increased appropriately to enhance stability; conversely, the noise intensity is decreased.

[0103] Furthermore, prior knowledge of environmental geometry and material properties can be used to estimate the appropriate noise intensity for different regions. For example, a lower noise intensity can be used for flat areas, while a higher noise intensity can be used for edge areas. Step 305 involves inputting the fused point cloud data into the U-Net network to obtain the enhanced point cloud data.

[0104] In this embodiment, if the number of point cloud diffusions reaches a preset threshold, the aforementioned execution entity can input the fused point cloud data into the U-Net network to obtain enhanced point cloud data.

[0105] The U-Net network can be used in the backdiffusion process. The U-Net network has a U-shaped structure, consisting of a symmetrical encoder and decoder. The encoder extracts high-level feature representations from the fused point cloud data. The decoder maps these feature representations back to the dimensions of the fused point cloud data. The fused point cloud data obtained by the 3D diffusion method is input into the U-Net network for denoising. Through its powerful feature extraction and mapping capabilities, the U-Net network further removes noise from the point cloud data, improving its clarity and accuracy.

[0106] It should be noted that the point cloud enhancement method provided in this embodiment can either input only the fused point cloud data obtained from the last point cloud diffusion into the U-Net network for denoising processing, or input the fused point cloud data obtained from each point cloud diffusion into the U-Net network for denoising processing; there is no limitation here.

[0107] from Figure 3 It can be seen from this that, with Figure 2 Compared to the corresponding embodiments, the point cloud enhancement method in this embodiment emphasizes the denoising step in process 300. Therefore, the solution described in this embodiment leverages the powerful feature extraction and mapping capabilities of the U-Net network to further remove noise from point cloud data, improving the clarity and accuracy of the point cloud data.

[0108] Figure 4 A scene diagram is shown that allows the point cloud augmentation method of this disclosure to be implemented. For example... Figure 4As shown, in the initial stage, millimeter-wave radar is used to acquire signals and obtain raw point cloud data. Then, for the raw point cloud data, 3D diffusion is first performed using a diffusion kernel function, followed by inverse diffusion using a U-Net network. After multiple iterations of diffusion, enhanced point cloud data is obtained. Finally, the enhanced point cloud data undergoes post-processing and can be used for applications such as target recognition.

[0109] When used in applications such as target recognition, it can enhance the point cloud resolution of millimeter-wave radar, enabling the system to identify targets more accurately and improving the precision and reliability of target detection. In complex environments and under interference conditions, it can optimize the radar signal processing, improving the system's robustness and stability.

[0110] Common application scenarios for target recognition may include:

[0111] 1. Autonomous Driving: Autonomous vehicles need to perceive their surroundings in real time, including roads, vehicles, and pedestrians. Millimeter-wave radar, as one of the sensors for autonomous driving, can provide information such as distance and speed. By using point cloud diffusion, the point cloud resolution of millimeter-wave radar can be enhanced, improving the accuracy of target detection, thereby enhancing the safety and reliability of the autonomous driving system.

[0112] 2. Drone Detection and Obstacle Avoidance: Drones need to detect and avoid obstacles during flight. Millimeter-wave radar can help drones perceive the environment ahead, and point cloud diffusion can further enhance the radar's perception capabilities, enabling drones to more accurately identify obstacles and make obstacle avoidance decisions.

[0113] 3. Intelligent Security: In the field of intelligent security, millimeter-wave radar can be used to monitor and detect intruders. Point cloud diffusion can enhance the radar's point cloud resolution, improve the accuracy of intruder detection, and provide more accurate target information for security systems.

[0114] 4. Industrial Automation: In the field of industrial automation, millimeter-wave radar can be used in scenarios such as material handling and robot navigation. Point cloud diffusion can enhance the radar's perception capabilities, enabling robots to more accurately identify objects and paths, thereby improving production efficiency.

[0115] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a point cloud enhancement device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0116] like Figure 5As shown, the point cloud enhancement device 500 of this embodiment may include a diffusion module 501 and a generation module 502. The diffusion module 501 is configured to diffuse the original point cloud points using surrounding point cloud points in the original point cloud data to obtain diffused point cloud points; the generation module 502 is configured to generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points.

[0117] In this embodiment, the specific processing of the diffusion module 501 and the generation module 502 in the point cloud enhancement device 500 and the resulting technical effects can be referred to respectively. Figure 1 The relevant descriptions of steps 101-102 in the corresponding embodiments will not be repeated here.

[0118] In some optional implementations of this embodiment, the diffusion module 501 is further configured to: use a diffusion kernel function to calculate the weighted sum of the surrounding point cloud points of the original point cloud points to obtain the diffused point cloud points, wherein the diffusion kernel function describes the degree of influence of each point cloud point on the surrounding point cloud points.

[0119] In some optional implementations of this embodiment, the diffusion kernel function includes a Gaussian function or a uniform function.

[0120] In some optional implementations of this embodiment, the generation module 502 includes: a fusion submodule configured to fuse the original point cloud points and the diffused point cloud points to generate fused point cloud data; and a generation submodule configured to generate enhanced point cloud data based on the fused point cloud data.

[0121] In some optional implementations of this embodiment, the generation submodule is further configured to: in response to determining that the number of point cloud diffusions has not reached a preset threshold, use the fused point cloud data as new original point cloud data and perform point cloud diffusion again to obtain new fused point cloud data; in response to determining that the number of point cloud diffusions has reached a preset threshold, generate enhanced point cloud data based on the fused point cloud data.

[0122] In some optional implementations of this embodiment, the generation submodule is further configured to: perform denoising processing on the fused point cloud data to generate enhanced point cloud data.

[0123] In some optional implementations of this embodiment, the generation submodule is further configured to input the fused point cloud data into the U-Net network to obtain enhanced point cloud data.

[0124] In some optional implementations of this embodiment, the U-Net network includes an encoder and a decoder; and the generation submodule is further configured to: use the encoder to extract features from the fused point cloud data to obtain a feature representation of the fused point cloud data; and use the decoder to map the feature representation back to the size of the fused point cloud data to obtain the enhanced point cloud data.

[0125] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0126] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0127] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0128] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0129] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0130] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as point cloud enhancement methods. For example, in some embodiments, the point cloud enhancement method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the point cloud enhancement method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform point cloud enhancement methods by any other suitable means (e.g., by means of firmware).

[0131] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0133] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0135] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0136] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0137] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0138] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A point cloud enhancement method, comprising: By using the surrounding point cloud points in the original point cloud data, the original point cloud points are diffused to obtain diffused point cloud points. Based on the original point cloud points and the diffused point cloud points, enhanced point cloud data is generated; The process of generating enhanced point cloud data based on the original point cloud points and the diffused point cloud points includes: The original point cloud points and the diffused point cloud points are fused to generate fused point cloud data; In response to the determination that the number of point cloud diffusions has not reached the preset threshold, the fused point cloud data is used as new original point cloud data, and point cloud diffusion is performed again to obtain new fused point cloud data; In response to the determination that the number of point cloud diffusions has reached the preset threshold, enhanced point cloud data is generated based on the fused point cloud data; The process of generating enhanced point cloud data based on the fused point cloud data includes: The fused point cloud data is input into the U-Net network to obtain the enhanced point cloud data; The step of using surrounding point cloud points from the original point cloud data to perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points includes: Using a diffusion kernel function, the weighted sum of the surrounding point cloud points of the original point cloud point is calculated to obtain the diffused point cloud point, wherein the diffusion kernel function describes the degree of influence of each point cloud point on the surrounding point cloud points.

2. The method of claim 1, wherein, The diffusion kernel function includes a Gaussian function or a uniform function.

3. The method of claim 1, wherein, The U-Net network includes encoders and decoders; as well as The step of inputting the fused point cloud data into the U-Net network to obtain the enhanced point cloud data includes: The encoder is used to extract features from the fused point cloud data to obtain a feature representation of the fused point cloud data. The enhanced point cloud data is obtained by mapping the feature representation back to the dimensions of the fused point cloud data using the decoder.

4. A point cloud enhancement device, comprising: The diffusion module is configured to use the surrounding point cloud points of the original point cloud points in the original point cloud data to perform point cloud diffusion on the original point cloud points to obtain diffused point cloud points. The generation module is configured to generate enhanced point cloud data based on the original point cloud points and the diffused point cloud points; The generation module includes: The fusion submodule is configured to fuse the original point cloud points and the diffused point cloud points to generate fused point cloud data; The generation submodule is configured to, in response to determining that the number of point cloud diffusions has not reached a preset threshold, use the fused point cloud data as new original point cloud data and perform point cloud diffusion again to obtain new fused point cloud data; and in response to determining that the number of point cloud diffusions has reached the preset threshold, generate enhanced point cloud data based on the fused point cloud data. The generation submodule is further configured to: The fused point cloud data is input into the U-Net network to obtain the enhanced point cloud data; The diffusion module is further configured to: Using a diffusion kernel function, the weighted sum of the surrounding point cloud points of the original point cloud point is calculated to obtain the diffused point cloud point, wherein the diffusion kernel function describes the degree of influence of each point cloud point on the surrounding point cloud points.

5. The apparatus of claim 4, wherein, The diffusion kernel function includes a Gaussian function or a uniform function.

6. The apparatus of claim 4, wherein, The U-Net network includes an encoder and a decoder; and The generation submodule is further configured to: The encoder is used to extract features from the fused point cloud data to obtain a feature representation of the fused point cloud data. The enhanced point cloud data is obtained by mapping the feature representation back to the dimensions of the fused point cloud data using the decoder.

7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-3.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-3.

Images