Point cloud generation method and device based on millimeter wave and visual image fusion

Abstract

The invention provides a point cloud generation method and device based on millimeter wave and visual image fusion. Acquiring laser radar data of a historical shielding scene, and generating a target supervision point cloud based on the laser radar data; identifying the historical shielding scene based on a millimeter wave radar and a visual camera to obtain historical millimeter wave features and historical visual features; taking the target supervision point cloud as decoder supervision, inputting the historical millimeter wave features and the historical visual features into a decoder, and training the decoder to predict and generate a shielded target point cloud and an unshielded target point cloud; and identifying a to-be-identified shielding scene based on the millimeter wave radar and the visual camera to obtain millimeter wave features and visual features, and generating a scene point cloud of the to-be-identified shielding scene based on the millimeter wave features, the visual features and the trained decoder prediction. The point cloud generation method and device based on millimeter wave and visual image fusion are used for generating a high-precision point cloud in a complex scene.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for generating a point cloud based on the fusion of millimeter waves and visual images. Background Art

[0002] With the rapid development of technologies such as autonomous driving, intelligent robots, and augmented reality, the accuracy of environmental perception is increasingly required. In these scenarios, it is crucial to accurately identify obstacles, vehicles, pedestrians, and other targets in the surrounding environment.

[0003] In practical applications, laser radar and visual cameras are usually used as sensors for object detection. Laser radar acquires three-dimensional point cloud data of the surrounding environment by emitting laser pulses and receiving their reflected signals. It has the advantages of high precision and high density, but it is expensive and easily affected by weather conditions. It cannot accurately complete the construction of point clouds when there is occlusion. Visual cameras provide rich visual information by capturing images. They are low-cost and have a certain adaptability to environmental changes, but their effectiveness will be greatly reduced when there is insufficient light or occlusion. Therefore, traditional single sensors are difficult to meet the needs of high-precision perception in complex scenarios and cannot meet the usage standards. Summary of the invention

[0004] In view of this, the present application provides a point cloud generation method and device based on millimeter wave and visual image fusion, which is used to generate high-precision point clouds in complex scenes.

[0005] Specifically, the present application is implemented through the following technical solutions: The first aspect of the present application provides a point cloud generation method based on millimeter wave and visual image fusion, the method comprising: Acquire laser radar data of historical occlusion scenes, and generate a target supervision point cloud based on the laser radar data; The point cloud density of the occluded target in the laser radar data is less than a first preset threshold, the point cloud density of the occluded target in the target supervision point cloud is greater than the first preset threshold, and the point cloud contour of the occluded target in the target supervision point cloud is complete; Based on the millimeter wave radar and the visual camera, the historical occlusion scene is respectively identified to obtain the historical millimeter wave feature and the historical visual feature; Using the target supervision point cloud as a decoder supervision, inputting the historical millimeter wave features and the historical visual features into a decoder, and training the decoder to predict and generate an occluded target point cloud and an unoccluded target point cloud; Based on the millimeter wave radar and the visual camera, the occluded scene to be identified is identified to obtain millimeter wave features and visual features, and based on the millimeter wave features, the visual features and the trained decoder prediction, a scene point cloud of the occluded scene to be identified is generated.

[0006] The second aspect of the present application provides a point cloud generation device based on millimeter wave and visual image fusion, the device comprising an acquisition module, a recognition module, a prediction module and a generation module; wherein, The acquisition module is used to acquire the laser radar data of the historical occlusion scene and generate the target supervision point cloud based on the laser radar data; The point cloud density of the occluded target in the laser radar data is less than a first preset threshold, the point cloud density of the occluded target in the target supervision point cloud is greater than the first preset threshold, and the point cloud contour of the occluded target in the target supervision point cloud is complete; The recognition module is used to respectively recognize the historical occlusion scene based on the millimeter wave radar and the visual camera to obtain the historical millimeter wave features and the historical visual features; The prediction module is used to use the target supervision point cloud as a decoder supervision, input the historical millimeter wave features and the historical visual features into a decoder, and train the decoder to predict and generate an occluded target point cloud and an unoccluded target point cloud; The generation module is used to identify the occluded scene to be identified based on the millimeter wave radar and the visual camera, obtain millimeter wave features and visual features, and generate a scene point cloud of the occluded scene to be identified based on the millimeter wave features, the visual features and the trained decoder prediction.

[0007] The point cloud generation method and device based on the fusion of millimeter wave and visual image provided by the present application first obtain the laser radar data of the historical occlusion scene, and generate the target supervision point cloud based on the laser radar data; wherein, the point cloud density of the occluded target in the laser radar data is less than the first preset threshold, the point cloud density of the occluded target in the target supervision point cloud is greater than the first preset threshold, and the point cloud contour of the occluded target in the target supervision point cloud is complete; then the historical occlusion scene is identified based on the millimeter wave radar and the visual camera, and the historical millimeter wave features and the historical visual features are obtained; then the target supervision point cloud is used as the decoder supervision, the historical millimeter wave features and the historical visual features are input into the decoder, and the decoder is trained to predict and generate the occluded target point cloud and the unoccluded target point cloud; finally, the occluded scene to be identified is identified based on the millimeter wave radar and the visual camera, and the millimeter wave features and the visual features are obtained, and the scene point cloud of the occluded scene to be identified is predicted based on the millimeter wave features, the visual features and the trained decoder. The method provided by the present invention uses the point cloud information of the laser radar as the supervision data during training, and comprehensively uses the image information of the millimeter wave radar and the visual camera to generate the complete point cloud information of the occluded scene, thereby improving the integrity and accuracy of the point cloud finally generated. Specifically, laser radar is good at detecting the boundaries of objects. Therefore, the contour information obtained by laser radar is used as the boundary and position hint of point cloud generation to improve the accuracy of the point cloud predicted by the encoder and decoder. Furthermore, millimeter wave radar can obtain clear image information under harsh conditions such as low light, improve the adaptability of the point cloud generation method, and complete the unobstructed information collection under any circumstances. Finally, the visual camera can quickly obtain clear image information at a low cost. The method provided by the present invention utilizes the advantages of three types of information acquisition means, which not only provides sufficient input image information for the input predicted by the encoder and decoder, but also helps the encoder and decoder to quickly locate the relevant image area and contour information, thereby improving the efficiency and accuracy of point cloud generation. In this way, the millimeter wave features of the obscured part can be effectively obtained through the penetration ability of the millimeter wave radar, and at the same time, the visual features containing rich contextual information can be effectively obtained through the high-level semantic information of the visual camera. Furthermore, the occluded target point cloud and the unobstructed target point cloud generated by the millimeter wave features and visual features not only contain the information of the occluded scene, but also integrate the semantic information of the visual image, so that the generated point cloud is closer to the real occluded scene to be identified. Furthermore, the accuracy and robustness of the scene point cloud can be guaranteed through the supervision of the target supervision point cloud. In this way, the combination of millimeter wave radar and visual images can fully integrate the information of the occluded part and other parts of the occluded scene to be identified, ensuring the accuracy and reliability of the scene point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flowchart of Example 1 of a point cloud generation method based on millimeter wave and visual image fusion provided in this application; Figure 2 A hardware structure diagram of a point cloud generation device based on the fusion of millimeter waves and visual images in the present application; Figure 3 This is a structural diagram of Example 1 of a point cloud generation device based on millimeter wave and visual image fusion provided in this application. DETAILED DESCRIPTION

[0009] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0010] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0011] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0012] The present application provides a point cloud generation method and device based on millimeter wave and visual image fusion, which are used to generate high-precision point clouds in complex scenes.

[0013] Specific embodiments are given below to introduce the technical solution of the present application in detail.

[0014] Figure 1 This is a flowchart of the first embodiment of the point cloud generation method based on millimeter wave and visual image fusion provided by this application. Please refer to Figure 1 , the method provided in this embodiment may include: S101, obtaining laser radar data of historical occlusion scenes, and generating a target supervision point cloud based on the laser radar data; Among them, the point cloud density of the occluded target in the laser radar data is less than the first preset threshold, the point cloud density of the occluded target in the target supervision point cloud is greater than the first preset threshold, and the point cloud contour of the occluded target in the target supervision point cloud is complete.

[0015] Specifically, historical occlusion scenarios refer to scenarios in which the detector cannot directly obtain complete data due to occlusion by other objects in environmental perception tasks. In other words, the targets in these scenarios may be partially or completely invisible due to occlusion by other objects. It should be noted that in the scene of object recognition, accurately identifying and understanding the targets in these occlusion scenarios is crucial to ensure the safety and reliability of the system.

[0016] In specific implementation, all scenes that have been detected historically may be screened, and the parts that meet the occlusion scene requirements may be determined as historical occlusion scenes.

[0017] Furthermore, the lidar data is obtained by a lidar sensor. It should be noted that lidar is a sensor that calculates distance and speed by emitting a laser beam and measuring its reflection time.

[0018] In specific implementation, the lidar can emit laser pulses to the historical occlusion scene and receive the signals reflected from the surfaces of objects in the historical occlusion scene. By measuring the round-trip time of the laser pulses, the lidar can calculate the distance to each point in the surrounding environment and record these points in the form of three-dimensional coordinates based on the angle and height information of the laser emission.

[0019] Furthermore, the target supervision point cloud is generated based on the LiDAR data. It should be noted that the target supervision point cloud contains all the historical occlusion scenes. Supervision through the target supervision point cloud can ensure that the supervised target is as close to the target supervision point cloud as possible, thus being closer to the real historical occlusion scene.

[0020] In specific implementation, the specific value of the first preset threshold is determined according to actual needs, and is not limited in this embodiment. For example, in one embodiment, the first preset threshold can be determined according to the minimum point cloud density value required for object detection.

[0021] Furthermore, the point cloud density of the point cloud corresponding to the occluded target in the lidar data is low due to occlusion, and the point cloud density of all positions in the target supervision point cloud should be greater than the first preset threshold. Therefore, after converting the lidar data into a point cloud, the point cloud needs to be filled to prevent the point cloud from being too sparse and the outline from being incomplete.

[0022] It should be noted that the lidar data is used to supervise the data of the millimeter wave radar and the visual camera during the training process. It is understandable that when generating point clouds, accurate point cloud generation can be completed only through the data of the radar millimeter wave and the visual camera.

[0023] A specific embodiment is given below to introduce the acquisition process of the target supervision point cloud in detail: (1) Splitting the laser radar data into a specified number of target areas, and determining the target density of the point cloud in each target area.

[0024] Specifically, the specific value of the specified number is set according to actual needs, and is not limited in this embodiment. For example, in one embodiment, the specified number can be set to 6 according to the number of surfaces in the laser radar data.

[0025] It should be noted that when splitting the LiDAR data into target areas, it is necessary to ensure that the size and shape of each target area are exactly the same.

[0026] A specific embodiment is given below to introduce the process of splitting lidar data in detail: Step 1: Obtain the geometric center and corner points of the target object in the laser radar data.

[0027] Specifically, the target contained in the laser radar data is obtained. It should be noted that the specific type of the target object is determined according to actual needs, and this is not limited in this embodiment. In specific implementation, the target to be measured may include background, pedestrians, vehicles, other objects, etc. in the historical occlusion scene.

[0028] In specific implementation, the target contained in the laser radar data can be obtained based on a traditional target detection method, or based on a neural network target detection method. This is not limited in this embodiment.

[0029] Further, target detection is performed on the laser radar data to obtain all target objects therein. It should be noted that the detected target objects can be annotated with 3D boxes, and a corresponding target object can be completely divided into the 3D box by a 3D box.

[0030] Furthermore, the data corresponding to the target object is parsed into a series of three-dimensional coordinate points, through which the position and shape of the target object can be represented.

[0031] Step 2: Using the geometric center as the vertex of the target area, generating multiple bottom surfaces based on the corner points, and generating the target area based on the vertices and the bottom surfaces; wherein the number of the bottom surfaces is equal to the specified number; and each of the target areas has the same shape.

[0032] Specifically, the bottom surface composed of the corner points has the same size and shape, and the distance from the geometric center is also the same, so that each target area can be kept consistent.

[0033] In specific implementation, the corner points are used to generate the corresponding number of bottom surfaces according to the specified number, and the corner points of each bottom surface are connected to the geometric center to form the target area. For example, in one embodiment, if the specified number is 6, 6 bottom surfaces are generated according to 8 corner points, and then these 6 bottom surfaces are connected to the geometric center to form 6 complete three-dimensional target areas. At this time, the shape of the target area is a triangular pyramid.

[0034] It should be noted that the bottom surface can be determined based on the smallest 3D frame of the frame column target object. In specific implementation, a 3D frame can be generated based on a traditional 3D frame generation method, and the 8 corner points of the generated 3D frame are connected with the geometric center to form 6 triangular pyramids to obtain the target area.

[0035] The point cloud generation method based on the fusion of millimeter waves and visual images provided in this embodiment can obtain a target area of ​​exactly the same size and shape by obtaining the combined center and corner points. In this way, by completing the three-dimensional shape modeling through the target area, the geometric structure and morphology in the historical occlusion scene can be better understood, thereby providing strong support for the completion of the target supervision point cloud. Furthermore, by dividing the target area, the point cloud can be enhanced within the lidar data, ensuring the accuracy and reliability of data processing and reducing the computing cost.

[0036] (2) The target area corresponding to the target density less than the first preset threshold is determined as the area to be enhanced, and the number of point clouds in the area to be enhanced is increased to obtain the target supervision point cloud.

[0037] Specifically, the number of point clouds in the target area is calculated, and the number of point clouds is divided by the volume of the target area to obtain the corresponding target density. It should be noted that each target area has a corresponding target density.

[0038] Furthermore, the target area with a density lower than the target is determined as the area to be enhanced. In specific implementation, the number of point clouds in the area to be enhanced can be increased by using a traditional point cloud thickening method, or by using a neural network point cloud thickening method.

[0039] A specific embodiment is given below to introduce the acquisition process of the target supervision point cloud in detail: Step 1: Determine the target area corresponding to the target density greater than the first preset threshold as a sample area; wherein the sample area is of the same type as the area to be enhanced.

[0040] In a specific implementation, a target area with a target density greater than a first preset threshold is determined as a sample area. It can be understood that the number of point clouds in the sample area is large and the point cloud density is dense.

[0041] It should be noted that the sample region and the region to be enhanced are of the same type, have the same size and shape, and have the same type of corresponding specific data.

[0042] Step 2: Proportionally splice the point cloud in the sample area into the area to be enhanced to obtain the target supervised point cloud.

[0043] Specifically, the part of the point cloud with low density in the area to be enhanced needs to be supplemented, and the point cloud at the corresponding position of the area to be enhanced and the sample area is spliced ​​and copied to the sample area. It should be noted that the ratio of the number of point clouds spliced ​​each time is determined according to actual needs, and this is not limited in this embodiment. For example, in one embodiment, 30% of the point cloud at position A in the sample area can be extracted and spliced ​​to position A in the area to be enhanced.

[0044] Furthermore, after filling the point clouds of all positions in the area to be enhanced, the target supervision point cloud is obtained.

[0045] The point cloud generation method based on the fusion of millimeter waves and visual images provided in this embodiment is to splice the point cloud in the area to be enhanced by the sample area whose point cloud density meets the requirements, so that the point cloud density in the area to be enhanced meets the requirements. In this way, the target supervision point cloud can contain rich data, which can effectively reflect all the information in the occluded area to be identified, and provide high-quality supervision information. Furthermore, by splicing the point cloud in the area to be enhanced by the sample area, the boundary contour in each target area is guaranteed to be complete, and the edge blur is prevented, which can improve the accuracy and reliability of point cloud generation.

[0046] The point cloud generation method based on the fusion of millimeter waves and visual images provided in this embodiment first splits the laser radar data into a specified number of target areas, determines the target density of the point cloud in each target area, and then determines the target area corresponding to the target density less than the first preset threshold as the area to be enhanced, increases the number of point clouds in the area to be enhanced, and obtains the target supervision point cloud. In this way, by enhancing the laser radar data and completing the point cloud of the occluded target, the overall quality of the point cloud can be significantly improved. The enhanced point cloud L is more complete in shape and contour, has a higher density, and can better reflect the target situation in the real environment. Furthermore, through high-quality target supervision point clouds, a more accurate and reliable supervision signal can be provided for point cloud generation, which helps to improve the accuracy and robustness of point cloud generation.

[0047] S102: Identify the historical occlusion scene based on the millimeter wave radar and the visual camera respectively, and obtain historical millimeter wave features and historical visual features.

[0048] Specifically, millimeter-wave radar is a radar sensor that works in the millimeter-wave frequency band. Millimeter-wave radar uses electromagnetic waves for detection and ranging. It has the characteristics of stable detection performance, long effective distance, and good environmental applicability.

[0049] In specific implementation, when the electromagnetic waves emitted by the millimeter-wave radar encounter the target, they will be reflected and received by the radar. By measuring the time difference and phase difference between the transmitted wave and the reflected wave, information such as the distance and azimuth of the target can be calculated.

[0050] Furthermore, the visual camera collects light emitted from the surroundings and irradiates the light-sensitive surface through the camera lens, thereby generating a clear image. It should be noted that the visual camera has the advantages of low price and high resolution of environmental images.

[0051] In specific implementation, the specific models of the millimeter wave radar and the visual camera are determined according to actual needs, and are not limited in this embodiment. For example, in one embodiment, a monocular camera can be selected as the visual camera, or a binocular camera can be selected as the visual camera.

[0052] Furthermore, feature extraction is performed on the data captured by the millimeter-wave radar and the visual camera to obtain millimeter-wave features and visual features.

[0053] A specific embodiment is given below to introduce the process of obtaining millimeter wave characteristics in detail: (1) Detecting the historical occlusion scene based on the millimeter-wave radar to obtain historical millimeter-wave information.

[0054] Specifically, after the millimeter-wave radar is used to detect the historical occlusion scene, the corresponding historical millimeter-wave information is obtained. The received historical millimeter-wave information is processed by a radar signal processing algorithm to extract parameters such as the distance and angle of different targets in the historical occlusion scene, and these parameters are parsed to form the historical millimeter-wave information. For example, in one embodiment, a millimeter-wave radar is used to detect an object blocked by a vehicle in a historical scene, and the reflected signal is processed by an FFT algorithm to obtain the historical millimeter-wave information corresponding to the object.

[0055] (2) The historical millimeter wave information is reduced in dimension and mapped into a bird's-eye view space to obtain two-dimensional data.

[0056] Specifically, the bird's-eye view space is a representation space that projects three-dimensional space information onto a two-dimensional plane. It should be noted that in the bird's-eye view space, the position and shape of objects such as obstacles can be more intuitively represented.

[0057] Furthermore, after dimensionality reduction mapping, the historical millimeter-wave radar information is converted into two-dimensional data in the bird's-eye view space, which is usually represented in the form of a two-dimensional matrix or image. In specific implementation, each pixel or grid unit in the bird's-eye view space corresponds to a position in the bird's-eye view space, and each position has corresponding information.

[0058] In specific implementation, the points in the historical millimeter wave information can be converted into the coordinate system of the bird's-eye view space, and positioned on the plane of the bird's-eye view space according to the distance and angle information of the target.

[0059] (3) Performing feature extraction on the two-dimensional data to obtain the historical millimeter wave features.

[0060] In specific implementation, the millimeter wave features corresponding to the two-dimensional data can be extracted based on the traditional feature extraction method, or the millimeter wave features corresponding to the two-dimensional data can be extracted based on the feature extraction method of the neural network. This is not limited in this embodiment. For example, in one embodiment, the two-dimensional data is input into the convolutional network CNN, and feature extraction is performed to obtain historical millimeter wave features. For another example, in another embodiment, the two-dimensional data is input into a pre-trained feature extraction Radar Encoder module, and the historical millimeter wave features are extracted through the Radar Encoder module.

[0061] The point cloud generation method based on the fusion of millimeter wave and visual image provided in this embodiment first detects the historical occlusion scene based on the millimeter wave radar to obtain the historical millimeter wave information, and then reduces the dimension of the historical millimeter wave information and maps it to the bird's-eye view space to obtain two-dimensional data, and finally extracts the features of the two-dimensional data to obtain the historical millimeter wave features. In this way, the accuracy and precision of target detection can be improved by obtaining the key information such as distance and angle information provided by the millimeter wave radar and optimizing the deep learning model to obtain the historical millimeter wave features. Furthermore, the acquisition of historical millimeter wave features enables the information of the millimeter wave radar to be fused with the information of other sensors, thereby comprehensively utilizing the advantages of different sensors and improving the overall performance and robustness of the system. Furthermore, through the multipath reflection and certain penetration ability of the millimeter wave radar, it is possible to detect targets that are blocked or out of sight. Therefore, by extracting historical millimeter wave features, the system's perception of complex environments can be significantly enhanced. In this way, by obtaining accurate historical millimeter wave features, it is possible to better adapt to different environmental and scene requirements and improve the accuracy and reliability of point cloud generation.

[0062] A specific embodiment is given below to introduce the process of obtaining visual features in detail: (1) Detecting the historical occlusion scene based on the visual camera to obtain historical visual information.

[0063] Specifically, a visual camera captures light from the surrounding environment through a lens to form an image. It can be understood that the image taken by the visual camera contains rich color, texture and shape information.

[0064] It should be noted that the historical visual information may exist in the form of RGB images, and the static and dynamic information in the historical occlusion scenes may be saved through the historical visual information.

[0065] (2) Extracting features from the historical visual information to obtain primary features.

[0066] In specific implementation, the primary features corresponding to the historical visual information can be extracted based on the traditional feature extraction method, or based on the feature extraction method of the neural network, which is not limited in this embodiment. For example, in one embodiment, the historical visual information is input into the convolutional network CNN, and the feature extraction is performed to obtain the primary features. For another example, in another embodiment, the historical visual information is input into the pre-trained feature extraction CameraEncoder module, and the primary features are extracted through the Camera Encoder module.

[0067] (3) Generate a predicted depth map based on the primary features.

[0068] Specifically, the distance information of objects in the three-dimensional scene is inferred from the primary features by using depth estimation. In specific implementation, a predicted depth map can be generated based on the visual clues in the primary features.

[0069] It should be noted that the predicted depth map can be generated based on a traditional depth estimation method or a neural network depth estimation method, and this is not limited in the present embodiment.

[0070] (4) Based on a preset calibration matrix and the predicted depth map, the primary features are mapped into a bird's-eye view space to obtain the historical visual features.

[0071] Specifically, the calibration matrix can transform the coordinate system of the visual camera and the coordinate system of the bird's-eye view space, and after processing the predicted depth map based on the calibration matrix, the historical visual features in the bird's-eye view space are obtained. It should be noted that the calibration matrix contains the geometric transformation parameters required to transform points in the camera image into the bird's-eye view space.

[0072] In specific implementation, the calibration matrix is ​​used to transform the coordinates of each primary feature from the visual camera coordinate system to the bird's-eye view space coordinate system. It should be noted that the depth information can be used to calculate the position of the pixel in the bird's-eye view space.

[0073] The point cloud generation method based on the fusion of millimeter wave and visual image provided in this embodiment first detects the historical occlusion scene based on the visual camera to obtain the historical visual information, then extracts the features of the historical visual information to obtain the primary features, then estimates the primary features in depth to generate the predicted depth map, and then maps the primary features to the bird's-eye view space based on the preset calibration matrix and the predicted depth map to obtain the historical visual features. In this way, by capturing the light of the occluded environment to be identified by the visual camera to form an image, the key information in the image, such as edges, corners, texture features, etc., can be further highlighted, thereby providing rich environmental information for subsequent processing. Further, the historical visual features not only contain the appearance information of the target, but also imply the key information such as the shape, size, and position of the target. Further, the historical visual features, as important information output by the visual camera, can improve the reliability and accuracy of point cloud generation. Further, in the process of acquiring the historical visual features, they are processed by multiple steps such as feature extraction, depth estimation, and coordinate transformation, which can reduce the influence of noise and interference and improve the stability and reliability of the historical visual features. In this way, by obtaining accurate historical visual features, it is possible to better adapt to different environmental and scene requirements and improve the accuracy and reliability of point cloud generation.

[0074] S103, using the target supervision point cloud as decoder supervision, inputting the historical millimeter wave features and the historical visual features into the decoder, and training the decoder to predict and generate occluded target point clouds and unoccluded target point clouds.

[0075] Specifically, the decoder is a complex neural network structure that is responsible for decoding high-level feature information into specific point cloud data. In specific implementation, the decoder gradually expands the input historical millimeter wave features and historical visual features into point cloud data with spatial dimensions through multi-layer convolution, transposed convolution, upsampling and other operations.

[0076] When this step is implemented specifically, an initial decoder may be trained based on historical millimeter wave features and historical visual features to obtain a trained decoder.

[0077] A specific embodiment is given below to introduce in detail the process of obtaining the occluded target point cloud and the unoccluded target point cloud: (1) Amplifying the distinguishing features between the historical millimeter wave features and the historical visual features to obtain the occlusion features of the occluded target; amplifying the common features between the historical millimeter wave features and the historical visual features to obtain the common shared features.

[0078] Specifically, by magnifying the distinguishing features between the historical millimeter wave features and the historical visual features, the uniqueness between the two can be determined. By magnifying the common features between the historical millimeter wave features and the historical visual features, the commonality between the two can be determined.

[0079] A specific embodiment is given below to introduce in detail the process of obtaining the distinguishing features and the same features: Step 1: differentially amplify the historical millimeter wave feature and the historical visual feature to obtain the distinguishing feature.

[0080] Specifically, a differential operation is performed on the millimeter-wave radar features and the historical visual features to highlight the differences between the two. It should be noted that these differences mainly reflect the target information that the millimeter-wave radar can detect but the visual camera may not be able to see due to occlusion, that is, the occlusion characteristics of the occluded object.

[0081] Further, the features obtained by differential amplification are determined as distinguishing features.

[0082] Step 2: Sum and amplify the historical millimeter wave feature and the historical visual feature to obtain the same feature.

[0083] Specifically, the millimeter wave radar features and the historical visual features are summed to retain the same information between the two. It should be noted that the same information mainly reflects the unobstructed parts of the scene, including unobstructed objects and backgrounds, that is, the same features.

[0084] Further, the features obtained by summing and amplifying are determined to be the same features.

[0085] The point cloud generation method based on the fusion of millimeter waves and visual images provided in this embodiment obtains distinguishing features by differentially amplifying historical millimeter wave features and historical visual features, and obtains identical features by summing and amplifying historical millimeter wave features and historical visual features. In this way, through differential amplification, those obscured targets can be obtained and accurate obscured target point clouds can be generated. Furthermore, through summing and amplification, a comprehensive point cloud representation containing all visible targets and background information in the scene can be generated. In this way, through identical features and distinguishing features, the comprehensiveness and accuracy of environmental perception can be improved.

[0086] (2) Based on the decoder processing the distinguishing features, the occluded target point cloud is obtained; based on the decoder processing the same features, the unoccluded target point cloud is obtained; wherein the unoccluded target point cloud includes a point cloud of the unoccluded object and a point cloud of the background.

[0087] Specifically, the distinguishing features obtained after differential amplification are input into the decoder, and the decoder decodes the distinguishing features, maps the high-level distinguishing features to specific point cloud coordinates and attributes in the three-dimensional space, and generates the occluded target point cloud.

[0088] Furthermore, the same features obtained after summing and amplification are input into the decoder, which decodes the distinguishing features, maps the high-level same features to specific point cloud coordinates and attributes in the three-dimensional space, and generates an unobstructed target point cloud.

[0089] It should be noted that since the decoder is trained using the target supervision point cloud as supervision, when the decoder generates occluded target point clouds and unoccluded target point clouds, the decoder can ensure that the generated occluded target point clouds and unoccluded target point clouds are close to the actual scene.

[0090] Specifically, the same features obtained after summing and amplification can be directly input into the decoder, and the target supervision point cloud is used as supervision information, and the decoder directly predicts and generates the point cloud of the unoccluded target.

[0091] A specific embodiment is given below to introduce in detail the process of obtaining the occluded target point cloud: (1) Determine the position information of the occlusion target corresponding to the occlusion feature and generate a region of interest.

[0092] Specifically, firstly, the occlusion features obtained after differential amplification are analyzed to determine the position information of the occlusion target in the scene.

[0093] In specific implementation, a classifier, a regressor or a clustering algorithm may be used to identify the position corresponding to the occlusion feature and determine it as the region of interest.

[0094] It should be noted that the region of interest may include the location corresponding to the occlusion feature and the area of ​​data within a certain range around it.

[0095] The occlusion features corresponding to the occluded target can be input into the feature detector to locate the region of interest.

[0096] (2) Inputting the features in the region of interest into the decoder, and obtaining high-dimensional target coordinates and target attributes based on the decoder.

[0097] In specific implementation, features within the region of interest may be extracted based on a traditional feature extraction method, or may be extracted based on a neural network feature extraction method, which is not limited in this embodiment.

[0098] Furthermore, the features in the extracted region of interest are input into the decoder, and the features are decoded and transformed by the decoder to generate high-dimensional target coordinates and target attributes.

[0099] (3) De-noising the target coordinates and the target attributes based on the target supervision point cloud to generate the occluded target point cloud.

[0100] Specifically, the target coordinates and target attributes are supervised through the target supervision point cloud to generate a high-quality occluded target point cloud.

[0101] In specific implementation, the features of the region of interest can be directly input into the decoder, and the target supervision point cloud can be used as supervision information, and the decoder can directly predict and generate the occluded target point cloud.

[0102] The point cloud generation method based on the fusion of millimeter waves and visual images provided in this embodiment first determines the position information of the occluded target corresponding to the occlusion feature, generates a region of interest, and then inputs the features in the region of interest into the decoder, obtains high-dimensional target coordinates and target attributes based on the decoder, and finally denoises the target coordinates and target attributes based on the target supervision point cloud to generate an occluded target point cloud. In this way, the amount of calculation for subsequent processing is reduced through the region of interest, and attention is focused on the occluded target, thereby ensuring the accuracy and reliability of the occluded target point cloud. Furthermore, by analyzing the occlusion features and determining the position information of the occluded target, the occluded target can be identified more accurately. In this way, the reliability and accuracy of the point cloud generation corresponding to the occluded scene to be identified can be guaranteed through accurate and reliable occluded point clouds.

[0103] In this step, the decoder is trained through the target supervision point cloud, historical millimeter wave features and historical visual features to obtain a trained decoder.

[0104] S104, identifying the occluded scene to be identified based on the millimeter wave radar and the visual camera, obtaining millimeter wave features and visual features, and generating a scene point cloud of the occluded scene to be identified based on the millimeter wave features, the visual features and the trained decoder prediction.

[0105] Specifically, the occlusion scene to be identified is the same as the historical occlusion scene, both of which are scenes containing occlusion. In specific implementation, when point cloud generation is required, the corresponding scene is determined as the occlusion scene to be identified.

[0106] Furthermore, based on the detection of the occluded scene to be identified by the millimeter-wave radar and the visual camera, the corresponding millimeter-wave features and visual features can be generated.

[0107] Furthermore, after extracting the millimeter wave features and visual features corresponding to the occluded scene to be identified, the millimeter wave features and visual features are input into the trained decoder, and the scene point cloud of the occluded scene to be identified is generated by the decoder.

[0108] It can be understood that the scene point cloud includes the point cloud of unobstructed objects in the occluded scene to be identified, and also includes the point cloud of occluded objects. In specific implementation, the decoder combines the millimeter wave features and the visual features to determine the point cloud of unobstructed objects and the point cloud of occluded objects respectively, and combines the two to obtain the scene point cloud.

[0109] Furthermore, since the point cloud of the unobstructed object and the point cloud of the obstructed object can be generated in different coordinate systems or perspectives, they need to be aligned first. In specific implementation, operations such as coordinate transformation, rotation and translation can be performed to ensure the spatial consistency of the point cloud of the unobstructed object and the point cloud of the obstructed object.

[0110] Furthermore, the point cloud of the unobstructed object and the point cloud of the obstructed object can be fused based on the traditional point cloud fusion method, or the point cloud of the unobstructed object and the point cloud of the obstructed object can be fused based on the point cloud fusion method of the neural network. In this embodiment, there is no limitation on this. In specific implementation, a scene point cloud containing all the information of the obstructed scene to be identified can be generated by superposition, weighted average or more complex fusion algorithm.

[0111] The point cloud generation method based on the fusion of millimeter wave and visual image provided by the present application first obtains the laser radar data of the historical occlusion scene, and generates the target supervision point cloud based on the laser radar data; wherein, the point cloud density of the occluded target in the laser radar data is less than the first preset threshold, the point cloud density of the occluded target in the target supervision point cloud is greater than the first preset threshold, and the point cloud contour of the occluded target in the target supervision point cloud is complete; then, the historical occlusion scene is identified based on the millimeter wave radar and the visual camera, and the historical millimeter wave features and the historical visual features are obtained; then, the target supervision point cloud is used as the decoder supervision, and the historical millimeter wave features and the historical visual features are input into the decoder, and the decoder is trained to predict and generate the occluded target point cloud and the unoccluded target point cloud; finally, the occluded scene to be identified is identified based on the millimeter wave radar and the visual camera, and the millimeter wave features and the visual features are obtained, and the scene point cloud of the occluded scene to be identified is generated based on the millimeter wave features, the visual features and the trained decoder prediction. In this way, the millimeter wave features of the occluded part can be effectively obtained through the penetration ability of the millimeter wave radar, and at the same time, the visual features containing rich contextual information can be effectively obtained through the high-level semantic information of the visual camera. Furthermore, the unobstructed point cloud is determined based on the same features of the millimeter-wave features and the visual features, and the occluded point cloud is determined based on the distinguishing features of the millimeter-wave features and the visual features. In this way, the combination of the occluded point cloud and the unobstructed point cloud can effectively detect and identify the obstructed target. The penetration ability of the millimeter-wave radar makes up for the shortcomings of the visual image in the case of occlusion, and the high-level semantic information of the visual image helps to improve the accuracy of detection. Furthermore, when generating the occluded point cloud and the unobstructed point cloud, the data enhancement method is used to obtain the target supervised point cloud with complete and dense shape contours to generate the target training sample decoder to ensure the accuracy and reliability of the point cloud generation. In this way, the scene point cloud finally generated contains the complete information of all targets in the scene, which helps to reduce the misrecognition caused by the responsible environment and ensures the reliability and accuracy of the point cloud generation.

[0112] Corresponding to the aforementioned embodiment of a point cloud generation method based on the fusion of millimeter waves and visual images, the present application also provides an embodiment of a point cloud generation device based on the fusion of millimeter waves and visual images.

[0113] The present application provides an embodiment of a point cloud generation device based on the fusion of millimeter waves and visual images, which can be applied to a point cloud generation device based on the fusion of millimeter waves and visual images. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of the point cloud generation device based on the fusion of millimeter waves and visual images reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From a hardware perspective, if Figure 2 As shown, it is a hardware structure diagram of a point cloud generation device based on millimeter wave and visual image fusion in the present application, except Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, the point cloud generation device based on millimeter wave and visual image fusion in which the device in the embodiment is located may also include other hardware according to the actual function of the point cloud generation device based on millimeter wave and visual image fusion, which will not be elaborated here.

[0114] Figure 3 This is a schematic diagram of the structure of the first embodiment of the point cloud generation device based on millimeter wave and visual image fusion provided by this application. Figure 3 The device provided in this embodiment includes an acquisition module 310, an identification module 320, a prediction module 330 and a generation module 340; wherein, The acquisition module 310 is used to acquire the laser radar data of the historical occlusion scene and generate the target supervision point cloud based on the laser radar data; The point cloud density of the occluded target in the laser radar data is less than a first preset threshold, the point cloud density of the occluded target in the target supervision point cloud is greater than the first preset threshold, and the point cloud contour of the occluded target in the target supervision point cloud is complete; The identification module 320 is used to identify the historical occlusion scene based on the millimeter wave radar and the visual camera, respectively, to obtain the historical millimeter wave features and the historical visual features; The prediction module 330 is used to use the target supervision point cloud as a decoder supervision, input the historical millimeter wave features and the historical visual features into the decoder, and train the decoder to predict and generate an occluded target point cloud and an unoccluded target point cloud; The generation module 340 is used to identify the occluded scene to be identified based on the millimeter wave radar and the visual camera, obtain millimeter wave features and visual features, and generate a scene point cloud of the occluded scene to be identified based on the millimeter wave features, the visual features and the trained decoder prediction.

[0115] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0116] Optionally, the acquisition module 310 is specifically used to split the laser radar data into a specified number of target areas, and determine the target density of the point cloud in each of the target areas; The acquisition module 310 is further specifically configured to determine the target area corresponding to the target density less than the first preset threshold as the area to be enhanced, increase the number of point clouds in the area to be enhanced, and obtain the target supervision point cloud.

[0117] Optionally, the acquisition module 310 is further specifically configured to determine the target area corresponding to the target density greater than the first preset threshold as a sample area; wherein the sample area is of the same type as the area to be enhanced; The acquisition module 310 is further specifically configured to splice the point cloud in the sample area into the area to be enhanced in proportion to obtain the target supervised point cloud.

[0118] Optionally, the acquisition module 310 is further specifically used to acquire the geometric center and corner points of the target object in the laser radar data; The acquisition module 310 is further specifically used to use the geometric center as the vertex of the target area, generate multiple bottom surfaces based on the corner points, and connect the vertex and the corner points of the bottom surfaces to generate the target area; wherein the number of the bottom surfaces is equal to the specified number; and the shapes of each of the target areas are the same.

[0119] Optionally, the identification module 320 is specifically configured to detect the historical occlusion scene based on the millimeter wave radar to obtain historical millimeter wave information; The identification module 320 is further specifically used to reduce the dimension of the historical millimeter wave information and map it into a bird's-eye view space to obtain two-dimensional data; The identification module 320 is further specifically configured to extract features from the two-dimensional data to obtain the historical millimeter wave features.

[0120] Optionally, the recognition module 320 is further specifically configured to detect the historical occlusion scene based on the visual camera to obtain historical visual information; The recognition module 320 is further specifically used to extract features from the historical visual information to obtain primary features; The recognition module 320 is further specifically configured to generate a predicted depth map based on the primary features; The recognition module 320 is further specifically configured to map the primary features into a bird's-eye view space based on a preset calibration matrix and the predicted depth map to obtain the historical visual features.

[0121] Optionally, the prediction module 330 is specifically used to amplify the distinguishing features of the historical millimeter wave features and the historical visual features to obtain the occlusion features of the occluded target; amplify the same features of the historical millimeter wave features and the historical visual features to obtain the same common features; The prediction module 330 is further specifically used to obtain the occluded target point cloud based on the decoder processing the distinguishing features; and obtain the unoccluded target point cloud based on the decoder processing the same features; wherein the unoccluded target point cloud includes the point cloud of the unoccluded object and the point cloud of the background.

[0122] Optionally, the prediction module 330 is further specifically used to determine the position information of the occlusion target corresponding to the occlusion feature, and generate a region of interest; The prediction module 330 is further specifically configured to input the features in the region of interest into the decoder, and obtain high-dimensional target coordinates and target attributes based on the decoder; The prediction module 330 is further specifically configured to perform denoising processing on the target coordinates and the target attributes based on the target supervision point cloud to generate the occluded target point cloud.

[0123] Optionally, the prediction module 330 is further specifically used to differentially amplify the historical millimeter wave feature and the historical visual feature to obtain the distinguishing feature; The prediction module 330 is further specifically configured to sum and amplify the historical millimeter wave feature and the historical visual feature to obtain the same feature.

[0124] Please continue to refer to Figure 2 The present application also provides a point cloud generation device based on millimeter wave and visual image fusion, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0125] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of any one of the methods provided in the present application are implemented.

[0126] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0127] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0128] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A point cloud generation method based on millimeter wave and visual image fusion, characterized in that: The method comprises: Acquire laser radar data of historical occlusion scenes, and generate a target supervision point cloud based on the laser radar data; The point cloud density of the occluded target in the laser radar data is less than a first preset threshold, the point cloud density of the occluded target in the target supervision point cloud is greater than the first preset threshold, and the point cloud contour of the occluded target in the target supervision point cloud is complete; Based on the millimeter wave radar and the visual camera, the historical occlusion scene is respectively identified to obtain the historical millimeter wave feature and the historical visual feature; Using the target supervision point cloud as a decoder supervision, inputting the historical millimeter wave features and the historical visual features into the decoder, and training the decoder to predict and generate an occluded target point cloud and an unoccluded target point cloud; Based on the millimeter wave radar and the visual camera, the occluded scene to be identified is identified to obtain millimeter wave features and visual features, and based on the millimeter wave features, the visual features and the trained decoder prediction, a scene point cloud of the occluded scene to be identified is generated.

2. The method according to claim 1, characterized in that The step of acquiring laser radar data of a historical occlusion scene and generating a target supervision point cloud based on the laser radar data includes: Splitting the laser radar data into a specified number of target areas, and determining the target density of the point cloud within each of the target areas; The target area corresponding to the target density less than the first preset threshold is determined as the area to be enhanced, and the number of point clouds in the area to be enhanced is increased to obtain the target supervision point cloud.

3. The method according to claim 2, characterized in that The step of determining the target area corresponding to the target density less than the first preset threshold as the area to be enhanced, increasing the number of point clouds in the area to be enhanced, and obtaining the target supervision point cloud includes: Determine the target area corresponding to the target density greater than the first preset threshold as a sample area; wherein the sample area is of the same type as the area to be enhanced; The point cloud in the sample area is spliced ​​into the area to be enhanced in proportion to obtain the target supervised point cloud.

4. The method according to claim 2, characterized in that: The step of dividing the laser radar data into a specified number of target areas and determining the target density of the point cloud in each target area includes: Obtaining the geometric center and corner points of the target object in the laser radar data; The geometric center is used as the vertex of the target area, multiple bottom surfaces are generated based on the corner points, and the target area is generated by connecting the vertex and the corner points of the bottom surfaces; wherein the number of the bottom surfaces is equal to the specified number; and the shapes of the target areas are the same.

5. The method according to claim 1, characterized in that The millimeter wave radar and the visual camera are used to respectively identify the historical occlusion scenes to obtain the historical millimeter wave features and the historical visual features, including: Detecting the historical occlusion scene based on the millimeter wave radar, obtaining historical millimeter wave information; Mapping the historical millimeter wave information into a bird's-eye view space by reducing the dimension of the historical millimeter wave information to obtain two-dimensional data; Feature extraction is performed on the two-dimensional data to obtain the historical millimeter wave features.

6. The method according to claim 1, characterized in that The millimeter wave radar and the visual camera are used to respectively identify the historical occlusion scenes to obtain the historical millimeter wave features and the historical visual features, including: Detecting the historical occlusion scene based on the visual camera to obtain historical visual information; Extracting features from the historical visual information to obtain primary features; generating a predicted depth map based on the primary features; Based on a preset calibration matrix and the predicted depth map, the primary features are mapped into a bird's-eye view space to obtain the historical visual features.

7. The method according to claim 1, characterized in that The method uses the target supervision point cloud as a decoder supervision, inputs the historical millimeter wave features and the historical visual features into a decoder, and trains the decoder to predict and generate an occluded target point cloud and an unoccluded target point cloud, including: Amplify the distinguishing features between the historical millimeter wave features and the historical visual features to obtain the occlusion features of the occluded target; amplify the same features between the historical millimeter wave features and the historical visual features to obtain the same common features; Based on the decoder processing the distinguishing features, the occluded target point cloud is obtained; based on the decoder processing the same features, the unoccluded target point cloud is obtained; wherein the unoccluded target point cloud includes the point cloud of the unoccluded object and the point cloud of the background.

8. The method according to claim 7, characterized in that The processing of the distinguishing features based on the decoder to obtain the occluded target point cloud includes: Determine the position information of the occlusion target corresponding to the occlusion feature, and generate a region of interest; Inputting the features in the region of interest into the decoder, and obtaining high-dimensional target coordinates and target attributes based on the decoder; The target coordinates and the target attributes are denoised based on the target supervision point cloud to generate the occluded target point cloud.

9. The method according to claim 7, characterized in that: amplifying the distinguishing features of the historical millimeter wave features and the historical visual features to obtain an occlusion feature; Amplifying the same features of the historical millimeter wave features and the historical visual features to obtain the same features includes: differentially amplifying the historical millimeter wave feature and the historical visual feature to obtain the distinguishing feature; The historical millimeter wave feature and the historical visual feature are summed and amplified to obtain the same feature.

10. A point cloud generation device based on millimeter wave and visual image fusion, characterized in that: The device comprises an acquisition module, an identification module, a prediction module and a generation module; wherein, The acquisition module is used to acquire the laser radar data of the historical occlusion scene and generate the target supervision point cloud based on the laser radar data; The point cloud density of the occluded target in the laser radar data is less than a first preset threshold, the point cloud density of the occluded target in the target supervision point cloud is greater than the first preset threshold, and the point cloud contour of the occluded target in the target supervision point cloud is complete; The recognition module is used to respectively recognize the historical occlusion scene based on the millimeter wave radar and the visual camera to obtain the historical millimeter wave features and the historical visual features; The prediction module is used to use the target supervision point cloud as a decoder supervision, input the historical millimeter wave features and the historical visual features into the decoder, and train the decoder to predict and generate an occluded target point cloud and an unoccluded target point cloud; The generation module is used to identify the occluded scene to be identified based on the millimeter wave radar and the visual camera, obtain millimeter wave features and visual features, and generate a scene point cloud of the occluded scene to be identified based on the millimeter wave features, the visual features and the trained decoder prediction.

Images