A Method and System for Image and Point Cloud Feature Fusion Based on Conditional Generative Adversarial Networks

Abstract

This invention belongs to the field of image data fusion technology, and provides a method and system for image and point cloud feature fusion based on conditional generative adversarial networks. The method involves acquiring first point cloud data and RGB image data, and preprocessing the first point cloud data to obtain second point cloud data; calling a target conditional generative adversarial network model to generate target synthetic point cloud data based on the second point cloud data and RGB image data; generating bi-Gaussian distributed synthetic point cloud features based on the target synthetic point cloud data and the second point cloud data; and processing the target synthetic point cloud data, the second point cloud data, and the bi-Gaussian distributed synthetic point cloud features to obtain fused point cloud features. This solves the problem of how to effectively fuse image data and point cloud data, improving the performance of intelligent driving systems in unmanned mining areas.

Description

Technical Field

[0001] This invention relates to the field of image data fusion technology, and in particular to a method and system for fusing image and point cloud features based on conditional generative adversarial networks. Background Technology

[0002] The introduction of unmanned mining trucks and unmanned bulldozers in open-pit coal mines has greatly facilitated unmanned operations. Intelligent driving systems typically need to detect obstacles in the mining area, such as workers, material piles, and retaining walls, to ensure safe operation. Therefore, to improve operational efficiency while reducing labor costs, it is necessary to promote the intelligent development of target detection in mining areas. Currently, target detection methods can be divided into three types: image-based target detection, radar point cloud-based target detection, and image-point cloud fusion-based target detection. However, mining areas often have harsh natural conditions that severely affect the effectiveness of image-based detection. Furthermore, target detection based solely on radar point clouds does not utilize the target's texture, color, and shape. By using image-point cloud fusion-based target detection methods, the advantages of both modalities can be utilized simultaneously, compensating for the shortcomings of a single modality, thereby improving the accuracy and robustness of target detection.

[0003] Currently, object detection methods based on image and point cloud fusion mainly include the following: One method maps a 2D image into 3D space through depth estimation, then encodes it into Bird's Eye View (BEV) features, and fuses them with the BEV features of the point cloud data; another method projects each point in the LiDAR point cloud onto the output of an image semantic segmentation network, appends the segmentation results to the corresponding LiDAR points, thereby adding semantic information of the image to the point cloud, and finally inputs the point cloud with fused image features into a LiDAR-based 3D object detection network for detection; yet another method utilizes BEV images, a front view of the point cloud, and the three primary colors (Red, Green, Blue, RGB). Figure 3 The above methods fuse feature maps, making full use of the positional information of point cloud data and the semantic information of image data. Although the above methods have achieved good recognition results after experimental verification, due to the sparsity and noise of point cloud data, complex preprocessing is required in the early stage of model training, and feature alignment of different modalities is difficult, requiring a large amount of computing resources. In the process of complex processing and calculation, important feature information is inevitably lost.

[0004] Therefore, how to effectively fuse image data and point cloud data is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an image and point cloud feature fusion method and system based on conditional generative adversarial networks to solve the problem of how to effectively fuse image data and point cloud data in the prior art; that is, embodiments of the present invention can improve the performance of intelligent driving systems in unmanned mining areas.

[0006] According to one aspect of the present invention, an image and point cloud feature fusion method based on conditional generative adversarial networks is provided. The method includes: acquiring first point cloud data and RGB image data, and preprocessing the first point cloud data to obtain second point cloud data; invoking a target conditional generative adversarial network model to generate target synthetic point cloud data based on the second point cloud data and the RGB image data; generating bi-Gaussian distributed synthetic point cloud features according to the target synthetic point cloud data and the second point cloud data, and processing the target synthetic point cloud data, the second point cloud data, and the bi-Gaussian distributed synthetic point cloud features to obtain fused point cloud features.

[0007] In one embodiment, the step of invoking a target conditional generative adversarial network model to generate target synthetic point cloud data based on the second point cloud data and the RGB image data includes: extracting point cloud features based on the second point cloud data and extracting RGB image features based on the RGB image data; training the conditional generative adversarial network model based on the point cloud features and the RGB image features to obtain a target conditional generative adversarial network model; and processing the second point cloud data and the RGB image data based on the target conditional generative adversarial network model to generate target synthetic point cloud data.

[0008] In one embodiment, training a conditional generative adversarial network (GAN) model based on the point cloud features and the RGB image features to obtain a target GAN model includes: acquiring a random noise vector; generating synthetic point cloud features based on the random noise vector, the point cloud features, and the RGB image features; obtaining synthetic point cloud data based on the synthetic point cloud features; performing discrimination processing based on the synthetic point cloud features and the point cloud features to obtain a discrimination result; acquiring random point cloud features; generating a first loss value based on the synthetic point cloud features; generating a third loss value based on the synthetic point cloud data and the second point cloud data; generating a fourth loss value based on the synthetic point cloud features and the random point cloud features; performing a weighted summation of the first loss value, the third loss value, and the fourth loss value to obtain a target loss value; and optimizing the model parameters in the GAN model in the direction of reducing the target loss value to obtain the target GAN model.

[0009] In one embodiment, the step of generating a double Gaussian distribution synthetic point cloud feature based on the target synthetic point cloud data and the second point cloud data, and processing the target synthetic point cloud data, the second point cloud data, and the double Gaussian distribution synthetic point cloud feature to obtain a fused point cloud feature includes: performing Poisson disk sampling on the target synthetic point cloud data and the second point cloud data respectively to obtain sampled synthetic point cloud data and third point cloud data; performing feature extraction on the sampled synthetic point cloud data and the third point cloud data respectively to obtain sampled synthetic point cloud features and third point cloud features; performing BEV projection on the sampled synthetic point cloud features and the third point cloud features respectively to obtain BEV synthetic point cloud features and BEV point cloud features; performing double Gaussian distribution sampling on the BEV synthetic point cloud features and BEV point cloud features to obtain the double Gaussian distribution synthetic point cloud feature; and stacking the BEV synthetic point cloud feature, the BEV point cloud feature, and the double Gaussian distribution synthetic point cloud feature to obtain the fused point cloud feature.

[0010] In one embodiment, the first loss value, the third loss value, and the fourth loss value are weighted and summed to obtain the target loss value, wherein the target loss value is:

[0011]

[0012] Where α is the first weight, β is the second weight, and γ is the third weight. For the first loss value, For the third loss value, P G To synthesize point cloud data, For the second point cloud data, L contrast This is the fourth loss value.

[0013] In one embodiment, the BEV synthetic point cloud features and the BEV point cloud features are sampled using a dual Gaussian distribution to obtain the dual Gaussian distributed synthetic point cloud features, wherein the dual Gaussian distributed synthetic point cloud features are:

[0014] F sample ~p(F)

[0015]

[0016] Where π1 is the first mixing coefficient, π2 is the second mixing coefficient, π1+π2=1, (μ1,Σ1) are the mean and covariance of the BEV point cloud features, (μ2,Σ2) are the mean and covariance of the BEV synthesized point cloud features, and p(F) is the synthesized point cloud data with a double Gaussian distribution. It follows a Gaussian distribution.

[0017] According to another aspect of the present invention, an image and point cloud feature fusion system based on conditional generative adversarial networks is provided. The system includes a preprocessing module, an image data generation module, and an image feature fusion module. The preprocessing module is used to acquire first point cloud data and RGB image data, and preprocess the first point cloud data to obtain second point cloud data. The image data generation module is used to invoke a target conditional generative adversarial network model to generate target synthetic point cloud data based on the second point cloud data and the RGB image data. The image feature fusion module is used to generate bi-Gaussian distributed synthetic point cloud features based on the target synthetic point cloud data and the second point cloud data, and process the target synthetic point cloud data, the second point cloud data, and the bi-Gaussian distributed synthetic point cloud features to obtain fused point cloud features.

[0018] In one embodiment, the image data generation module includes: a feature extraction module, a model training module, and a generation module. The feature extraction module is used to extract point cloud features based on the second point cloud data and to extract RGB image features based on the RGB image data. The model training module is used to train a conditional generative adversarial network (GAN) model based on the point cloud features and the RGB image features to obtain a target conditional GAN ​​model. The generation module is used to process the second point cloud data and the RGB image data based on the target GAN model to generate target synthetic point cloud data.

[0019] In one embodiment, the model training module includes: a first generation unit, a discrimination unit, a calculation unit, and a parameter optimization unit. The first generation unit is used to acquire a random noise vector, generate synthetic point cloud features based on the random noise vector, the point cloud features, and the RGB image features, and obtain synthetic point cloud data based on the synthetic point cloud features. The discrimination unit is used to perform discrimination processing based on the synthetic point cloud features and the point cloud features to obtain a discrimination result. The calculation unit is used to acquire random point cloud features, generate a first loss value based on the synthetic point cloud features, generate a third loss value based on the synthetic point cloud data and the second point cloud data, generate a fourth loss value based on the synthetic point cloud features and the random point cloud features, and perform a weighted summation of the first loss value, the third loss value, and the fourth loss value to obtain a target loss value. The parameter optimization unit is used to optimize the model parameters in the conditional generative adversarial network model in the direction of reducing the target loss value to obtain the target conditional generative adversarial network model.

[0020] In one embodiment, the image feature fusion module includes a first sampling module, a feature extraction module, a projection module, a second sampling module, and a fusion module. The first sampling module performs Poisson disk sampling on the target synthetic point cloud data and the second point cloud data respectively to obtain sampled synthetic point cloud data and third point cloud data. The feature extraction module extracts features from the sampled synthetic point cloud data and the third point cloud data respectively to obtain sampled synthetic point cloud features and third point cloud features. The projection module projects the sampled synthetic point cloud features and the third point cloud features onto the BEV projection to obtain BEV synthetic point cloud features and BEV point cloud features. The second sampling module performs double Gaussian distribution sampling on the BEV synthetic point cloud features and the BEV point cloud features to obtain double Gaussian distributed synthetic point cloud features. The fusion module stacks the BEV synthetic point cloud features, the BEV point cloud features, and the double Gaussian distributed synthetic point cloud features to obtain fused point cloud features.

[0021] In summary, in this embodiment of the invention, by acquiring first point cloud data and RGB image data, preprocessing the first point cloud data to obtain second point cloud data, calling a target conditional generative adversarial network (GAN) model, generating target synthetic point cloud data based on the second point cloud data and the RGB image data, generating bi-Gaussian distribution synthetic point cloud features based on the target synthetic point cloud data and the second point cloud data, and processing the target synthetic point cloud data, the second point cloud data, and the bi-Gaussian distribution synthetic point cloud features to obtain fused point cloud features, the problem of how to effectively fuse image data and point cloud data is solved, improving the performance of intelligent driving systems in unmanned mining areas. Simultaneously, by training the conditional GAN ​​model based on the point cloud features and the RGB image features to obtain a target conditional GAN ​​model, and processing the second point cloud data and the RGB image data based on the target conditional GAN ​​model to generate target synthetic point cloud data, a unified data modality is achieved, thereby simplifying the subsequent feature extraction and fusion process. Attached Figure Description

[0022] Further details, features, and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flowchart illustrating the image and point cloud feature fusion method based on conditional generative adversarial networks disclosed in an embodiment of this application is shown.

[0024] Figure 2 Shown Figure 1 The flowchart of step S120 shown is a schematic diagram.

[0025] Figure 3 Shown Figure 2 The flowchart of step S122 shown is shown below;

[0026] Figure 4 Shown Figure 2 A schematic diagram of the structure of the conditional generative adversarial network model in step S122 is shown;

[0027] Figure 5 Shown Figure 1 The flowchart of step S130 shown is shown below;

[0028] Figure 6 A schematic diagram of the structure of the image and point cloud feature fusion system based on conditional generative adversarial networks disclosed in an embodiment of this application is shown. Detailed Implementation

[0029] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0030] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0032] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0034] It should be noted that the execution subject of the image and point cloud feature fusion method based on conditional generative adversarial networks provided in this embodiment of the invention can be one or more electronic devices, and this invention does not limit this. The electronic device can be a terminal (i.e., a client) or a server. Therefore, when the execution subject includes multiple electronic devices, and these multiple electronic devices include at least one terminal and at least one server, the image and point cloud feature fusion method based on conditional generative adversarial networks provided in this embodiment of the invention can be jointly executed by the terminal and the server. Accordingly, the terminal mentioned herein can include, but is not limited to: smartphones, tablets, laptops, desktop computers, smartwatches, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The server mentioned herein can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0035] Based on the above description, embodiments of the present invention propose an image and point cloud feature fusion method based on conditional generative adversarial networks (GANs). This GAN-based image and point cloud feature fusion method can be executed by the aforementioned electronic device (terminal or server); or, this GAN-based image and point cloud feature fusion method can be jointly executed by a terminal and a server. For ease of explanation, the following description will use the execution of this GAN-based image and point cloud feature fusion method by an electronic device as an example.

[0036] Please see Figure 1 This is a flowchart illustrating the image and point cloud feature fusion method based on conditional generative adversarial networks (GANs) disclosed in this application. The GAN-based image and point cloud feature fusion method solves the problem of effectively fusing features from image and point cloud data, thereby improving the performance of intelligent driving systems in unmanned mining areas. It should be noted that the GAN-based image and point cloud feature fusion method in this application is not limited to... Figure 1 The flowchart shown illustrates the steps and their order. Depending on different needs, the steps in the flowchart can be added, removed, or their order changed. In this embodiment, as... Figure 1As shown, the process of the image and point cloud feature fusion method based on conditional generative adversarial networks includes at least the following steps.

[0037] S110. Acquire the first point cloud data and RGB image data, and preprocess the first point cloud data to obtain the second point cloud data.

[0038] In this embodiment of the invention, a high-precision, long-range LiDAR (Light Laser Detection and Ranging) system is installed on a vehicle in an unmanned mining area. The LiDAR is used to collect the first point cloud data P of the entire scene centered on the vehicle itself. in This represents the i1th first point cloud data, which can include surface point cloud data where targets and obstacles are clearly visible. Simultaneously, cameras are installed on vehicles in the unmanned mining area to acquire RGB image data X. in This represents the i2th RGB image data.

[0039] Since the first point cloud data covers the entire unmanned mining area in 360°, while the RGB image data only includes target identification obstacles and their surrounding environment within the field of view, the first point cloud data needs to be preprocessed to become the second point cloud data. in This represents the i3rd second point cloud data, ensuring that the viewpoint regions of the second point cloud data and the RGB image data largely overlap. Preprocessing may include noise reduction, cropping, and viewpoint adjustment.

[0040] S120. Invoke the target conditional generative adversarial network model to generate target synthetic point cloud data based on the second point cloud data and the RGB image data.

[0041] like Figure 2 As shown, in an embodiment of the present invention, Figure 2 Step S120 includes at least the following steps:

[0042] S121. Based on the second point cloud data, feature extraction is performed to obtain point cloud features, and based on the RGB image data, feature extraction is performed to obtain RGB image features.

[0043] S122. Based on the point cloud features and the RGB image features, the conditional generative adversarial network model is trained to obtain the target conditional generative adversarial network model.

[0044] like Figure 3 As shown, in an embodiment of the present invention, Figure 3 Step S122 includes at least the following steps:

[0045] S1221. Obtain a random noise vector, generate synthetic point cloud features based on the random noise vector, the point cloud features, and the RGB image features, and obtain synthetic point cloud data based on the synthetic point cloud features.

[0046] In embodiments of the present invention, such as Figure 4 As shown, the Conditional Generative Adversarial Network (GAN) model 200 includes a generator G and a discriminator D. The generator can be a network structure containing an encoder-decoder. The encoder can include convolutional layers and pooling layers. The convolutional layers are used to capture local features, while the pooling layers are used to reduce the spatial dimension of the features. At the same time, the pooling layers are also used to preserve important semantic information, such as color, texture, and shape features in RGB image features. The decoder deconvolves the features extracted by the encoder and finally outputs a tensor of dimension [(point_size*3),1] through a fully connected layer, where point_size is the number of point clouds generated, and 3 represents the three-dimensional coordinates (x,y,z) of each point cloud.

[0047] The generator receives a random noise vector. and RGB image features F X As input, initial synthetic point cloud features are generated, and these features are processed through a fully connected layer to obtain initial synthetic point cloud data. Initial synthetic point cloud data As shown in the following formula:

[0048]

[0049] Among them, F X For RGB image features, It is a random noise vector. This is the initial synthesized point cloud data.

[0050] Meanwhile, point cloud features As a supervisory signal, a cross-attention mechanism is introduced into the generator. By applying the supervisory signal to the initial synthetic point cloud features generated by the generator, the generator can dynamically adjust the geometry of the point cloud to make it more closely resemble the point cloud features. Once the conditional generative adversarial network model has completed training, it is no longer necessary to apply the supervisory signal to the initial synthetic point cloud features. As a supervisory signal, synthetic point cloud features can be directly generated based on random noise vectors and RGB image features, and synthetic point cloud data can be obtained from these features. The cross-attention mechanism is as follows:

[0051]

[0052] Where Q is the query vector, and the query vector Q is derived from the point cloud features. Extracted from RGB image features F, K is the key vector, and the key vector K is extracted from the key vector F. X Extracted from RGB image features F, V is a value vector, and the value vector V is extracted from the RGB image features F. X Extract from QK T For the similarity between Q and K, d k d represents the dimension of the key. k As a scaling factor, to avoid the value being too large.

[0053] After introducing a cross-attention mechanism, the initial synthetic point cloud features are weighted and corrected to obtain corrected synthetic point cloud features. These corrected synthetic point cloud features are then processed through a fully connected layer to obtain corrected synthetic point cloud data. The generator G is guided to generate point clouds that more closely resemble real-world conditions, where the synthetic point cloud features... It can include initial synthetic point cloud features and corrected synthetic point cloud features Synthetic point cloud data P G This can include initial synthetic point cloud data And correct the synthetic point cloud data Correcting synthetic point cloud data As shown in the following formula:

[0054]

[0055] in, Given a random noise vector, CrossAttention(Q,K,V) is the cross-attention mechanism.

[0056] S1222. Based on the synthesized point cloud features and the point cloud features, a discrimination process is performed to obtain a discrimination result.

[0057] In this embodiment of the invention, the initial synthetic point cloud features are processed through a fully connected layer to obtain the initial synthetic point cloud data. Discriminator D receives the second point cloud data and the corrected synthetic point cloud data, and performs discrimination processing on the authenticity of the corrected synthetic point cloud data to obtain the discrimination result. The discrimination result of discriminator D As shown in the following formula:

[0058]

[0059] Let's take a judgment result of 0 or 1 as an example. When the value is 1, it means that the generator G produces completely realistic results; when... If the value is 0, it indicates that the generator G is not generating well.

[0060] S1223. Obtain random point cloud features, generate a first loss value based on the synthetic point cloud features, generate a third loss value based on the synthetic point cloud data and the second point cloud data, generate a fourth loss value based on the synthetic point cloud features and the random point cloud features, and perform a weighted summation of the first loss value, the third loss value and the fourth loss value to obtain the target loss value.

[0061] In this embodiment of the invention, based on synthetic point cloud features Generate the first loss value First loss value As shown in the following formula:

[0062]

[0063] Where G is the generator, These are synthetic point cloud features.

[0064] Based on point cloud features and synthetic point cloud features Generate a second loss value Second loss value As shown in the following formula:

[0065]

[0066] Where D is the discriminator. Point cloud features This represents the features of the synthesized point cloud.

[0067] To measure the point cloud synthesis effect, a Chamfer Distance (CD) metric is introduced. CD measures the bidirectional nearest neighbor distance between the synthesized point cloud data and the second point cloud data to optimize the generator's performance. This is based on the synthesized point cloud data P. G Second point cloud data Generate a third loss value Third loss value As shown in the following formula:

[0068]

[0069] Among them, P G To synthesize point cloud data, Let x be the second point cloud data, and P be the value of x. G The value of the x-axis, y is The value of the vertical axis.

[0070] Contrast loss (CD) assesses similarity by calculating the shortest distance between points, aiming to make the synthesized point cloud geometrically as close as possible to the real point cloud. However, it gives less consideration to the semantic information of the point cloud and the relative relationships between features. To further improve the semantic consistency between the synthesized point cloud features and the real point cloud features in the feature space, contrastive loss is introduced to obtain random point cloud features and construct positive sample pairs (synthetic point cloud features and real point cloud features) and negative sample pairs (synthetic point cloud features and random point cloud features). Based on the synthesized point cloud features... and random point cloud features Generate the fourth loss value L contrast The fourth loss value L contrast As shown in the following formula:

[0071]

[0072] in, Point cloud features To synthesize point cloud features, τ represents random point cloud features (i.e., randomly selected point cloud features from other scenes), τ is the temperature coefficient, τ is used to control the smoothness of the distribution, sim() is the cosine similarity, and k and K are both positive integers.

[0073] For the first loss value Third loss value and the fourth loss value L contrast We perform a weighted summation to obtain the target loss value L, which is given by the following formula:

[0074]

[0075] Wherein, α is the first weight, β is the second weight, and γ is the third weight. α, β, and γ can all be set by the developer or user, or automatically generated adaptively; this invention does not impose any limitations on this. The loss value may include the first loss value. Second loss value Third loss value Fourth loss value L contrast And the target loss value L.

[0076] S1224. Optimize the model parameters in the conditional generative adversarial network model in the direction of reducing the target loss value to obtain the target conditional generative adversarial network model.

[0077] In this embodiment of the invention, during backpropagation, the chain rule is used to calculate the target loss value L against the generator parameter θ. G The first gradient is given by the following formula:

[0078]

[0079] in, The first loss value, L is the third loss value. contrast For the fourth loss value, θ G Here are the generator parameters, L is the target loss value, α is the first weight, β is the second weight, and γ is the third weight.

[0080] Calculate the second loss value For discriminant parameter θ D The second gradient is given by the following formula:

[0081]

[0082] in, This is the second loss value. It is a synthetic point cloud feature. For point cloud features, θ D These are the parameters for the discriminator.

[0083] Following the direction of reducing the target loss value, the gradient descent method is used to optimize the model parameters in the conditional generative adversarial network model. The gradient descent method is as follows:

[0084]

[0085]

[0086] Where, θ G For generator parameters, θ D Here are the discriminator parameters, and L is the target loss value. η1 is the first learning rate, and η2 is the second learning rate.

[0087] S123. Based on the target condition generative adversarial network model, process the second point cloud data and the RGB image data to generate target synthetic point cloud data.

[0088] S130. Generate a double Gaussian distribution synthetic point cloud feature based on the target synthetic point cloud data and the second point cloud data, and process the target synthetic point cloud data, the second point cloud data and the double Gaussian distribution synthetic point cloud feature to obtain a fused point cloud feature.

[0089] like Figure 5 As shown, in an embodiment of the present invention, Figure 5 Step S130 includes at least the following steps:

[0090] S131. Perform Poisson disk sampling on the target synthetic point cloud data and the second point cloud data respectively to obtain sampled synthetic point cloud data and third point cloud data.

[0091] In this embodiment of the invention, the texture information of the surface of the obstacle in the mining area is an important feature for distinguishing it from the surrounding mineral soil. In order to generate higher quality point cloud data that retains the texture information in the image data, the Poisson disk sampling method is adopted. Poisson disk sampling is performed on the target synthetic point cloud data and the second point cloud data respectively. The Poisson disk sampling process is as follows: Select an initial point p0 as a sampling point, add the initial point p0 to the sampled point set S, and add the initial point p0 to the active point set A. In each iteration, randomly select a point p from the active point set A, and randomly generate k candidate points in a spherical region with point p as the center and 2r as the radius. For each candidate point q, check the distance between the candidate point q and all points in the sampled point set S. When the following condition is met:

[0092]

[0093] Where S is the set of sampled points, p is a point randomly selected from the set of active points A, q is a candidate point, and r is the radius.

[0094] When the above conditions are met, candidate point q is added to both the active point set A and the sampled point set S. When none of the k candidate points meet the above conditions, point p is removed from the active point set A. This iteration is repeated until the active point set A is empty. The sampled synthetic point cloud data is obtained after the first sampling process. And third point cloud data

[0095] S132. Perform feature extraction on the sampled synthetic point cloud data and the third point cloud data respectively to obtain the sampled synthetic point cloud features and the third point cloud features.

[0096] In this embodiment of the invention, a 3D backbone network is used to extract features from the sampled synthetic point cloud data and the third point cloud data respectively, mapping the high-dimensional point cloud data to a low-dimensional feature space to obtain the sampled synthetic point cloud features. and third point cloud features This allows for the capture of local and global information from the point cloud for subsequent tasks. Multilayer perceptrons (MLPs) enhance feature representation through nonlinear transformations, improving the model's robustness to noise and variations. The backbone can be PointNet, PointNet++, VoxelNet, PointPillars, etc.

[0097] S133. Perform BEV projection on the sampled synthetic point cloud features and the third point cloud features respectively to obtain BEV synthetic point cloud features and BEV point cloud features.

[0098] In this embodiment of the invention, the sampled and synthesized point cloud features are... The point cloud features are compressed and projected along the Z-axis onto the BEV plane, and the point features within each BEV grid cell are aggregated to obtain the BEV synthetic point cloud features. The third point cloud features are then compressed and projected along the Z-axis onto the BEV plane, and the point features within each BEV grid cell are aggregated to obtain the BEV point cloud features. For each point (x, y, z) in the point cloud, its grid cell is determined based on its x and y coordinates, and the x and y coordinates of the point are normalized to the grid index. The normalization formula is as follows:

[0099]

[0100] Where, x min ymin is the minimum x-coordinate of the point cloud to be projected, ymin is the minimum y-coordinate of the point cloud to be projected, and resolution is the resolution of the grid.

[0101] S134. The BEV synthetic point cloud features and the BEV point cloud features are sampled using a double Gaussian distribution to obtain the double Gaussian distributed synthetic point cloud features.

[0102] In this embodiment of the invention, BEV synthetic point cloud features and BEV point cloud features are sampled using a dual Gaussian distribution sampling method to obtain dual Gaussian distributed synthetic point cloud features. The dual Gaussian distribution (Mixture of Gaussians, MoG) is a probabilistic model commonly used to model complex data distributions. In point cloud processing tasks, the dual Gaussian distribution can be used to model the spatial distribution of point cloud features. The mean and covariance of the BEV point cloud features are (μ1, Σ1), and the mean and covariance of the BEV synthetic point cloud features are (μ2, Σ2). The dual Gaussian distribution probabilistic model is constructed as follows:

[0103]

[0104] Where π1 is the first mixing coefficient, π2 is the second mixing coefficient, satisfying π1+π2=1, (μ1,Σ1) are the mean and covariance of the BEV point cloud features, (μ2,Σ2) are the mean and covariance of the BEV synthesized point cloud features, and p(F) is the synthesized point cloud data with a double Gaussian distribution. It follows a Gaussian distribution.

[0105] The synthesized point cloud features obtained by sampling from a biGaussian distribution are shown in the following formula:

[0106] F sample ~p(F) Formula (17)

[0107] Features of synthesized point cloud with dual Gaussian distribution F sampleThis indicates that it can capture the multimodal distribution characteristics of BEV point cloud features and BEV synthetic point cloud features, thereby better fusing the information of the two for subsequent generation of diverse fused features and enhancing the generalization ability of the model.

[0108] S135. Stack the BEV synthesized point cloud features, the BEV point cloud features, and the double Gaussian distribution synthesized point cloud features to obtain the fused point cloud features.

[0109] In this embodiment of the invention, the point cloud feature F is synthesized using a dual Gaussian distribution. sample F sample The shape is (B, C, H, W). The BEV synthesized point cloud features, BEV point cloud features, and double Gaussian distribution synthesized point cloud features are stacked along the channel dimension. The shape of the stacked features is (B, 3*C, H, W). After passing through a 3*3 two-dimensional convolutional layer, the stacked features are output as a fused point cloud feature with the shape (B, C). out (H, W). Among them, C out The choice can be made based on the requirements for fusing point cloud features. BEV synthetic point cloud features, BEV point cloud features, and dual Gaussian distribution synthetic point cloud features are spatially aligned. After stacking, convolution operations are performed directly in the spatial dimension, combining RGB image data and the first point cloud data to enhance the expressive power of the features.

[0110] In summary, the image and point cloud feature fusion method based on conditional generative adversarial networks (GANs) of this application involves acquiring first point cloud data and RGB image data, preprocessing the first point cloud data to obtain second point cloud data, calling a target GAN model, generating target synthetic point cloud data based on the second point cloud data and the RGB image data, generating bi-Gaussian distribution synthetic point cloud features based on the target synthetic point cloud data and the second point cloud data, and processing the target synthetic point cloud data, the second point cloud data, and the bi-Gaussian distribution synthetic point cloud features to obtain fused point cloud features. This solves the problem of how to effectively fuse image data and point cloud data, improving the performance of intelligent driving systems in unmanned mining areas. Furthermore, by training the GAN model based on the point cloud features and the RGB image features to obtain a target GAN model, and then processing the second point cloud data and the RGB image data based on the target GAN model to generate the target synthetic point cloud data, a unified data modality is achieved, thereby simplifying the subsequent feature extraction and fusion process.

[0111] Please see Figure 6 This is a schematic diagram of the structure of the image and point cloud feature fusion system based on conditional generative adversarial networks disclosed in an embodiment of this application. In one embodiment, such as Figure 6As shown, this application provides an image and point cloud feature fusion system 100 based on conditional generative adversarial networks (GANs). The image and point cloud feature fusion system 100 based on GANs may include at least: a preprocessing module 110, an image data generation module 130, and an image feature fusion module 150. Information interaction exists between the preprocessing module 110 and the image data generation module 130, and information interaction exists between the image data generation module 130 and the image feature fusion module 150.

[0112] The preprocessing module 110 is used to acquire the first point cloud data and RGB image data, and to preprocess the first point cloud data to obtain the second point cloud data. Specifically, a high-precision, long-range LiDAR (Light Laser Detection and Ranging) system is installed on a vehicle in the unmanned mining area. The LiDAR is used to collect the first point cloud data P of the entire scene centered on the vehicle itself. in This represents the i1th first point cloud data, which can include surface point cloud data where targets and obstacles are clearly visible. Simultaneously, cameras are installed on vehicles in the unmanned mining area to acquire RGB image data X. in This represents the i2th RGB image data. Since the first point cloud data covers the entire unmanned mining area in 360°, while the RGB image data only includes target identification obstacles and their surrounding environment within the field of view, the first point cloud data needs to be preprocessed to become the second point cloud data. in This represents the i3rd second point cloud data, ensuring that the viewpoint regions of the second point cloud data and the RGB image data largely overlap. Preprocessing may include noise reduction, cropping, and viewpoint adjustment.

[0113] The image data generation module 130 is used to invoke a target conditional generative adversarial network model to generate target synthetic point cloud data based on the second point cloud data and the RGB image data. The image data generation module 130 may include at least: a feature extraction module 131, a model training module 133, and a generation module 135.

[0114] The feature extraction module 131 is used to extract features based on the second point cloud data to obtain point cloud features, and to extract features based on the RGB image data to obtain RGB image features.

[0115] The model training module 133 is used to train the conditional generative adversarial network model based on the point cloud features and the RGB image features to obtain the target conditional generative adversarial network model. The model training module 133 may include at least: a first generation unit 1331, a discriminator unit 1333, a computation unit 1335, and a parameter optimization unit 1336.

[0116] The first generation unit 1331 is used to acquire a random noise vector, generate synthetic point cloud features based on the random noise vector, the point cloud features, and the RGB image features, and obtain synthetic point cloud data based on the synthetic point cloud features. Specifically, the conditional generative adversarial network model includes a generator G and a discriminator D. The generator can be a network structure containing an encoder-decoder. The encoder can include convolutional layers and pooling layers. The convolutional layers are used to capture local features, while the pooling layers are used to reduce the spatial dimension of the features. At the same time, the pooling layers are also used to preserve important semantic information, such as color, texture, and shape features in the RGB image features. The decoder deconvolves the features extracted by the encoder and finally outputs a tensor with dimension [(point_size*3),1] through a fully connected layer, where point_size is the number of generated point clouds, and 3 represents the three-dimensional coordinates (x,y,z) of each point cloud.

[0117] The generator receives a random noise vector. and RGB image features F X As input, initial synthetic point cloud features are generated, and these features are processed through a fully connected layer to obtain initial synthetic point cloud data. Initial synthetic point cloud data As shown in the following formula:

[0118]

[0119] Among them, F X For RGB image features, It is a random noise vector. This is the initial synthesized point cloud data.

[0120] Meanwhile, point cloud features As a supervisory signal, a cross-attention mechanism is introduced into the generator. By applying the supervisory signal to the initial synthetic point cloud features generated by the generator, the generator can dynamically adjust the geometry of the point cloud to make it more closely resemble the point cloud features. Once the conditional generative adversarial network model has completed training, it is no longer necessary to apply the supervisory signal to the initial synthetic point cloud features. As a supervisory signal, synthetic point cloud features can be directly generated based on random noise vectors and RGB image features, and synthetic point cloud data can be obtained from these features. The cross-attention mechanism is as follows:

[0121]

[0122] Where Q is the query vector, and the query vector Q is derived from the point cloud features. Extracted from RGB image features F, K is the key vector, and the key vector K is extracted from the key vector F. X Extracted from RGB image features F, V is a value vector, and the value vector V is extracted from the RGB image features F. X Extract from QK T For the similarity between Q and K, d k d represents the dimension of the key. k As a scaling factor, to avoid the value being too large.

[0123] After introducing a cross-attention mechanism, the initial synthetic point cloud features are weighted and corrected to obtain corrected synthetic point cloud features. These corrected synthetic point cloud features are then processed through a fully connected layer to obtain corrected synthetic point cloud data. The generator G is guided to generate point clouds that more closely resemble real-world conditions, where the synthetic point cloud features... It can include initial synthetic point cloud features and corrected synthetic point cloud features Synthetic point cloud data P G This can include initial synthetic point cloud data And correct the synthetic point cloud data Correcting synthetic point cloud data As shown in the following formula:

[0124]

[0125] in, Given a random noise vector, CrossAttention(Q,K,V) is the cross-attention mechanism.

[0126] The discrimination unit 1333 is used to perform discrimination processing based on the synthesized point cloud features and the point cloud features to obtain a discrimination result. Specifically, the initial synthesized point cloud features are processed through a fully connected layer to obtain initial synthesized point cloud data. Discriminator D receives the second point cloud data and the corrected synthetic point cloud data, and performs discrimination processing on the authenticity of the corrected synthetic point cloud data to obtain the discrimination result. The discrimination result of discriminator D As shown in the following formula:

[0127]

[0128] Let's take a judgment result of 0 or 1 as an example. When the value is 1, it means that the generator G produces completely realistic results; when... If the value is 0, it indicates that the generator G is not generating well.

[0129] The calculation unit 1335 is used to acquire random point cloud features, generate a first loss value based on the synthetic point cloud features, generate a third loss value based on the synthetic point cloud data and the second point cloud data, generate a fourth loss value based on the synthetic point cloud features and the random point cloud features, and perform a weighted summation of the first loss value, the third loss value, and the fourth loss value to obtain the target loss value. Specifically, based on the synthetic point cloud features... Generate the first loss value First loss value As shown in the following formula:

[0130]

[0131] Where G is the generator, These are synthetic point cloud features.

[0132] Based on point cloud features and synthetic point cloud features Generate a second loss value Second loss value As shown in the following formula:

[0133]

[0134] Where D is the discriminator. Point cloud features This represents the features of the synthesized point cloud.

[0135] To measure the point cloud synthesis effect, a Chamfer Distance (CD) metric is introduced. CD measures the bidirectional nearest neighbor distance between the synthesized point cloud data and the second point cloud data to optimize the generator's performance. This is based on the synthesized point cloud data P. G Second point cloud data Generate a third loss value Third loss value As shown in the following formula:

[0136]

[0137] Among them, P G To synthesize point cloud data, Let x be the second point cloud data, and P be the value of x. G The value of the x-axis, y is The value of the vertical axis.

[0138] Contrast loss (CD) assesses similarity by calculating the shortest distance between points, aiming to make the synthesized point cloud geometrically as close as possible to the real point cloud. However, it gives less consideration to the semantic information of the point cloud and the relative relationships between features. To further improve the semantic consistency between the synthesized point cloud features and the real point cloud features in the feature space, contrastive loss is introduced, constructing positive sample pairs (synthetic point cloud features and real point cloud features) and negative sample pairs (synthetic point cloud features and random point cloud features). Based on the synthesized point cloud features... and random point cloud features Generate the fourth loss value L contrast The fourth loss value L contrast As shown in the following formula:

[0139]

[0140] in, Point cloud features To synthesize point cloud features, τ represents random point cloud features (i.e., randomly selected point cloud features from other scenes), τ is the temperature coefficient, τ is used to control the smoothness of the distribution, sim() is the cosine similarity, and k and K are both positive integers.

[0141] For the first loss value Third loss value and the fourth loss value L contrast We perform a weighted summation to obtain the target loss value L, which is given by the following formula:

[0142]

[0143] In this invention, α is the first weight, β is the second weight, and γ is the third weight. α, β, and γ can all be set by the developer or user or automatically generated by adaptive methods. This invention does not limit these settings.

[0144] The parameter optimization unit 1336 is used to optimize the model parameters in the conditional generative adversarial network model in the direction of reducing the target loss value to obtain the target conditional generative adversarial network model. Specifically, during backpropagation, the chain rule is used to calculate the target loss value L against the generator parameters θ. G The first gradient is given by the following formula:

[0145]

[0146] in, The first loss value, L is the third loss value. contrast For the fourth loss value, θ G Here are the generator parameters, L is the target loss value, α is the first weight, β is the second weight, and γ is the third weight.

[0147] Calculate the second loss value For discriminant parameter θ D The second gradient is given by the following formula:

[0148]

[0149] in, This is the second loss value. It is a synthetic point cloud feature. For point cloud features, θ D These are the parameters for the discriminator.

[0150] Following the direction of reducing the target loss value, the gradient descent method is used to optimize the model parameters in the conditional generative adversarial network model. The gradient descent method is as follows:

[0151]

[0152]

[0153] Where, θ G For generator parameters, θ D Here are the discriminator parameters, and L is the target loss value. η1 is the first learning rate, and η2 is the second learning rate.

[0154] The generation module 135 is used to generate an adversarial network model based on the target conditions, and to process the second point cloud data and the RGB image data to generate target synthetic point cloud data.

[0155] The image feature fusion module 150 is used to generate a double Gaussian distribution synthetic point cloud feature based on the target synthetic point cloud data and the second point cloud data, and to process the target synthetic point cloud data, the second point cloud data, and the double Gaussian distribution synthetic point cloud feature to obtain a fused point cloud feature. The image feature fusion module 150 may include at least: a first sampling module 151, a feature extraction module 153, a projection module 155, a second sampling module 157, and a fusion module 159.

[0156] The first sampling module 151 is used to perform Poisson disk sampling on the target synthetic point cloud data and the second point cloud data respectively to obtain sampled synthetic point cloud data and third point cloud data. Specifically, the texture information of the surface of the obstacle in the mining area is an important feature for distinguishing it from the surrounding mineral soil. In order to generate higher quality point cloud data that retains the texture information in the image data, the Poisson disk sampling method is adopted. The target synthetic point cloud data and the second point cloud data are sampled by Poisson disk sampling respectively. The Poisson disk sampling process is as follows: Select an initial point p0 as a sampling point, add the initial point p0 to the sampled point set S, and add the initial point p0 to the active point set A. In each iteration, randomly select a point p from the active point set A, and randomly generate k candidate points in a spherical region with point p as the center and 2r as the radius. For each candidate point q, check the distance between the candidate point q and all points in the sampled point set S. When the following condition is met:

[0157]

[0158] Where S is the set of sampled points, p is a point randomly selected from the set of active points A, q is a candidate point, and r is the radius.

[0159] When the above conditions are met, candidate point q is added to both the active point set A and the sampled point set S. When none of the k candidate points meet the above conditions, point p is removed from the active point set A. This iteration is repeated until the active point set A is empty. The sampled synthetic point cloud data is obtained after the first sampling process. And third point cloud data

[0160] The feature extraction module 153 is used to extract features from the sampled synthetic point cloud data and the third point cloud data respectively, to obtain sampled synthetic point cloud features and third point cloud features. Specifically, a 3D backbone network is used to extract features from the sampled synthetic point cloud data and the third point cloud data respectively, mapping the high-dimensional point cloud data to a low-dimensional feature space to obtain sampled synthetic point cloud features. and third point cloud features This allows for the capture of local and global information from the point cloud for subsequent tasks. Multilayer perceptrons (MLPs) enhance feature representation through nonlinear transformations, improving the model's robustness to noise and variations. The backbone can be PointNet, PointNet++, VoxelNet, PointPillars, etc.

[0161] The projection module 155 is used to perform BEV projection on the sampled synthetic point cloud features and the third point cloud features respectively, to obtain BEV synthetic point cloud features and BEV point cloud features. Specifically, it projects the sampled synthetic point cloud features... The point cloud features are compressed and projected along the Z-axis onto the BEV plane, and the point features within each BEV grid cell are aggregated to obtain the BEV synthetic point cloud features. The third point cloud features are then compressed and projected along the Z-axis onto the BEV plane, and the point features within each BEV grid cell are aggregated to obtain the BEV point cloud features. For each point (x, y, z) in the point cloud, its grid cell is determined based on its x and y coordinates, and the x and y coordinates of the point are normalized to the grid index. The normalization formula is as follows:

[0162]

[0163] Where, x min Let x be the minimum x-coordinate of the point cloud to be projected, and y be the minimum y-coordinate. min y is the minimum y-coordinate of the point cloud to be projected, and resolution is the resolution of the grid.

[0164] The second sampling module 157 is used to perform bi-Gaussian distribution sampling on the BEV synthetic point cloud features and the BEV point cloud features to obtain the bi-Gaussian distribution synthetic point cloud features. Specifically, the bi-Gaussian distribution synthetic point cloud features and the BEV point cloud features are sampled using bi-Gaussian distribution sampling to obtain the bi-Gaussian distribution synthetic point cloud features. The bi-Gaussian distribution (Mixture of Gaussians, MoG) is a probabilistic model commonly used to model complex data distributions. In point cloud processing tasks, the bi-Gaussian distribution can be used to model the spatial distribution of point cloud features. The mean and covariance of the BEV point cloud features are (μ1, Σ1), and the mean and covariance of the BEV synthetic point cloud features are (μ2, Σ2). The bi-Gaussian distribution probabilistic model is constructed as follows:

[0165]

[0166] Where π1 is the first mixing coefficient, π2 is the second mixing coefficient, satisfying π1+π2=1, (μ1,Σ1) are the mean and covariance of the BEV point cloud features, and (μ2,Σ2) are the mean and covariance of the BEV synthesized point cloud features.

[0167] The synthesized point cloud features obtained by sampling from a biGaussian distribution are shown in the following formula:

[0168] F sample ~p(F) Formula (17)

[0169] Features of synthesized point cloud with dual Gaussian distribution F sample This indicates that it can capture the multimodal distribution characteristics of BEV point cloud features and BEV synthetic point cloud features, thereby better fusing the information of the two for subsequent generation of diverse fused features and enhancing the generalization ability of the model.

[0170] The fusion module 159 is used to stack the BEV synthetic point cloud features, the BEV point cloud features, and the dual Gaussian distribution synthetic point cloud features to obtain the fused point cloud features. Specifically, the dual Gaussian distribution synthetic point cloud features F sample F sample The shape is (B, C, H, W). The BEV synthesized point cloud features, BEV point cloud features, and double Gaussian distribution synthesized point cloud features are stacked along the channel dimension. The shape of the stacked features is (B, 3*C, H, W). After passing through a 3*3 two-dimensional convolutional layer, the stacked features are output as a fused point cloud feature with the shape (B, C). out (H, W). Among them, C out The choice can be made based on the requirements for fusing point cloud features. BEV synthetic point cloud features, BEV point cloud features, and dual Gaussian distribution synthetic point cloud features are spatially aligned. After stacking, convolution operations are performed directly in the spatial dimension, combining RGB image data and the first point cloud data to enhance the expressive power of the features.

[0171] In summary, in the image and point cloud feature fusion system based on conditional generative adversarial networks of this application, the preprocessing module 110 acquires first point cloud data and RGB image data, preprocesses the first point cloud data to obtain second point cloud data, the image data generation module 130 calls the target conditional generative adversarial network model, generates target synthetic point cloud data based on the second point cloud data and the RGB image data, and the image feature fusion module 150 generates double Gaussian distribution synthetic point cloud features based on the target synthetic point cloud data and the second point cloud data, and processes the target synthetic point cloud data, the second point cloud data and the double Gaussian distribution synthetic point cloud features to obtain fused point cloud features. This solves the problem of how to effectively fuse image data and point cloud data, and improves the performance of intelligent driving systems in unmanned mining areas.

[0172] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0173] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for fusing image and point cloud features based on conditional generative adversarial networks, characterized in that, The image and point cloud feature fusion method based on conditional generative adversarial networks includes: Acquire first point cloud data and RGB image data, and preprocess the first point cloud data to obtain second point cloud data; The target conditional generative adversarial network model is invoked to generate target synthetic point cloud data based on the second point cloud data and the RGB image data; A double Gaussian distribution synthetic point cloud feature is generated based on the target synthetic point cloud data and the second point cloud data, and a fused point cloud feature is obtained by processing the target synthetic point cloud data, the second point cloud data, and the double Gaussian distribution synthetic point cloud feature. The step of calling the target conditional generative adversarial network model to generate target synthetic point cloud data based on the second point cloud data and the RGB image data includes: Point cloud features are obtained by extracting features from the second point cloud data, and RGB image features are obtained by extracting features from the RGB image data. Based on the point cloud features and the RGB image features, the conditional generative adversarial network model is trained to obtain the target conditional generative adversarial network model. Based on the target condition generative adversarial network model, the second point cloud data and the RGB image data are processed to generate target synthetic point cloud data; The step of generating a double Gaussian distribution synthetic point cloud feature based on the target synthetic point cloud data and the second point cloud data, and processing the target synthetic point cloud data, the second point cloud data, and the double Gaussian distribution synthetic point cloud feature to obtain a fused point cloud feature, includes: Poisson disk sampling is performed on the target synthetic point cloud data and the second point cloud data respectively to obtain sampled synthetic point cloud data and third point cloud data; Feature extraction is performed on the sampled synthetic point cloud data and the third point cloud data respectively to obtain the sampled synthetic point cloud features and the third point cloud features; BEV projection is performed on the sampled synthetic point cloud features and the third point cloud features respectively to obtain BEV synthetic point cloud features and BEV point cloud features. The BEV synthetic point cloud features and the BEV point cloud features are sampled using a double Gaussian distribution to obtain the double Gaussian distribution synthetic point cloud features. The BEV synthesized point cloud features, the BEV point cloud features, and the double Gaussian distribution synthesized point cloud features are stacked to obtain the fused point cloud features.

2. The image and point cloud feature fusion method based on conditional generative adversarial networks according to claim 1, characterized in that, The step of training a conditional generative adversarial network (GAN) model based on the point cloud features and the RGB image features to obtain a target GAN model includes: Obtain a random noise vector, generate synthetic point cloud features based on the random noise vector, the point cloud features, and the RGB image features, and obtain synthetic point cloud data based on the synthetic point cloud features; Based on the synthesized point cloud features and the point cloud features, a discrimination process is performed to obtain a discrimination result; Obtain random point cloud features, generate a first loss value based on the synthetic point cloud features, generate a third loss value based on the synthetic point cloud data and the second point cloud data, generate a fourth loss value based on the synthetic point cloud features and the random point cloud features, and perform a weighted summation of the first loss value, the third loss value and the fourth loss value to obtain the target loss value; The target conditional generative adversarial network model is obtained by optimizing the model parameters in the conditional generative adversarial network model in the direction of reducing the target loss value.

3. The image and point cloud feature fusion method based on conditional generative adversarial networks according to claim 2, characterized in that, The first loss value, the third loss value, and the fourth loss value are weighted and summed to obtain the target loss value, which is: Where α is the first weight, β is the second weight, and γ is the third weight. For the first loss value, For the third loss value, P G To synthesize point cloud data, For the second point cloud data, L contrast This is the fourth loss value.

4. The image and point cloud feature fusion method based on conditional generative adversarial networks according to claim 3, characterized in that, The BEV synthetic point cloud features and the BEV point cloud features are sampled using a double Gaussian distribution to obtain the double Gaussian distributed synthetic point cloud features, which are: F sample ~p(F) Where π1 is the first mixing coefficient, π2 is the second mixing coefficient, π1+π2=1, (μ1,Σ1) are the mean and covariance of the BEV point cloud features, (μ2,Σ2) are the mean and covariance of the BEV synthesized point cloud features, and p(F) is the synthesized point cloud data with a double Gaussian distribution. It follows a Gaussian distribution.

5. An image and point cloud feature fusion system based on conditional generative adversarial networks, characterized in that, The image and point cloud feature fusion system based on conditional generative adversarial networks includes: a preprocessing module, an image data generation module, and an image feature fusion module, wherein... The preprocessing module is used to acquire first point cloud data and RGB image data, and to preprocess the first point cloud data to obtain second point cloud data. The image data generation module is used to call the target conditional generative adversarial network model to generate target synthetic point cloud data based on the second point cloud data and the RGB image data; The image feature fusion module is used to generate a double Gaussian distribution synthetic point cloud feature based on the target synthetic point cloud data and the second point cloud data, and to process the target synthetic point cloud data, the second point cloud data and the double Gaussian distribution synthetic point cloud feature to obtain the fused point cloud feature. The image data generation module includes: a feature extraction module, a model training module, and a generation module, wherein... The feature extraction module is used to extract features based on the second point cloud data to obtain point cloud features, and to extract features based on the RGB image data to obtain RGB image features; The model training module is used to train the conditional generative adversarial network model based on the point cloud features and the RGB image features to obtain the target conditional generative adversarial network model. The generation module is used to generate an adversarial network model based on the target conditions, and to process the second point cloud data and the RGB image data to generate target synthetic point cloud data. The image feature fusion module includes a first sampling module, a feature extraction module, a projection module, a second sampling module, and a fusion module, wherein... The first sampling module is used to perform Poisson disk sampling on the target synthetic point cloud data and the second point cloud data respectively to obtain sampled synthetic point cloud data and third point cloud data; The feature extraction module is used to extract features from the sampled synthetic point cloud data and the third point cloud data respectively, to obtain the sampled synthetic point cloud features and the third point cloud features. The projection module is used to perform BEV projection on the sampled synthetic point cloud features and the third point cloud features respectively to obtain BEV synthetic point cloud features and BEV point cloud features. The second sampling module is used to perform double Gaussian distribution sampling on the BEV synthetic point cloud features and the BEV point cloud features to obtain the double Gaussian distribution synthetic point cloud features; The fusion module is used to stack the BEV synthesized point cloud features, the BEV point cloud features, and the double Gaussian distribution synthesized point cloud features to obtain the fused point cloud features.

6. The image and point cloud feature fusion system based on conditional generative adversarial networks according to claim 5, characterized in that, The model training module includes: a first generation unit, a discrimination unit, a computation unit, and a parameter optimization unit, wherein, The first generation unit is used to obtain a random noise vector, generate synthetic point cloud features based on the random noise vector, the point cloud features and the RGB image features, and obtain synthetic point cloud data based on the synthetic point cloud features; The discrimination unit is used to perform discrimination processing based on the synthetic point cloud features and the point cloud features to obtain a discrimination result; The computing unit is used to acquire random point cloud features, generate a first loss value based on the synthetic point cloud features, generate a third loss value based on the synthetic point cloud data and the second point cloud data, generate a fourth loss value based on the synthetic point cloud features and the random point cloud features, and perform a weighted summation of the first loss value, the third loss value and the fourth loss value to obtain the target loss value. The parameter optimization unit is used to optimize the model parameters in the conditional generative adversarial network model in the direction of reducing the target loss value to obtain the target conditional generative adversarial network model.

Images