Method and system for millimeter wave point cloud enhancement and generation based on flow matching

Abstract

This invention discloses a method and system for enhancing and generating millimeter-wave point clouds based on flow matching, belonging to the field of millimeter-wave intelligent sensing technology. To address the technical problems of sparse and noisy millimeter-wave point clouds, scarce labeled data, and unstable flow matching training, this invention constructs a reversible bidirectional mapping, uses a variational autoencoder to extract features from point cloud data and perform semantic alignment, utilizes a flow matching model to perform bidirectional mapping processing on the aligned latent features, and finally jointly trains the model to achieve the restoration and generation of the point cloud to be processed. This invention improves generation stability and semantic consistency through cross-modal latent variable alignment and source-sensing flow matching mechanisms, unifies the enhancement and generation tasks, and significantly improves the data quality of millimeter-wave point clouds and the performance of downstream sensing tasks.

Description

Technical Field

[0001] This invention belongs to the field of millimeter-wave intelligent sensing technology, specifically relating to a method and system for enhancing and generating millimeter-wave point clouds based on flow matching. Background Technology

[0002] With the popularization of smart home and Internet of Things concepts, human-centered behavior perception and state understanding have become key technologies for building a smart social ecosystem. Existing human perception methods mainly rely on deep learning technology, and their performance is highly limited by the scale and quality of training data. Among various perception methods, millimeter-wave radar has received widespread attention in the field of intelligent perception due to its significant advantages such as strong privacy protection, high environmental robustness, and low cost.

[0003] However, compared to optical sensing devices such as depth cameras or LiDAR, millimeter-wave radar-acquired point cloud data has serious inherent limitations. First, millimeter-wave point clouds suffer from extreme sparseness and severe noise interference, resulting in significant loss of geometric structural information about the target. Second, the annotation cost of millimeter-wave point clouds is extremely high, making it difficult to construct large-scale training datasets. To address these challenges, existing technical solutions mainly fall into two categories: one is physical simulation-based methods, which heavily rely on high-precision 3D models, resulting in high computational costs and weak generalization ability, making it difficult to adapt to diverse real-world scenarios; the other is diffusion model-based generation methods, which often use Gaussian noise as a prior distribution and only use millimeter-wave point clouds as conditional input, failing to fully explore and utilize the inherent structural characteristics of millimeter-wave data.

[0004] Although existing research has attempted to utilize LiDiff (see reference: Nunes L, Marcuzzi R, Mersch B, et al. Scaling diffusion models to real-world 3d lidar scene completion. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 14770-14780) or RadarDiff (see reference: Zhang R, Xue D, Wang Y, et al. Towards dense and accurate radar perception via efficient cross-modal diffusion model[J]. IEEE Robotics and Automation Letters, 2024, 9(9): Generative models such as 7429-7436 are used for point cloud completion and data augmentation, but they still have the following limitations: First, existing methods usually model point cloud augmentation and generation tasks independently, making it difficult to achieve efficient and stable bidirectional mapping within the same framework, and often relying on complex diffusion sampling processes, resulting in low computational efficiency; second, ensuring semantic consistency during cross-modal data conversion remains a significant challenge; finally, in the training process of unconditional flow matching models, there is a common potential path crossing problem, leading to instability in bidirectional transmission. Therefore, there is an urgent need for a technical solution that can process sparse and incomplete point clouds within a unified framework, achieve stable and reversible mapping, and balance computational efficiency, in order to support practical applications of downstream intelligent perception tasks such as human pose estimation and behavior recognition. Summary of the Invention

[0005] The purpose of this invention is to address the low data quality caused by the sparseness and high noise of millimeter-wave point clouds, as well as the scarcity of large-scale labeled data and the instability of traditional flow matching training. This invention proposes a millimeter-wave point cloud enhancement and generation method and system based on flow matching. By constructing a reversible bidirectional mapping between millimeter-wave point clouds and dense point clouds, and introducing cross-modal latent variable alignment and source-aware flow matching mechanisms, this invention improves the stability and semantic consistency of model generation, unifies point cloud enhancement and generation tasks, and significantly enhances the performance of downstream perception tasks.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] A method for enhancing and generating millimeter-wave point clouds based on flow matching includes the following steps: Acquire pairs of millimeter-wave sparse and dense point clouds and preprocess them to obtain point cloud data pairs; The point cloud data pairs are subjected to feature extraction and semantic alignment by the encoder of the variational autoencoder to obtain semantically aligned latent features; the latent features of the transformation are restored by the decoder to obtain the predicted point cloud enhancement or generation result. The latent features of the semantic alignment are bidirectionally mapped using a flow matching model to obtain the latent features of the transformation. The variational autoencoder and the stream matching model are jointly trained; The trained variational autoencoder and flow matching model are used to process the point cloud data to obtain the final point cloud augmentation or generation result.

[0008] Furthermore, pairs of millimeter-wave sparse and dense point clouds are acquired and preprocessed to obtain point cloud data pairs, including: Simultaneously collect sparse and dense point cloud data of the scene using multi-source sensing devices; The sparse and dense point clouds are normalized by calculating the centroid and performing coordinate translation. By uniformly downsampling or increasing the number of point clouds to a uniform scale, point cloud data pairs of a preset size can be obtained.

[0009] Furthermore, feature extraction and semantic alignment are performed on the point cloud data pairs using a variational autoencoder to obtain latent features for semantic alignment, including: The point cloud distribution parameters are predicted by the encoder, and the initial latent features are obtained by reparameterized sampling. By using cross-attention and self-attention mechanisms, semantic association and enhancement are performed on the initial latent features of different modalities to obtain semantically aligned latent features.

[0010] Furthermore, the latent features of the semantic alignment are bidirectionally mapped using a flow matching model to obtain the latent features of the transformation, including: A linear interpolation trajectory is constructed between sparse and dense feature distributions, and the velocity field at the corresponding time moment is predicted by a neural network; The potential characteristics of the transformation are obtained by integrating the velocity field along the time axis using an ordinary differential equation numerical solver.

[0011] Furthermore, the variational autoencoder and the stream matching model are jointly trained, including: Calculate the autoencoder loss, semantic alignment loss, and stream matching loss, and construct the total loss. The first stage of joint training is performed based on this total loss until the variational autoencoder converges. The variational autoencoder parameters are frozen, and the flow matching model is fine-tuned in the second stage using the flow matching loss.

[0012] Further, the autoencoder loss is calculated, including: Calculate the spatial distance deviation between the decoded and reconstructed point cloud and the original point cloud; The divergence constraint between the latent feature distribution and the standard normal distribution is calculated, and the autoencoder loss is obtained by weighting it with the spatial distance deviation.

[0013] Further, the semantic alignment loss is calculated, including: Calculate the cosine similarity of cross-modal latent features in the same pair of samples; Contrastive learning constraints are constructed based on the cosine similarity, and semantic alignment loss is obtained by maximizing the similarity of positive sample pairs.

[0014] Further, the flow matching loss is calculated, including: Randomly select the potential features of the starting point or the ending point as the source point conditional input stream matching model; The mean square error between the output velocity field of the calculation model and the true value of the derivative of the interpolated trajectory is used to obtain the source point conditional flow matching loss.

[0015] Furthermore, the trained variational autoencoder and flow matching model are used to process the point cloud data to obtain the final point cloud augmentation or generation results, including: The source latent features of the point cloud to be processed are extracted by the encoder and used as the starting point and conditional input stream matching model. The latent features of the target are obtained through integral reasoning, and the final point cloud enhancement or generation result is restored through the decoder.

[0016] A millimeter-wave point cloud enhancement and generation system based on flow matching, comprising: The data preprocessing module is used to acquire and preprocess pairs of millimeter-wave sparse and dense point clouds to obtain point cloud data pairs. A variational autoencoder is used to extract features and semantically align the point cloud data pairs through an encoder to obtain semantically aligned latent features; and a decoder is used to restore the transformed latent features to obtain the predicted point cloud enhancement or generation result. A stream matching model is used to perform bidirectional mapping processing on the semantically aligned latent features to obtain the transformed latent features; The joint training module is used to jointly train the variational autoencoder and the flow matching model. The trained variational autoencoder and flow matching model are used to process the point cloud data to be processed, and the final point cloud enhancement or generation result is obtained.

[0017] The present invention has achieved the following beneficial effects.

[0018] 1. This invention employs a unified flow matching framework to construct a reversible mapping between millimeter-wave point clouds and dense point clouds. It simultaneously achieves bidirectional modeling for point cloud enhancement and generation through a single model, effectively solving the problems of low data quality and scarce labeled data in millimeter-wave sensing tasks.

[0019] 2. This invention maps two types of point clouds to a shared semantic space through a cross-modal latent alignment mechanism, enhances semantic consistency based on contrastive learning, overcomes the problems of point cloud disorder and semantic misalignment between modalities, and significantly improves the structural consistency and realism of the generated point clouds.

[0020] 3. This invention introduces a source point conditional flow matching strategy. By explicitly adding source point location information as guidance in the mapping path, it effectively alleviates the prediction ambiguity at path intersections in traditional unconditional flow matching, and improves the smoothness, stability and generation accuracy of model training.

[0021] 4. This invention has strong cross-domain application potential. By improving the quality and feature representation capabilities of point cloud data, it can effectively enhance the performance of downstream perception tasks such as human pose estimation and behavior recognition, and is applicable to various practical scenarios such as autonomous driving, intelligent perception, and digital healthcare. Attached Figure Description

[0022] Figure 1 This is a flowchart of the millimeter-wave point cloud enhancement and generation method based on flow matching in the embodiments; Figure 2 This is a flowchart illustrating the flow matching-based millimeter-wave point cloud enhancement and generation process in this embodiment. Figure 3 This is a block diagram of the millimeter-wave point cloud enhancement and generation system based on flow matching in the embodiment. Figure 4 This is a visualization of the experimental results of the method of this invention on the MM-Fi dataset. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] This invention provides a method for enhancing and generating millimeter-wave point clouds based on flow matching, the processing flow of which is as follows: Figure 1 As shown. This method enhances and generates millimeter-wave point clouds through cross-modal latent representation alignment and source point conditional flow matching, as... Figure 2As shown. In the cross-modal latent alignment process, semantic space mapping of point clouds of different modalities is achieved through point cloud encoding, cross-attention alignment of shared query vectors, contrastive learning semantic enhancement, and point cloud decoding. In the source point conditional flow matching process, bidirectional conversion between modalities is achieved through latent space linear interpolation, source point conditional velocity field learning, forward inference, and backward inference. The specific steps of this method are as follows.

[0025] Step S1: Obtain pairs of millimeter-wave sparse point clouds and dense point clouds and perform preprocessing to obtain point cloud data pairs.

[0026] Step S11: Acquire multimodal point cloud data The sparse point cloud X of the scene is obtained by deploying millimeter-wave radar, while the dense point cloud Y of the same scene at the same time is obtained by deploying depth camera or lidar.

[0027] In an optional embodiment of the present invention, a sparse point cloud X of the scene is acquired by deploying an IWR6843 or IWR1443Boost millimeter-wave radar.

[0028] In other embodiments of the present invention, the dense point cloud Y can also be obtained by constructing a three-dimensional model of the scene and then using physical simulation methods.

[0029] Step S12: Preprocess the point cloud data. The acquired sparse point cloud X and dense point cloud Y are normalized and their point counts are standardized. Specifically, the centroids of sparse point cloud X and dense point cloud Y are calculated, and the corresponding centroid coordinates are subtracted from the coordinates of all points, thereby translating the centroids of the point clouds to the origin of the world coordinate system, completing the normalization process. Different number thresholds are set for sparse point cloud X and dense point cloud Y respectively. For point clouds with more points than the threshold, the number of points is reduced to the threshold through uniform downsampling. For point clouds with fewer points than the threshold, the number of points is supplemented to the threshold by adding zero points, thus achieving the normalization of the point cloud count.

[0030] In an optional embodiment of the present invention, the number of sparse point clouds X is unified to 256 points, and the number of dense point clouds Y is unified to 512 points.

[0031] Step S2: The point cloud data pairs are subjected to feature extraction and semantic alignment by the encoder of the variational autoencoder to obtain semantically aligned latent features; the latent features of the transformation are restored by the decoder to obtain the predicted point cloud enhancement or generation result.

[0032] Step S21: Construct a mode-specific variational autoencoder Based on the VecSet model (3DShape2VecSet: A 3D Shape Representation for NeuralFields and Generative Diffusion Models), two variational autoencoders are constructed for a sparse point cloud X and a dense point cloud Y, respectively. The two autoencoders have identical structures and independent parameters. Each variational autoencoder consists of an encoder and a decoder. The encoder and decoder for the sparse point cloud are denoted as follows: and The encoder and decoder for dense point clouds are denoted as follows: and .

[0033] In other embodiments of the present invention, the VecSet model can also be implemented using any variational autoencoder that meets the conditions.

[0034] Step S22: Extract point cloud features and perform reparameterized sampling. The encoder is used to extract latent features from sparse point cloud X and dense point cloud Y respectively. and Both have the same shape. During the extraction process, the encoder predicts the mean of the latent representation distribution. Sum of logarithmic variance And latent features are obtained by sampling using reparameterization techniques. To enable backpropagation of gradients during training, the sampling formula is as follows: in, This represents random noise sampled from a standard normal distribution; This indicates element-wise multiplication.

[0035] In an optional embodiment of the present invention, the sparse point cloud X is passed through an encoder. Processed Dense point cloud Y passes through encoder Processed Potential characteristics for A dimensional vector represents a latent vector corresponding to a point cloud, which is represented by N D-dimensional latent features.

[0036] Step S23: Perform semantic alignment using attention mechanisms and contrastive learning. Introduce a set of shared, learnable query vectors ,in By employing a cross-attention mechanism to semantically align latent features from different modalities and utilizing a three-layer self-attention mechanism for feature enhancement, semantically consistent latent representations are obtained. and : CrossAttention represents the cross-attention operation; SelfAttention represents the self-attention operation.

[0037] Step S24, perform point cloud reconstruction Using decoder and semantically aligned latent representation and The high-dimensional compressed features are then converted back to point cloud spatial coordinates. This step establishes a deterministic connection between the latent space and the original point cloud data by mapping the high-dimensional compressed features back to the three-dimensional physical space. This provides a basis for subsequent calculation of point cloud reconstruction loss and for measuring the model's ability to recover geometric structures under different modalities, ensuring that the extracted semantically consistent features still retain complete spatial geometric information.

[0038] Step S3: The potential features of the semantic alignment are bidirectionally mapped using a flow matching model to obtain the potential features of the transformation.

[0039] Step S31, bidirectional stream matching mapping Flow matching model through time An ordinary differential equation (ODE) on a sample from the original distribution is defined. To target distribution samples Mapping between: in, Indicates that the weight is The velocity field parameterized by the neural network; t represents time.

[0040] Path defined by linear interpolation , making As a starting point ( ), As the end point ( ).

[0041] Using the velocity field, inference is performed through an ODE numerical solver to achieve... and Reversible continuous transformations between them: By performing a positive integration along the time interval from 0 to 1, Convert to By performing inverse integration along time 1 to 0, Convert to .

[0042] In an optional embodiment of the present invention, in order to achieve condition generation, additional input information c is added during the velocity field learning process for condition constraints, that is, the model learns the velocity field. .

[0043] Step S32, Source conditional flow matching To address the issue of ambiguous velocity field predictions caused by path intersections between different sample pairs at the same time t, this method explicitly introduces latent representations of the starting point or ending point as conditional information into the model input, enabling the model to make accurate directional judgments at path intersections based on the source point's identity.

[0044] Step S4: Jointly train the variational autoencoder and the stream matching model.

[0045] Step S41: Construct the total loss function for joint training. The total loss function during joint training is defined as follows: in, Represents the total loss function; This represents the variational autoencoder loss; Indicates semantic alignment loss; This represents the flow matching loss.

[0046] Step S411, calculate the autoencoder loss The chamfer distance and bulldozer distance are used as loss functions to measure the reconstruction effect, and the total loss function of the variational autoencoder is constructed by combining KL divergence constraints. : in, Indicates chamfer distance loss; Indicates the distance loss of the bulldozer; Indicates the weighting coefficient; This represents the KL divergence constraint. The variational autoencoder parameters are optimized using the backpropagation algorithm until the model converges.

[0047] Step S412, calculate semantic alignment loss To further enhance the semantic correspondence between modalities, InfoNCE loss is applied to the latent vectors. Apply semantic constraints: in, Indicates cosine similarity; Let B represent the temperature coefficient; B represent the batch size; and M represent the number of query vectors. This loss function maximizes the similarity of latent features at corresponding positions within the same pair of samples while minimizing the similarity of latent features with other samples.

[0048] Step S413, calculate the flow matching loss The flow matching model minimizes the predicted velocity field and the true velocity. The mean squared error (MSE) between the ground truth values ​​is used for training. The calculation formula is as follows: in, Corresponding endpoint latent representation , Corresponding starting point potential representation .

[0049] During training, select randomly (e.g., with a 50% probability). or As a source condition The source point latent representation is passed as part of the input to the flow matching model, and information is fused using a cross-attention mechanism. The loss function of the source point conditional flow matching model is... The definition is as follows: in, Indicate the source condition; This represents the function for calculating the mean squared error.

[0050] Step S42, Perform the first phase of joint training. In the initial training phase, the variational autoencoder and the flow matching model are jointly trained until the variational autoencoder converges.

[0051] During joint training, a gradient pruning strategy is employed to process gradients, mitigating optimization conflicts between loss functions of different tasks and stabilizing the training process. This is achieved through iterative optimization of the overall loss function. This continues until the variational autoencoder model reaches convergence.

[0052] Step S43: Perform the second-stage fine-tuning of the flow matching model. After the variational autoencoder converges, all parameters of the variational autoencoder are frozen, and the result is obtained solely based on the flow matching loss. The source-point conditional flow matching model is iteratively fine-tuned to achieve refined training of the flow matching module.

[0053] In an optional embodiment of the present invention, when the variational autoencoder model converges after 10 consecutive iterations and the reconstruction loss of the variational autoencoder no longer decreases, the model is then entered into the second stage of fine-tuning.

[0054] Step S5: Use the trained variational autoencoder and flow matching model to process the point cloud data to obtain the final point cloud enhancement or generation result.

[0055] During the inference phase, for the millimeter-wave point cloud augmentation task, the sparse point cloud X to be augmented is input into the encoder. The latent representation is calculated. This was then used as the starting and source conditional input stream matching model. The target latent representation is output by positive integration along time t from 0 to 1. Finally, the decoder is used. The point cloud is restored to a high-quality dense point cloud; for point cloud generation tasks, the dense point cloud Y is input into the encoder. Obtaining the latent representation Using it as the starting point and source condition, the flow matching model is applied. The latent representation predicted by the inverse integral from 1 to 0 along time t is output. Ultimately, the decoder is used. It is restored to a simulated millimeter-wave sparse point cloud.

[0056] This invention also provides a millimeter-wave point cloud enhancement and generation system based on flow matching, such as... Figure 3 As shown, it includes: The data preprocessing module is used to acquire and preprocess pairs of millimeter-wave sparse and dense point clouds to obtain point cloud data pairs. A variational autoencoder is used to extract features and semantically align the point cloud data pairs through an encoder to obtain semantically aligned latent features; and a decoder is used to restore the transformed latent features to obtain the predicted point cloud enhancement or generation result. A stream matching model is used to perform bidirectional mapping processing on the semantically aligned latent features to obtain the transformed latent features; The joint training module is used to jointly train the variational autoencoder and the flow matching model. The trained variational autoencoder and flow matching model are used to process the point cloud data to be processed, and the final point cloud enhancement or generation result is obtained.

[0057] Experimental test: 1. Experimental Environment and Hardware / Software Configuration: This experiment is implemented using the Python programming language, and the system has good cross-platform compatibility. Specific hardware requirements are as follows: Intel Core i7 or higher processor, at least 8GB of RAM, and at least 16GB of hard disk space; the computing core uses an NVIDIA GeForce RTX 3090 graphics card with 24GB of video memory. The software environment is based on the Ubuntu 16.04 operating system and the PyTorch 1.12.1 deep learning framework.

[0058] 2. Parameter Settings and Optimization Strategy: The Adam optimization algorithm was used for gradient descent to solve the neural network parameters. The specific experimental parameters were set as follows: initial learning rate of 0.0001, number of training iterations (Epochs) of 200, and batch size of 32.

[0059] 3. Data Source and Preprocessing: The experimental data used the open-source multi-modal non-intrusive 4D human dataset MM-Fi. This dataset covers 27 categories of daily and rehabilitation activity records from 40 subjects. The training and test sets were divided according to official standards. To address the sparsity of millimeter-wave point clouds, composite frames were constructed by accumulating four consecutive frames of point clouds, and extremely sparse samples with an average of less than 10 points in the original frames (i.e., less than 50 points in the composite frame) were removed. After preprocessing, the training set contained 207,610 samples, and the test set contained 69,271 samples.

[0060] 4. Evaluation metrics: This experiment uses Chamfer Distance (CD) and EarthMover's Distance (EMD) as the core quantitative metrics for evaluating the point cloud generation effect.

[0061] 5. Quantitative evaluation results on the test set show: (1) In the point cloud generation task (generating millimeter wave point cloud from dense point cloud), the average chamfer distance between the generated point cloud and the ground truth is 2.69 and the average bulldozer distance is 8.49.

[0062] (2) In the point cloud augmentation task (from millimeter wave point cloud to dense point cloud), the average chamfer distance between the augmentation result and the true value is 0.75 and the average bulldozer distance is 3.69.

[0063] Visualization results as follows Figure 4 As shown.

[0064] 6. To verify the actual performance of this method for millimeter-wave point cloud enhancement, this embodiment introduces an action recognition task for comparative testing. The experiment uses the 3DInAction (3DInAction: Understanding Human Actions in 3DPoint Clouds) model as a benchmark: (1) The original 108,305 millimeter-wave point cloud samples were used for training, and the classification accuracy on the test set was 67.8%.

[0065] (2) The above samples were enhanced using the method of the present invention, and the classification accuracy of the test set was increased to 79.0% by using only the enhanced samples for training.

[0066] Experimental data show that the present invention can significantly improve the performance of neural network models in downstream perception tasks.

[0067] 7. Experimental Conclusions: The above experiments verify that the system proposed in this invention possesses excellent millimeter-wave point cloud enhancement and generation capabilities, demonstrating strong robustness. This invention not only provides high-quality input data for intelligent sensing applications but also enables low-cost expansion of the scale of labeled data, providing technical support for a wider range of intelligent sensing scenarios.

[0068] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A method for millimeter wave point cloud enhancement and generation based on flow matching, characterized in that, Includes the following steps: Acquire pairs of millimeter-wave sparse and dense point clouds and preprocess them to obtain point cloud data pairs; The point cloud data pairs are subjected to feature extraction and semantic alignment by a variational autoencoder to obtain latent features for semantic alignment. The latent features of the transformation are restored by the decoder to obtain the predicted point cloud enhancement or generation result; The latent features of the semantic alignment are bidirectionally mapped using a flow matching model to obtain the latent features of the transformation. The variational autoencoder and the stream matching model are jointly trained; The trained variational autoencoder and flow matching model are used to process the point cloud data to obtain the final point cloud augmentation or generation result.

2. The method of claim 1, wherein, Acquire paired millimeter-wave sparse and dense point clouds and preprocess them to obtain point cloud data pairs, including: Simultaneously collect sparse and dense point cloud data of the scene using multi-source sensing devices; The sparse and dense point clouds are normalized by calculating the centroid and performing coordinate translation. By uniformly downsampling or increasing the number of point clouds to a uniform scale, point cloud data pairs of a preset size can be obtained.

3. The method of claim 1, wherein, The point cloud data pairs are subjected to feature extraction and semantic alignment using a variational autoencoder to obtain latent features for semantic alignment, including: The point cloud distribution parameters are predicted by the encoder, and the initial latent features are obtained by reparameterized sampling. By using cross-attention and self-attention mechanisms, semantic association and enhancement are performed on the initial latent features of different modalities to obtain semantically aligned latent features.

4. The method of claim 1, wherein, The latent features of the semantic alignment are bidirectionally mapped using a flow matching model to obtain the latent features of the transformation, including: A linear interpolation trajectory is constructed between sparse and dense feature distributions, and the velocity field at the corresponding time moment is predicted by a neural network; The potential characteristics of the transformation are obtained by integrating the velocity field along the time axis using an ordinary differential equation numerical solver.

5. The method of claim 1, wherein, Joint training of the variational autoencoder and the stream matching model includes: Calculate the autoencoder loss, semantic alignment loss, and stream matching loss, and construct the total loss. The first stage of joint training is performed based on this total loss until the variational autoencoder converges. The variational autoencoder parameters are frozen, and the flow matching model is fine-tuned in the second stage using the flow matching loss.

6. The method of claim 5, wherein, Calculating the autoencoder loss includes: Calculate the spatial distance deviation between the decoded and reconstructed point cloud and the original point cloud; The divergence constraint between the latent feature distribution and the standard normal distribution is calculated, and the autoencoder loss is obtained by weighting it with the spatial distance deviation.

7. The method of claim 5, wherein, Calculating the semantic alignment loss includes: Calculate the cosine similarity of cross-modal latent features in the same pair of samples; Contrastive learning constraints are constructed based on the cosine similarity, and semantic alignment loss is obtained by maximizing the similarity of positive sample pairs.

8. The method of claim 5, wherein, Calculate the flow matching loss, including: Randomly select the potential features of the starting point or the ending point as the source point conditional input stream matching model; The mean square error between the output velocity field of the calculation model and the true value of the derivative of the interpolated trajectory is used to obtain the source point conditional flow matching loss.

9. The method of claim 1, wherein, The trained variational autoencoder and flow matching model are used to process the point cloud data to obtain the final point cloud augmentation or generation results, including: The source latent features of the point cloud to be processed are extracted by the encoder and used as the starting point and conditional input stream matching model. The latent features of the target are obtained through integral reasoning, and the final point cloud enhancement or generation result is restored through the decoder.

10. A flow matching based millimeter wave point cloud augmentation and generation system, comprising: include: The data preprocessing module is used to acquire and preprocess pairs of millimeter-wave sparse and dense point clouds to obtain point cloud data pairs. A variational autoencoder is used to extract features and semantically align the point cloud data pairs through an encoder, thereby obtaining latent features for semantic alignment. The latent features of the transformation are restored by the decoder to obtain the predicted point cloud enhancement or generation result; A stream matching model is used to perform bidirectional mapping processing on the semantically aligned latent features to obtain the transformed latent features; The joint training module is used to jointly train the variational autoencoder and the flow matching model. The trained variational autoencoder and flow matching model are used to process the point cloud data to be processed, and the final point cloud enhancement or generation result is obtained.

Images